<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Data Quality &amp; Governance on Ghost in the data</title><link>https://ghostinthedata.info/categories/data-quality--governance/</link><description>Ghost in the data</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Ghost in the data</copyright><lastBuildDate>Sat, 11 Jul 2026 09:00:00 +1000</lastBuildDate><atom:link href="https://ghostinthedata.info/categories/data-quality--governance/index.xml" rel="self" type="application/rss+xml"/><item><title>Data Engineers: Just Do Code Reviews</title><link>https://ghostinthedata.info/posts/2026/2026-07-11-code-reviews-just-do-it/</link><pubDate>Sat, 11 Jul 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-07-11-code-reviews-just-do-it/</guid><author>Chris Hillman</author><description>The case for code reviews in software engineering was settled decades ago. Data engineering hasn't caught up — and the cost is measurable in hundreds of millions of dollars per incident.</description><content:encoded>&lt;p>In &lt;em>Code Complete&lt;/em>, Steve McConnell writes:&lt;/p>
&lt;blockquote>
&lt;p>Software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, &lt;strong>the average effectiveness of design and code inspections are 55 and 60 percent&lt;/strong>.&lt;/p>&lt;/blockquote>
&lt;p>That was published in 1993. The evidence has only accumulated since. &lt;strong>Peer code review is the single most effective defect-removal technique available to software teams.&lt;/strong> Nothing else — not tests, not CI pipelines, not observability tooling — comes close on a per-hour basis.&lt;/p>
&lt;p>Software engineers have known this for fifty years. Data engineers mostly pretend it doesn&amp;rsquo;t apply to them.&lt;/p>
&lt;p>It does. And the stakes, if anything, are higher.&lt;/p>
&lt;hr>
&lt;h3 id="why-the-excuses-dont-survive-contact-with-the-data">Why the excuses don&amp;rsquo;t survive contact with the data&lt;/h3>
&lt;p>The objections are familiar. &lt;em>We&amp;rsquo;re not real software engineers. We&amp;rsquo;re too small a team. We need to move fast. Our tools don&amp;rsquo;t support it.&lt;/em> None of them hold up.&lt;/p>
&lt;p>The evidence on review effectiveness is overwhelming. Michael Fagan&amp;rsquo;s original 1976 IBM study found formal inspections caught 82% of errors before unit testing even began. NASA&amp;rsquo;s Software Engineering Labouratory found code reading detected roughly twice as many defects per hour as testing. Russell&amp;rsquo;s 1991 AT&amp;amp;T study found that each hour spent in review avoided 33 hours of downstream maintenance — making inspections up to 20× more cost-effective than testing per defect found.&lt;/p>
&lt;p>Modern numbers are just as clear. The SmartBear/Cisco study of 2,500 reviews across 3.2 million lines of code established that reviews conducted within sensible limits — under 400 lines per session, under 500 lines per hour, sessions capped at 60 minutes — yield 70–90% defect removal. Google&amp;rsquo;s case study of 9 million reviewed changes across 25,000 engineers proved the practice scales to 20,000 commits a day, with median review latency under four hours. The DORA 2023 &lt;em>State of DevOps&lt;/em> report, surveying nearly 3,000 practitioners, found that teams with faster code reviews had 50% better software delivery performance. Elite DORA performers ship with a 7× lower change failure rate than low performers.&lt;/p>
&lt;p>SmartBear&amp;rsquo;s &lt;em>State of Code Review&lt;/em> asked 800 respondents what most improves quality. Code review ranked first, with unit testing a distant second. Among teams satisfied with their software quality, 80% used tool-based peer review and 82% had documented review guidelines.&lt;/p>
&lt;p>This is not a close debate in software engineering. It shouldn&amp;rsquo;t be one in data engineering either.&lt;/p>
&lt;hr>
&lt;h3 id="data-pipelines-fail-differently--and-that-makes-reviews-more-important-not-less">Data pipelines fail differently — and that makes reviews more important, not less&lt;/h3>
&lt;p>Here&amp;rsquo;s the thing that makes the data engineering situation more precarious: application bugs are loud. They throw exceptions, crash processes, page on-call at 2am. The feedback loop that keeps application engineers honest — run it, watch it break — works quickly.&lt;/p>
&lt;p>Data bugs are quiet. The pipeline is green. Row counts look reasonable. The dashboard renders. And the number is wrong.&lt;/p>
&lt;p>Benn Stancil describes it precisely: silent data errors &amp;ldquo;don&amp;rsquo;t look suspicious and they trigger no warnings, in observability tools or in our manual spot checks. Instead, they often linger undetected, slowly and silently pushing the analytical assets that use them further and further from reality — until, during a board meeting…&amp;rdquo;&lt;/p>
&lt;p>Barr Moses of Monte Carlo has noted that time-to-detection for silent data errors can be measured in months, not minutes. Her 2023 survey of 200 data leaders found the average organisation deals with 67 data incidents per month, with average resolution time up 166% year over year to 15 hours per incident, and 68% of teams taking four or more hours just to detect a problem. In the 2024 follow-up, two-thirds of data leaders had experienced a data incident costing over $100,000 in the previous six months.&lt;/p>
&lt;p>The &amp;ldquo;run it and see if it breaks&amp;rdquo; safety net that keeps application code marginally honest doesn&amp;rsquo;t exist for pipelines. The only reliable substitute is another human reading the code before it runs.&lt;/p>
&lt;hr>
&lt;h3 id="the-public-case-studies-have-already-priced-in-what-happens-when-you-skip-it">The public case studies have already priced in what happens when you skip it&lt;/h3>
&lt;p>These are not theoretical risks.&lt;/p>
&lt;p>&lt;strong>Unity Software&lt;/strong> told investors in Q1 2022 that ingesting bad training data from a large customer cost it roughly $110 million in 2022 revenue, erasing approximately $5 billion in market capitalisation in a day. A data pipeline integrity problem, undetected until it had been compounding for quarters.&lt;/p>
&lt;p>&lt;strong>Citigroup&lt;/strong> paid a $400 million OCC fine in 2020 for long-standing failure to establish effective risk management and data governance programs — followed by another $135.6 million in 2024 for insufficient progress on remediation. Data quality and governance failures, sustained over years.&lt;/p>
&lt;p>&lt;strong>Equifax&lt;/strong> sent incorrect credit scores to lenders for three weeks in 2022, shifting scores by 25 points or more for around 300,000 consumers. The downstream decisions those scores informed — loan approvals, interest rates, credit limits — cannot be unwound.&lt;/p>
&lt;p>&lt;strong>Public Health England&lt;/strong> lost roughly 16,000 positive COVID-19 test results in October 2020 because an .xls pipeline hit Excel&amp;rsquo;s row limit. Those contacts were not traced.&lt;/p>
&lt;p>Every one of these is, at root, a data-pipeline change that no sufficiently rigorous second set of eyes reviewed before it reached production. The review wouldn&amp;rsquo;t have guaranteed they caught it. The absence of a review guaranteed they had no chance to.&lt;/p>
&lt;hr>
&lt;h3 id="the-broader-cost-is-already-priced-in--you-just-dont-see-the-bill">The broader cost is already priced in — you just don&amp;rsquo;t see the bill&lt;/h3>
&lt;p>Gartner&amp;rsquo;s running estimate is that poor data quality costs the average organisation $12.9 million per year. Thomas Redman, writing in &lt;em>Harvard Business Review&lt;/em>, put the drag on the US economy at $3.1 trillion annually. His most uncomfortable finding, from measuring 75 executives&amp;rsquo; own data: only 3% of company data meets basic quality standards, and 47% of newly-created records contain at least one critical error.&lt;/p>
&lt;p>The practitioner surveys agree. Great Expectations&amp;rsquo; 2022 study found 77% of data practitioners report quality issues, with 91% saying those issues hurt company performance. dbt Labs&amp;rsquo; &lt;em>State of Analytics Engineering 2024&lt;/em> reported 57% of data professionals cite poor data quality as their top challenge — up from 41% in 2022. The problem is getting worse. Monte Carlo&amp;rsquo;s telemetry puts it at roughly one incident per fifteen tables per year in production. Forrester&amp;rsquo;s 2023 &lt;em>Data Culture and Literacy Survey&lt;/em> found more than a quarter of respondents lose over $5 million a year to poor data quality, with 7% losing more than $25 million.&lt;/p>
&lt;p>The worst number: Monte Carlo&amp;rsquo;s 2022 data suggests data engineers spend roughly 40% of their working week — two full days — firefighting quality issues. Every hour of that is an hour a peer review before merge might have prevented.&lt;/p>
&lt;hr>
&lt;h3 id="what-the-data-community-has-actually-said-about-this">What the data community has actually said about this&lt;/h3>
&lt;p>Michael Kaminsky at Locally Optimistic is clear: &amp;ldquo;Code review for analytics is often substantively different from code review for software engineering because reviewers need to check the business logic and the analytical methods as well as the code.&amp;rdquo; Different, meaning harder and more consequential — not optional.&lt;/p>
&lt;p>Tristan Handy&amp;rsquo;s argument that mart models should be treated as stable interfaces, with versions and deprecation windows like APIs, is now the foundation of dbt&amp;rsquo;s contracts feature. It is also the right frame for what a reviewer is protecting: not just the code, but the downstream trust placed in it.&lt;/p>
&lt;p>Emilie Schario&amp;rsquo;s observation remains accurate: &amp;ldquo;Data is behind software development when it comes to learning and implementing the best practices of DevOps.&amp;rdquo; The review is how you close the gap.&lt;/p>
&lt;p>And Maxime Beauchemin, who built Airflow and Superset, has documented the cultural cost of data engineering&amp;rsquo;s second-class status. When the team is small, when velocity is the priority, when &amp;ldquo;we&amp;rsquo;re not really software engineers&amp;rdquo; is the ambient assumption, reviews are the first thing to go. And the work becomes fragile, tribal, and unreviewable — not because of the tools, but because of the habits.&lt;/p>
&lt;p>Datafold&amp;rsquo;s write-up of analytics-engineer review culture names the failure mode precisely: &amp;ldquo;Faced with the prospect of move fast and break stuff vs. move slow, carefully review, and maybe break less stuff, most reviewers will default to the first option, and &amp;lsquo;LGTM&amp;rsquo; their way through the day.&amp;rdquo;&lt;/p>
&lt;p>That is not a review culture. That is a rubber-stamp culture with extra steps.&lt;/p>
&lt;hr>
&lt;h3 id="what-a-data-engineering-review-should-actually-cover">What a data engineering review should actually cover&lt;/h3>
&lt;p>The community consensus — distilled from dbt Labs&amp;rsquo; PR guidance, the GitLab Data Team Handbook, Locally Optimistic, Datafold, and the dbt Discourse — converges on a specific checklist. One focused on the failure modes unique to data, not imported wholesale from software engineering.&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Area&lt;/th>
 &lt;th>What the reviewer checks&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>&lt;strong>SQL logic and grain&lt;/strong>&lt;/td>
 &lt;td>No silent fan-outs from joins; stated grain matches actual grain; CTEs are readable; row counts make sense against production&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>dbt design&lt;/strong>&lt;/td>
 &lt;td>&lt;code>ref()&lt;/code> used throughout; sources declared; correct materialisation type; incremental models have &lt;code>unique_key&lt;/code> and a sane predicate&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Schema and breaking changes&lt;/strong>&lt;/td>
 &lt;td>Contracts enforced for public models; version bumps with a deprecation window; data-diff output reviewed&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Tests&lt;/strong>&lt;/td>
 &lt;td>New models carry &lt;code>unique&lt;/code> and &lt;code>not_null&lt;/code> at the grain; grain-changing transforms have data tests; business rules encoded as tests, not comments&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Docs and metadata&lt;/strong>&lt;/td>
 &lt;td>Model and column descriptions present; source freshness configured; PII flags set&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Idempotency and performance&lt;/strong>&lt;/td>
 &lt;td>Re-runs produce identical results; no pinned &lt;code>CURRENT_TIMESTAMP&lt;/code>; partition and cluster keys considered; warehouse cost estimated&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>This is not a burdensome list. It&amp;rsquo;s the checklist a thorough engineer would run through mentally anyway — just written down and shared, so someone else can run through it too.&lt;/p>
&lt;hr>
&lt;h3 id="just-do-it">Just do it&lt;/h3>
&lt;p>The case for code review in software engineering was settled by Fagan, IBM, NASA, Cisco, Microsoft, Google, and DORA — across decades and millions of commits. Reading code finds roughly 70% of defects before they ship. Teams that review fast deliver 50% better. Silent errors cost enterprises tens of millions per year and mid-market teams their credibility with the business.&lt;/p>
&lt;p>The case for code review in data engineering is not weaker than in software engineering. It&amp;rsquo;s stronger, because pipelines fail silently, errors compound downstream before anyone notices, and the financial consequences show up on earnings calls rather than error logs.&lt;/p>
&lt;p>The practitioners who have thought hardest about this — Kaminsky, Stancil, Moses, Sanderson, Beauchemin, Handy — all land in the same place. Every SQL change, every dbt model, every DAG, every schema migration deserves a second set of eyes before it touches production. Not because data engineers are software engineers, but because the cost of pretending the practice doesn&amp;rsquo;t apply has become impossible to afford.&lt;/p>
&lt;p>The tools exist. The checklists exist. The statistics exist. The only thing missing is the decision to stop skipping it.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Code Review</category><category>Data Quality</category><category>Data Engineering</category><category>SQL</category><category>dbt</category><category>Team Culture</category><category>Best Practices</category></item><item><title>SQL Tells You What. Comments Tell You Why.</title><link>https://ghostinthedata.info/posts/2026/2026-06-06-code-tells-what-comments-why/</link><pubDate>Sat, 06 Jun 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-06-06-code-tells-what-comments-why/</guid><author>Chris Hillman</author><description>SQL is a declarative language — it tells you what the query does, never why. Here's why that distinction matters more in data engineering than anywhere else.</description><content:encoded>&lt;p>The best SQL doesn&amp;rsquo;t need comments. Write meaningful CTE names, descriptive aliases, clear column labels — and a skilled reader will follow your logic without a single annotation. That&amp;rsquo;s the right instinct.&lt;/p>
&lt;p>It&amp;rsquo;s also only half right.&lt;/p>
&lt;p>SQL is a declarative language. You&amp;rsquo;re not writing &lt;em>how&lt;/em> the database retrieves your data; you&amp;rsquo;re writing &lt;em>what&lt;/em> you want. That&amp;rsquo;s a useful distinction, because &amp;ldquo;what&amp;rdquo; and &amp;ldquo;why&amp;rdquo; are very different questions, and SQL can answer exactly one of them.&lt;/p>
&lt;p>A query can be perfectly, elegantly readable and still be completely opaque about the reason it exists. The name of your CTE, no matter how well chosen, cannot explain the business decision that gave birth to it.&lt;/p>
&lt;hr>
&lt;h3 id="the-self-documenting-ceiling">The self-documenting ceiling&lt;/h3>
&lt;p>Consider this query:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(order_total) &lt;span style="color:#66d9ef">AS&lt;/span> lifetime_value,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATEDIFF(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#66d9ef">MIN&lt;/span>(order_date), &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> customer_age_days
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">90&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> status_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">2&lt;/span>, &lt;span style="color:#ae81ff">9&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Clean SQL. No ambiguity about &lt;em>what&lt;/em> it does. But try answering these questions from the code alone:&lt;/p>
&lt;p>Why 90 days? Why not 60, or 180, or the full history? Is 90 the standard customer lifetime window, a finance reporting period, the SLA in a merchant agreement, or the number someone picked in a meeting four years ago and nobody has questioned since?&lt;/p>
&lt;p>What are status codes 1, 2, and 9? You might guess from context — pending, draft, cancelled — but you&amp;rsquo;re guessing. More importantly: why are they excluded? Is this a business rule about what counts as &amp;ldquo;real&amp;rdquo; revenue, or a data quality workaround because those status codes appear when an upstream webhook fires twice?&lt;/p>
&lt;p>These are not trivial questions. The 90-day cutoff defines what &amp;ldquo;active customer&amp;rdquo; means across your entire reporting layer. The excluded status codes determine what gets counted as revenue. Change either without understanding the original intent, and you&amp;rsquo;re not refactoring — you&amp;rsquo;re silently redefining business logic that stakeholders are relying on.&lt;/p>
&lt;p>Better SQL helps, but only so far:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(order_total) &lt;span style="color:#66d9ef">AS&lt;/span> lifetime_value,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATEDIFF(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#66d9ef">MIN&lt;/span>(order_date), &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> customer_age_days
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">90&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> status_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#75715e">-- pending
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>, &lt;span style="color:#75715e">-- draft
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#ae81ff">9&lt;/span> &lt;span style="color:#75715e">-- cancelled
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This is better. You&amp;rsquo;ve replaced the mystery with labels. But inline labels aren&amp;rsquo;t explanations — they&amp;rsquo;re just naming the what more explicitly. You still don&amp;rsquo;t know why.&lt;/p>
&lt;hr>
&lt;h3 id="what-sql-literally-cannot-tell-you">What SQL literally cannot tell you&lt;/h3>
&lt;p>SQL cannot explain why the program was written the way it was. It cannot discuss the reasons certain alternative approaches were taken. It cannot tell you that the business definition changed, that the upstream system is broken, or that this logic exists specifically to handle a problem that was supposed to be fixed in Q2 and wasn&amp;rsquo;t.&lt;/p>
&lt;p>Here&amp;rsquo;s the kind of context that belongs in a comment:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Customer lifetime window: 90-day lookback aligns with the SLA in the Merchant Agreement (v3.2).
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Reviewed and confirmed with Finance, Dec 2024. Don&amp;#39;t change without checking with that team first.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- status_codes 1, 2, 9 excluded per the order lifecycle defined in the legacy Salesforce migration.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- These codes appear as artefacts when the Salesforce sync fires before the order is confirmed.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The upstream fix was deprioritised (JIRA: DATA-3841). Removing this filter will cause double-counting.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(order_total) &lt;span style="color:#66d9ef">AS&lt;/span> lifetime_value,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATEDIFF(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#66d9ef">MIN&lt;/span>(order_date), &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> customer_age_days
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">&amp;gt;=&lt;/span> DATEADD(&lt;span style="color:#66d9ef">day&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">90&lt;/span>, &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> status_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">2&lt;/span>, &lt;span style="color:#ae81ff">9&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That comment block is five lines. It contains: the business rationale, the stakeholder who owns the decision, the date it was confirmed, the upstream system causing the problem, the known issue reference, and the consequence of removing the filter. None of that information is recoverable from the SQL. All of it is load-bearing.&lt;/p>
&lt;hr>
&lt;h3 id="the-deduplication-that-nobody-remembers-adding">The deduplication that nobody remembers adding&lt;/h3>
&lt;p>Have you seen this line before:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>QUALIFY ROW_NUMBER() OVER (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARTITION &lt;span style="color:#66d9ef">BY&lt;/span> customer_id, order_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> ingested_at &lt;span style="color:#66d9ef">DESC&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The SQL is completely transparent about what it does: take the most recently ingested row for each customer/order combination. What it cannot tell you is why duplicates exist in the first place, whether the source of those duplicates has been fixed, whether this deduplication logic is still necessary, or whether &amp;ldquo;most recently ingested&amp;rdquo; is actually the right tiebreaker for your use case.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Deduplicating on customer_id + order_id to handle duplicate webhook events from Eftpos.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Eftpos retries failed webhooks, which can fire the same order_created event multiple times.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Taking the latest ingested_at gives us the most recent event state (status, amount, etc.)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Revisit if we adopt Eftpos&amp;#39;s idempotency keys at the integration layer — this dedup may become redundant.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Background: this logic was added after the October 2024 incident (post-mortem in Confluence).
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>QUALIFY ROW_NUMBER() OVER (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARTITION &lt;span style="color:#66d9ef">BY&lt;/span> customer_id, order_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> ingested_at &lt;span style="color:#66d9ef">DESC&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The second version tells the reader everything they need to know to maintain this safely: the upstream cause, the logic behind the tiebreaker, when to reconsider it, and where to find the history. Without that comment, every future engineer who touches this code has to reverse-engineer context that no longer exists anywhere.&lt;/p>
&lt;hr>
&lt;h3 id="when-two-models-define-the-same-word-differently">When two models define the same word differently&lt;/h3>
&lt;p>This one doesn&amp;rsquo;t generate an error. It generates confusion at the worst possible moment.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- IMPORTANT: &amp;#39;churned&amp;#39; here means &amp;gt;90 days since last order.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This differs from mkt_customer_segments, which uses &amp;gt;60 days for winback campaign targeting.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Both definitions are intentional. Customer Success uses 90 days because cohort analysis
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- showed that 30% of &amp;#34;60-day churned&amp;#34; customers reorder organically within the next 30 days
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- and shouldn&amp;#39;t receive a discount. Marketing uses the stricter threshold to maximise winback opportunities.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The discrepancy is documented and known. Do not &amp;#34;fix&amp;#34; one to match the other.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> days_since_last_order &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">30&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;active&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> days_since_last_order &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">90&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;at_risk&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;churned&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> customer_segment
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Without that comment, someone eventually &amp;ldquo;cleans up&amp;rdquo; one of the two definitions because having different churn thresholds in two models looks like a bug. They&amp;rsquo;re wrong, and they&amp;rsquo;re about to break a winback campaign that&amp;rsquo;s been performing well for six months.&lt;/p>
&lt;p>The SQL cannot warn them. The comment can.&lt;/p>
&lt;hr>
&lt;h3 id="dbt-descriptions-arent-exempt">dbt descriptions aren&amp;rsquo;t exempt&lt;/h3>
&lt;p>dbt gives you a proper home for the &amp;ldquo;why&amp;rdquo;: model descriptions and column-level descriptions in your YAML.&lt;/p>
&lt;p>Bad:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">fct_orders&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;The orders fact table. Contains order data.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">columns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_segment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;The customer segment based on days since last order.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That description restates what the model name already says. It is a comment-shaped void.&lt;/p>
&lt;p>Good:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">fct_orders&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &amp;gt;&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Order-level fact table used as the single source of truth for revenue reporting.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Revenue excludes test accounts (prefixed &amp;#39;INTERNAL_&amp;#39;) and gift card redemptions,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> per the Finance revenue recognition policy agreed January 2025.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> See Confluence: Revenue Recognition Standards v3.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Do not use this model for marketing attribution — use mkt_attributed_revenue instead,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> which applies different channel logic.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">columns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_segment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &amp;gt;&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Lifecycle segment defined by the Customer Success team, Q1 2025.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Thresholds (30/90 days) were derived from cohort analysis showing a median
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> reorder window of 28 days. Intentionally differs from the Marketing segment
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> definition in mkt_customer_segments, which uses a 60-day churn threshold
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> for winback campaign purposes. Both are correct for their respective use cases.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That description does something a column name never can: it tells you who owns the definition, when it was set, what analysis underpins it, and — critically — where the definition intentionally diverges from a similar field elsewhere in the warehouse. That last part is the thing that saves someone three hours of confused Teams messages.&lt;/p>
&lt;hr>
&lt;h3 id="the-archaeology-problem">The archaeology problem&lt;/h3>
&lt;p>Data teams inherit codebases. It happens constantly, and it will keep happening. Engineers leave. Consultants finish their contracts. Reorgs shuffle ownership. What gets left behind is the SQL, and whatever context wasn&amp;rsquo;t written down is gone.&lt;/p>
&lt;p>Good SQL survives the handover. It&amp;rsquo;s readable, consistent, and correct. But &amp;ldquo;readable&amp;rdquo; in the sense of mechanically parseable is not the same as &amp;ldquo;understandable&amp;rdquo; in the sense of knowing what business problem the code was solving, whose decision it was, and what you&amp;rsquo;d need to change if that decision changed.&lt;/p>
&lt;p>Comments are how you write for the engineer who inherits this in two years. They&amp;rsquo;re how you write for yourself in six months. They&amp;rsquo;re how you ensure that a working pipeline stays working through the context collapse that happens every time anyone leaves a team.&lt;/p>
&lt;p>SQL can only tell you what. Try not to shortchange the people reading it in either direction.&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>SQL</category><category>dbt</category><category>Documentation</category><category>Data Quality</category><category>Code Comments</category><category>Data Pipelines</category><category>Best Practices</category></item><item><title>The Broken Window in Your Data Pipeline</title><link>https://ghostinthedata.info/posts/2026/2026-05-09-broken-window-theory/</link><pubDate>Sat, 09 May 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-05-09-broken-window-theory/</guid><author>Chris Hillman</author><description>A single ignored data quality issue doesn't stay local. In pipelines, broken windows travel — and by the time anyone notices, the damage is already downstream.</description><content:encoded>&lt;p>There&amp;rsquo;s a particular kind of data problem that doesn&amp;rsquo;t announce itself. It accumulates.&lt;/p>
&lt;p>We were receiving Salesforce data through delta extraction — sensible in theory, because full snapshots can run to hundreds of terabytes and less than 1% of records change on any given day. The problem is that deltas require someone to know what &amp;ldquo;changed&amp;rdquo; means. In Salesforce, that&amp;rsquo;s less obvious than it sounds. Watch a &lt;code>last_modified&lt;/code> column and you&amp;rsquo;ll miss objects that get updated when a related object changes, without their own timestamp reflecting it.&lt;/p>
&lt;p>Over time: drift. Orphan records. Data that &lt;em>looks&lt;/em> current because the record exists, but isn&amp;rsquo;t. The fix was documented — run a full snapshot periodically to correct the accumulated drift, painful as that was — and everyone with any context on the system knew it.&lt;/p>
&lt;p>What happened was roughly this: the drift accumulated silently until something downstream looked wrong. An investigation traced it back to the delta logic. The full snapshot was run. The problem was resolved. Notes were written. The workaround was filed away.&lt;/p>
&lt;p>And then, about twelve months later, the same conversation happened again.&lt;/p>
&lt;hr>
&lt;p>What made that moment stick wasn&amp;rsquo;t the technical failure. It was the recognition that the window had been broken for a while. Everybody who walked past it knew it was broken. We&amp;rsquo;d even put a note next to it.&lt;/p>
&lt;p>I&amp;rsquo;ve been that person — the one who wrote the documentation, filed the Jira ticket, and moved on. Which is probably why I remember the twelve-month cycle so clearly. And why I started paying attention to the pattern of it, across teams and companies, long after that particular incident.&lt;/p>
&lt;p>We just hadn&amp;rsquo;t fixed it.&lt;/p>
&lt;hr>
&lt;p>Bear with me here, because what I&amp;rsquo;m about to describe starts with an abandoned car in the Bronx in 1969 — and I promise it ends somewhere relevant to your dbt models.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="a-broken-window-in-the-bronx-a-broken-window-in-your-warehouse">A broken window in the Bronx, a broken window in your warehouse&lt;/h3>
&lt;/br>
&lt;p>A Stanford psychologist named Philip Zimbardo ran a strange experiment. He abandoned two cars — one in the Bronx, one in Palo Alto — and watched what happened.&lt;/p>
&lt;p>The Bronx car was stripped within twenty-four hours. Within three days it was completely gutted.
The Palo Alto car sat untouched for a week — until Zimbardo himself walked up with a sledgehammer and broke a window. Within hours, it had been stripped too.&lt;/p>
&lt;p>Same car. Different signal.&lt;/p>
&lt;p>Thirteen years later, criminologists James Q. Wilson and George Kelling built a theory on that experiment. If a window in a building is broken and left unrepaired, they argued, all the rest will soon follow. Not because there&amp;rsquo;s a particular breed of window-breaker lurking around, but because an unrepaired window sends a message: &lt;em>nobody here cares&lt;/em>. And once that signal is broadcast, breaking more windows costs nothing.&lt;/p>
&lt;p>The insight was semiotic, not structural. It wasn&amp;rsquo;t about windows. It was about what an unrepaired window communicates about the norms of a place.&lt;/p>
&lt;p>The theory eventually found its way into software. Andrew Hunt and David Thomas put it into &lt;em>The Pragmatic Programmer&lt;/em> almost verbatim: don&amp;rsquo;t leave broken windows — bad designs, wrong decisions, poor code — unrepaired. They&amp;rsquo;d watched clean systems deteriorate quickly once windows started breaking. The mechanism was the same. A developer looking at a messy codebase thinks: &lt;em>if someone else got away with being careless, maybe I can too.&lt;/em> The norm shifts. The entropy accelerates.&lt;/p>
&lt;p>Researchers at Empirical Software Engineering ran a controlled experiment — twenty-nine developers, codebases seeded with either high or low technical-debt density — and found exactly what Hunt and Thomas had intuited. Pre-existing debt measurably caused developers to introduce &lt;em>more&lt;/em> debt. Non-descriptive variable names. Duplicated logic instead of reuse. Additional code smells. The broken window was statistically contagious.&lt;/p>
&lt;p>It&amp;rsquo;s a compelling idea, and it maps well to software. But it doesn&amp;rsquo;t map perfectly to data engineering. And the gap between &amp;ldquo;maps well&amp;rdquo; and &amp;ldquo;maps perfectly&amp;rdquo; is where teams get into serious trouble.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-thing-thats-different-about-data">The thing that&amp;rsquo;s different about data&lt;/h3>
&lt;/br>
&lt;p>When a broken window appears in a codebase, it&amp;rsquo;s local. A messy function, an undocumented module, a class that&amp;rsquo;s doing five things at once — these are ugly, they invite imitation, and they slow future development. But they stay where they are. They don&amp;rsquo;t go anywhere.&lt;/p>
&lt;p>A broken window in a data pipeline doesn&amp;rsquo;t stay where it is.&lt;/p>
&lt;p>It travels.&lt;/p>
&lt;p>That&amp;rsquo;s the thing nobody really talks about when they apply broken-windows thinking to data work. In a pipeline, everything is connected. A schema drift in a source table doesn&amp;rsquo;t just make that table annoying to work with — it silently corrupts every model downstream that touches that field. Which means it corrupts every dashboard that uses those models. Which means it corrupts the metrics those dashboards expose. Which means it corrupts the business decisions made from those metrics.&lt;/p>
&lt;p>The broken window is in row one. By the time someone notices, the damage is in the boardroom.&lt;/p>
&lt;p>And unlike software, where the damage is visible — a stacktrace, a failing build, a crash — data damage is often invisible. The pipeline still runs. The dashboard still renders. The report still reconciles. The numbers just happen to be wrong, quietly, for reasons nobody can easily trace.&lt;/p>
&lt;p>Software bugs produce noise. Bad data produces silence.&lt;/p>
&lt;p>In software, the broken window signals disorder and invites imitation. In data pipelines, it does all of that &lt;em>and&lt;/em> it propagates at machine speed through every downstream system that trusts the upstream to be clean. By the time the propagation is discovered, it&amp;rsquo;s usually been underway for a while.&lt;/p>
&lt;p>This is the propagation problem. It&amp;rsquo;s not just that bad data begets more bad data. It&amp;rsquo;s that one bad window can contaminate an entire watershed.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="what-a-contaminated-watershed-actually-looks-like">What a contaminated watershed actually looks like&lt;/h3>
&lt;/br>
&lt;p>In May 2022, Unity Software — the game engine company — disclosed something remarkable in their SEC earnings filing. Their Audience Pinpointer ML model, the system that powered their ad targeting business, had ingested bad data from a large customer. That data had corrupted their training set.&lt;/p>
&lt;p>CEO John Riccitiello on the earnings call: &amp;ldquo;we lost the value of a portion of our training data due in part to us ingesting bad data from a large customer.&amp;rdquo;&lt;/p>
&lt;p>The bad data didn&amp;rsquo;t just produce one wrong prediction. It poisoned the model weights. Every subsequent retrain was built on a contaminated foundation. The model had to be taken offline, the bad data removed, and training restarted from scratch. The estimated impact was $110 million in revenue for 2022. The stock dropped 37% in a single day. Market cap losses in the billions.&lt;/p>
&lt;p>The broken window wasn&amp;rsquo;t in Unity&amp;rsquo;s systems — it was in data they were &lt;em>ingesting&lt;/em>. Once it crossed the boundary into their training pipeline, it propagated in the only direction data knows: forward and downstream, embedding itself into every layer of the system that touched it.&lt;/p>
&lt;p>Riccitiello&amp;rsquo;s pledge after the fact was almost poignant: &amp;ldquo;We are deploying monitoring, alerting and recovery systems and processes to promptly mitigate future events.&amp;rdquo; The observability came after the disaster. The window had already broken every other window in the building.&lt;/p>
&lt;hr>
&lt;p>These are the normal failure mode of connected data systems. The propagation isn&amp;rsquo;t a bug in the design — it&amp;rsquo;s inherent to the architecture. Data flows in one direction. Trust flows with it. The moment you have a pipeline, you have propagation risk.&lt;/p>
&lt;p>The question isn&amp;rsquo;t whether your broken windows will propagate. It&amp;rsquo;s how far they&amp;rsquo;ll travel before anyone notices.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-tools-we-built-to-tolerate-this">The tools we built to tolerate this&lt;/h3>
&lt;/br>
&lt;p>Modern data tooling has given us genuinely sophisticated ways to formally accept broken windows.&lt;/p>
&lt;p>dbt — the transformation tool most data engineering teams live inside — has a &lt;code>severity&lt;/code> configuration on data tests. You can set a test to &lt;code>warn&lt;/code> instead of &lt;code>error&lt;/code>. The test runs, detects a problem, and&amp;hellip; doesn&amp;rsquo;t fail the pipeline. It records the warning and moves on.&lt;/p>
&lt;p>The documentation is cheerful about it: &amp;ldquo;Maybe 1 duplicate record can count as a warning, but 10 duplicate records should count as an error.&amp;rdquo;&lt;/p>
&lt;p>In principle, this is sensible. In practice, the &lt;code>warn&lt;/code> threshold becomes a Schelling point. Teams configure tests to warn to avoid breaking CI. The warnings accumulate. And then — almost invariably — the warnings become wallpaper. &lt;em>(Go check your own test results right now. Count how many have been in warn state for more than two weeks. I&amp;rsquo;ll wait.)&lt;/em>&lt;/p>
&lt;p>dbt also has a feature called &lt;code>store_failures&lt;/code> that saves failed records to an audit schema table. The test is doing its job. The failures are being recorded. They overwrite the previous run&amp;rsquo;s failures on the next run. If nobody is actively querying that audit table — and almost nobody is — the failures exist only to make the test feel like it&amp;rsquo;s being taken seriously.&lt;/p>
&lt;p>It&amp;rsquo;s a passive graveyard. The window is monitored. Nobody fixes the glass.&lt;/p>
&lt;p>Airflow has an equivalent pattern. The &lt;code>soft_fail&lt;/code> parameter on sensors means that if an exception is raised — the source system is down, the file hasn&amp;rsquo;t arrived — the task is marked as &lt;em>skipped&lt;/em> rather than &lt;em>failed&lt;/em>. Downstream tasks, depending on their trigger rules, may also skip. An entire branch of your DAG quietly collapses to a skipped state, which most pipelines treat as benign, and your stakeholders get a dashboard with no data in it instead of an error message.&lt;/p>
&lt;p>Retries do something similar. A flaky source that fails 30% of the time gets &lt;code>retries=3&lt;/code> configured. The task eventually succeeds on the third attempt. The 30% failure rate never surfaces as an anomaly in any meaningful way. Until the source dies entirely, at which point the symptom everyone responds to is not &amp;ldquo;this has been flaky for six months&amp;rdquo; but &amp;ldquo;this suddenly started failing today.&amp;rdquo;&lt;/p>
&lt;p>None of this is the fault of the tools. dbt and Airflow are doing what they&amp;rsquo;re designed to do. The issue is that the default ergonomics of both make &lt;em>tolerating&lt;/em> failure significantly easier than &lt;em>stopping propagation&lt;/em>. &amp;ldquo;Don&amp;rsquo;t break the build&amp;rdquo; is a more convenient goal than &amp;ldquo;don&amp;rsquo;t ship broken data downstream.&amp;rdquo; The tools give you excellent knobs for the former and adequate knobs for the latter.&lt;/p>
&lt;p>Chad Sanderson — formerly at Convoy, now building in the data contracts space — has a name for what this produces over time. He calls it the POSIWID principle: the Purpose Of a System Is What It Does. If your data pipelines are consistently producing low-quality data, and your team consistently tolerates that, then whatever you think the purpose of your data platform is, its actual purpose is to enable teams to move fast, ship without accountability, and tolerate breakages.&lt;/p>
&lt;p>The broken windows aren&amp;rsquo;t exceptions. They&amp;rsquo;re the product.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="weve-built-monitoring-why-isnt-it-working">We&amp;rsquo;ve built monitoring. Why isn&amp;rsquo;t it working?&lt;/h3>
&lt;/br>
&lt;p>Data observability emerged as a formal discipline. The concept — largely popularised by Barr Moses at Monte Carlo — drew directly from the software world&amp;rsquo;s SRE and distributed systems monitoring practices: freshness, distribution, volume, schema, lineage. Five pillars. Automated anomaly detection. Alerts in Slack when something looks wrong.&lt;/p>
&lt;p>The framing was right. The problem is what happened next.&lt;/p>
&lt;p>Monte Carlo&amp;rsquo;s own telemetry — across millions of monitored tables — shows that alert engagement drops 15% when a Slack channel exceeds 50 alerts per week, and a further 20% past 100 per week. The data observability system has a data quality problem: the signal-to-noise ratio degrades, and so does the human response.&lt;/p>
&lt;p>This isn&amp;rsquo;t specific to data. The clinical literature on alarm fatigue is sobering. ICU environments average more than 150 alarms per bed per day. Studies consistently find that 72% to 99% of those alarms are non-actionable. The clinical community&amp;rsquo;s response to this was institutional — the Joint Commission made alarm safety a National Patient Safety Goal in 2014 — because they recognised that a monitoring system that produces more noise than signal doesn&amp;rsquo;t just fail to help. It actively worsens outcomes by training humans to stop responding.&lt;/p>
&lt;p>Cybersecurity teams face the same thing. Security operations centres receive thousands of alerts daily; most go unaddressed, not because analysts are lazy, but because the ratio of genuine signals to false positives has degraded to the point where sustained attention is cognitively impossible.&lt;/p>
&lt;p>The Google SRE book is direct on this: &amp;ldquo;Every page should be actionable. If a page merely merits a robotic response, it shouldn&amp;rsquo;t be a page.&amp;rdquo;&lt;/p>
&lt;p>In data engineering, the equivalent of a &amp;ldquo;robotic response&amp;rdquo; is the acknowledged-and-unresolved alert. The monitor that fires every Tuesday morning, gets a thumbs up in Teams, gets added to the &amp;ldquo;known issues&amp;rdquo; document, and never gets fixed. The window is now monitored. The fact that it&amp;rsquo;s monitored makes the team feel responsible. The window stays broken.&lt;/p>
&lt;p>There&amp;rsquo;s a term for this: observability theatre. The infrastructure of visibility exists. The dashboards are green. Nobody&amp;rsquo;s actually looking at the glass.&lt;/p>
&lt;p>This is where the broken windows metaphor earns its keep most fully. Wilson and Kelling weren&amp;rsquo;t saying that disorder is bad because it looks bad. They were saying that an unrepaired broken window sends a signal that no one cares — and that signal is the actual mechanism of decay. Monitoring a broken window without repairing it sends exactly the same signal. Possibly a worse one, because now everyone knows the problem is being tracked and nobody&amp;rsquo;s acting on it. The norm becomes: acknowledged problems are not necessarily fixed problems.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-stakeholder-sees-it-differently">The stakeholder sees it differently&lt;/h3>
&lt;/br>
&lt;p>Here is the asymmetry that data teams consistently underestimate.&lt;/p>
&lt;p>When an engineer looks at a broken window in a data pipeline — a column with a known null rate, a test that&amp;rsquo;s been in warn state for three weeks, a DAG that soft-fails every second run — they see technical debt. A problem to eventually be addressed. Something that exists on a spectrum of severity, probably not at the top of the list.&lt;/p>
&lt;p>When a business stakeholder encounters the downstream consequence of that broken window — a dashboard that contradicts a number they just presented to the CFO, a metric that moved in an unexplained direction, a report that doesn&amp;rsquo;t reconcile with another report — they don&amp;rsquo;t think &amp;ldquo;technical debt.&amp;rdquo; They think: &lt;em>can I trust the data team?&lt;/em>&lt;/p>
&lt;p>Benn Stancil — one of the clearest thinkers writing about this — put it well: &amp;ldquo;Trust is built, and blown up, by the outputs — and specifically, the consistency of those outputs.&amp;rdquo; He uses a vivid image that I keep coming back to: &amp;ldquo;Data and the dashboards that display it create a shared sense of reality. Looking at two dashboards that don&amp;rsquo;t match is like looking out two adjacent windows and not seeing the same thing.&amp;rdquo;&lt;/p>
&lt;p>That&amp;rsquo;s the experience of broken-window propagation from the stakeholder&amp;rsquo;s side. Two adjacent windows. Different views. A reality that doesn&amp;rsquo;t cohere.&lt;/p>
&lt;p>Monte Carlo&amp;rsquo;s 2023 State of Data Quality research put a number to the trust inversion that most data engineers already sense. In 2022, 47% of respondents reported that business stakeholders identified data issues first &amp;ldquo;all or most of the time&amp;rdquo; — more often than the data team itself. By 2023, that figure had risen to 74%. &lt;em>(Three in four. Let that land.)&lt;/em>&lt;/p>
&lt;p>Think about what that means. In three out of four data incidents, the people who rely on the data found the problem before the people who built it. The pipeline runs, the test passes, the dashboard renders, and somewhere downstream a business analyst is staring at a number that doesn&amp;rsquo;t look right and is about to send a message that starts with: &amp;ldquo;Quick question about this figure&amp;hellip;&amp;rdquo;&lt;/p>
&lt;p>The damage isn&amp;rsquo;t technical. It&amp;rsquo;s relational. And it compounds in a way that&amp;rsquo;s harder to reverse than any schema migration.&lt;/p>
&lt;p>Thomas Redman — who has spent decades studying data quality — made this point in Harvard Business Review: &amp;ldquo;When data are unreliable, managers quickly lose faith in them and fall back on their intuition to make decisions.&amp;rdquo; Once that happens, you haven&amp;rsquo;t just produced bad data. You&amp;rsquo;ve trained decision-makers to ignore good data too, because they can no longer distinguish between the two.&lt;/p>
&lt;p>The broken window didn&amp;rsquo;t just propagate through the pipeline. It propagated into the culture.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="nobody-owns-the-broken-window">Nobody owns the broken window&lt;/h3>
&lt;/br>
&lt;p>There&amp;rsquo;s a specific pattern worth naming, because it&amp;rsquo;s where most data teams I&amp;rsquo;ve worked with have the most unacknowledged broken windows: the orphaned asset.&lt;/p>
&lt;p>The pipeline that was built by an engineer who left eighteen months ago. The table in the warehouse that exists in the data catalogue but whose owner field says &amp;ldquo;Unknown&amp;rdquo; or, worse, the name of someone who&amp;rsquo;s no longer at the company. The dbt model that runs in production, that feeds two dashboards, that nobody on the current team can fully explain.&lt;/p>
&lt;p>These aren&amp;rsquo;t broken in any obvious sense. They run. They load. The tests pass, or they&amp;rsquo;re set to warn. But they&amp;rsquo;re broken windows in the original sense: they signal that nobody cares. And because nobody owns them, nobody&amp;rsquo;s in a position to repair them even when something goes wrong.&lt;/p>
&lt;p>What typically happens is this: a new engineer joins the team. They need to understand how the data flows. They look at the catalogue. They look at the undocumented table. They look at the model that references it. They look at the Jira ticket from fourteen months ago that says &amp;ldquo;Known issue — downstream teams aware.&amp;rdquo; And they make a rational decision: don&amp;rsquo;t touch it, build around it, replicate it if necessary.&lt;/p>
&lt;p>The broken window has now inspired a new window. Same mechanism as the Bronx car. Different materials.&lt;/p>
&lt;p>The organisational research on this is unambiguous. Ron Westrum&amp;rsquo;s typology of organisational cultures — validated empirically by the DORA research programme across thousands of software teams — found that information flow predicts safety and performance more reliably than almost any structural variable. In pathological cultures, information is hoarded or withheld for political reasons. In generative cultures, information flows freely, failures are shared, and problems get fixed because surfacing problems is rewarded rather than penalised.&lt;/p>
&lt;p>Amy Edmondson&amp;rsquo;s research on psychological safety adds the other half: 85% of employees have withheld important information from their manager due to fear of speaking up. In data teams, this looks like: the junior engineer who noticed the null rate had been climbing for two weeks and didn&amp;rsquo;t raise it because they weren&amp;rsquo;t sure it was their call to make. The analyst who suspected the metric definition had drifted but didn&amp;rsquo;t want to slow down the dashboard delivery. The data engineer who knew the pipeline was flaky but figured someone more senior would have noticed if it really mattered.&lt;/p>
&lt;p>The broken window gets left unrepaired not because nobody sees it, but because the culture hasn&amp;rsquo;t made repair feel safe or worthwhile.&lt;/p>
&lt;p>Edmondson on this: &amp;ldquo;If there&amp;rsquo;s no bad news, remind yourself: It&amp;rsquo;s not that it&amp;rsquo;s not there. It&amp;rsquo;s that you&amp;rsquo;re not hearing about it.&amp;rdquo;&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-original-theorys-mistake--and-ours">The original theory&amp;rsquo;s mistake — and ours&lt;/h3>
&lt;/br>
&lt;p>Before we get to what to actually do about this, it&amp;rsquo;s worth pausing on where the original theory went wrong. Because data engineering is at risk of making exactly the same mistake.&lt;/p>
&lt;p>When Wilson and Kelling published their 1982 Atlantic article, the theory was nuanced. It was about signals and norms. It was explicitly a theory about community — about how residents and police could work together to maintain shared standards in public spaces. It was, in Kelling&amp;rsquo;s own framing, a theory of collective efficacy.&lt;/p>
&lt;p>What cities did with it was zero tolerance. Mass arrests for minor offences. Stop-and-frisk. 685,000 stops in New York City in a single year, more than 85% of them finding nothing at all.&lt;/p>
&lt;p>Kelling&amp;rsquo;s reaction, when he learned how comprehensively his theory had been misapplied: &amp;ldquo;Oh, shit.&amp;rdquo;&lt;/p>
&lt;p>The software world made a version of the same mistake when it imported broken windows thinking. &amp;ldquo;Don&amp;rsquo;t live with broken windows&amp;rdquo; became, for some teams, a linting rule for everything, a zero-tolerance policy for code smells, a culture of perfectionism that burned people out and produced beautiful codebases that never shipped.&lt;/p>
&lt;p>Adam Tornhill&amp;rsquo;s research at CodeScene offers the corrective: not all broken windows matter equally. In a 400,000-line codebase with 89 developers, his analysis found that 4% of the code was responsible for 72% of the defects. The broken windows that needed fixing weren&amp;rsquo;t distributed evenly. They clustered in hotspots — files that were both frequently changed and highly complex. Fix the hotspots. Let the cold, ugly, stable corner of the codebase sit.&lt;/p>
&lt;p>The principle for data engineering is the same. Zero-tolerance data quality enforcement — failing every pipeline on every test at severity error — produces fragile systems and tired teams. It&amp;rsquo;s the data equivalent of arresting everyone for jaywalking. The signal gets lost in the noise.&lt;/p>
&lt;p>What the original theory was actually pointing at was something more like: maintain the norm. Make it visible that people care. Ensure that the broken windows that matter — the ones that propagate, the ones in the high-traffic, high-trust parts of the system — get fixed promptly and publicly. Not every window. The ones that signal whether anyone&amp;rsquo;s paying attention.&lt;/p>
&lt;hr>
&lt;h3 id="what-collective-efficacy-looks-like-in-a-data-team">What collective efficacy looks like in a data team&lt;/h3>
&lt;/br>
&lt;p>Sampson, Raudenbush, and Earls — the Chicago sociologists who ran the most rigorous empirical test of broken windows theory — found that the variable that actually predicted neighbourhood safety wasn&amp;rsquo;t the presence or absence of disorder. It was &lt;em>collective efficacy&lt;/em>: the combination of social cohesion and shared willingness to intervene. Neighbours who knew each other, trusted each other, and were willing to act on behalf of each other&amp;rsquo;s wellbeing.&lt;/p>
&lt;p>The policy prescription that emerges from their work is very different from zero tolerance. It&amp;rsquo;s community investment. Shared ownership. Making the costs of non-intervention visible.&lt;/p>
&lt;p>The equivalent in data engineering starts with one thing, and if you do nothing else in this list, do this one:&lt;/p>
&lt;p>&lt;strong>Make propagation visible before anything else.&lt;/strong>&lt;/p>
&lt;p>Column-level data lineage — now available in most modern data observability platforms and increasingly in the open-source OpenLineage standard — lets you answer the question: if this column is wrong, what does it break? That question should be answerable in seconds, not hours. Teams that can visualise propagation chains respond to broken windows faster because they can see the radius of the damage before they decide whether to act.&lt;/p>
&lt;p>This matters beyond incident response. When you can show an engineer that the null column they&amp;rsquo;re tolerating in a staging model feeds seven downstream gold-layer tables, three dashboards, and a Snowflake share that two other teams consume — the calculus on whether to fix it changes. The broken window stops being an abstract code quality concern and becomes a blast radius. That&amp;rsquo;s a much more compelling argument for repair than &amp;ldquo;we should improve our data quality culture.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Treat ownership as load-bearing infrastructure, not housekeeping.&lt;/strong>&lt;/p>
&lt;p>Every pipeline, every table, every model should have a named owner. Not a team. A person. The Jira ticket that says &amp;ldquo;Known issue — downstream teams aware&amp;rdquo; with no assigned owner is the data equivalent of a broken window with an orange cone next to it. The cone acknowledges the hazard. Nobody&amp;rsquo;s fixing the glass.&lt;/p>
&lt;p>The counterargument is always resourcing — &amp;ldquo;we don&amp;rsquo;t have time to own everything properly.&amp;rdquo; That&amp;rsquo;s true, and worth taking seriously. But the right response isn&amp;rsquo;t to pretend you own things you don&amp;rsquo;t. It&amp;rsquo;s to make orphaned assets visible and have an honest conversation about whether the organisation can afford to run production pipelines with no accountable maintainer. Most of the time, when the question is asked that directly, the answer is no.&lt;/p>
&lt;p>&lt;strong>Calibrate your tolerance patterns deliberately — and actually revisit them.&lt;/strong>&lt;/p>
&lt;p>The dbt &lt;code>severity: warn&lt;/code> setting and Airflow&amp;rsquo;s &lt;code>soft_fail&lt;/code> are legitimate tools when they&amp;rsquo;re conscious decisions: &amp;ldquo;this condition is a signal worth tracking but not a pipeline-stopper, and here&amp;rsquo;s the threshold at which that changes.&amp;rdquo; The problem is that almost nobody uses them that way. They&amp;rsquo;re the path of least resistance to avoid a broken CI build — and six months later you audit your test results and discover forty tests set to warn that haven&amp;rsquo;t been at zero failures since the day they were written.&lt;/p>
&lt;p>Treat warn-severity tests the way you treat Jira tickets that never get triaged. Set a review cadence. If a test has been consistently warning for more than two weeks without an associated investigation, it&amp;rsquo;s either a bug that needs fixing or a threshold that needs changing. &amp;ldquo;Known issue&amp;rdquo; is not a status. It&amp;rsquo;s an admission.&lt;/p>
&lt;p>&lt;strong>Fix alert fatigue before it hollows out your monitoring culture.&lt;/strong>&lt;/p>
&lt;p>The Teams channel where every data quality alert lands is the digital equivalent of a neighbourhood where every broken window gets photographed and logged and nobody ever fixes one. The log is evidence that someone noticed. It&amp;rsquo;s not evidence that anyone will act.&lt;/p>
&lt;p>Set alert thresholds that produce actionable signals. The Google SRE principle applies directly: if an alert merits a robotic response, it shouldn&amp;rsquo;t be an alert. If your first instinct on seeing a particular monitor fire is to click acknowledge and move on, that monitor is producing noise, not signal. Change it or delete it. Grouping alerts by lineage — &amp;ldquo;these five monitors fired because of one upstream schema change&amp;rdquo; rather than five separate pings — reduces volume while making propagation visible at the moment of failure, which is exactly when you want it.&lt;/p>
&lt;p>&lt;strong>Make broken windows visible to leadership, because right now the cost is invisible.&lt;/strong>&lt;/p>
&lt;p>Data quality work is famously hard to demonstrate. Pipelines that run cleanly leave no artefact. A well-maintained model with a 0% null rate looks identical to a freshly built one. There&amp;rsquo;s no natural demo format for &amp;ldquo;nothing bad happened this week.&amp;rdquo;&lt;/p>
&lt;p>This invisibility is part of why broken windows accumulate. The cost of prevention is hidden in maintenance time that doesn&amp;rsquo;t get counted. The cost of failure gets absorbed by analysts who spend their Tuesdays reconciling numbers instead of answering strategic questions, by data engineers who spend their Fridays investigating stakeholder tickets, by business decisions made on incorrect information that can&amp;rsquo;t be traced back to a specific incident.&lt;/p>
&lt;p>Quantify the propagation radius when incidents do occur. How many downstream models were affected? How many stakeholders were exposed to incorrect data, and for how long? What was the resolution effort in hours? Those numbers, tracked consistently, build a case for investment that abstract arguments about data quality never will.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-window-was-broken-before-i-noticed">The window was broken before I noticed&lt;/h3>
&lt;/br>
&lt;p>What we ended up doing was building the fix into the operating rhythm. A scheduled data drift catch-up: a full snapshot weekly, or monthly, depending on the volatility of the Salesforce object — not to respond to an incident, but to systematically correct the accumulated drift before it became visible to anyone downstream. We stopped waiting for the escalation. We made the repair part of the architecture.&lt;/p>
&lt;p>The broken window was still there, technically. The delta logic still had its blind spots around hidden relationships. But we stopped waiting for it to propagate before we addressed it. Some broken windows aren&amp;rsquo;t things you permanently seal — they&amp;rsquo;re things you build a maintenance routine around. Knowing that is the fix.&lt;/p>
&lt;p>What stayed with me wasn&amp;rsquo;t the technical solution, which was straightforward enough once we committed to it. What stayed with me was the calendar predictability of the escalation cycle before the fix. The fact that we had a known issue, a known workaround, documentation of both — and still managed to have the same downstream discovery, the same investigation, the same remediation conversation, almost exactly twelve months apart.&lt;/p>
&lt;p>The window wasn&amp;rsquo;t hidden. It was visible in the delta logs if you knew where to look. We just hadn&amp;rsquo;t made it someone&amp;rsquo;s job to look, and we hadn&amp;rsquo;t made repair part of the routine. We&amp;rsquo;d fixed the glass, filed the paperwork, and assumed the problem was solved. Until the same signal appeared in the same downstream reports, a year later.&lt;/p>
&lt;hr>
&lt;p>What I keep coming back to, though, is something that was true of that experience and is true of most of the data quality failures I&amp;rsquo;ve seen since: the window wasn&amp;rsquo;t broken secretly. It wasn&amp;rsquo;t hidden. It was visible, it was acknowledged, and it was left.&lt;/p>
&lt;p>We had the monitoring. We had the test. We had the Jira ticket.&lt;/p>
&lt;p>We had, in other words, all the infrastructure of concern. What we didn&amp;rsquo;t have was the collective agreement that repair mattered — that the propagation radius of that one broken window was large enough, and trust-eroding enough, that it justified stopping what we were doing and fixing the glass.&lt;/p>
&lt;p>That&amp;rsquo;s the thing broken windows theory is actually about, underneath all the criminology and the code smells and the schema drift. It&amp;rsquo;s about the signal that unrepaired damage sends. Not to the criminals or the developers or the data consumers. To the people who are supposed to care about the system.&lt;/p>
&lt;p>When a broken window sits long enough in a data pipeline, it stops being a problem and starts being a norm. The new engineer doesn&amp;rsquo;t flag it — they work around it. The analyst doesn&amp;rsquo;t escalate it — they add a caveat to their report. The data engineer doesn&amp;rsquo;t fix it — they document it.&lt;/p>
&lt;p>And somewhere downstream, a business decision gets made on numbers that were broken before anyone thought to check.&lt;/p>
&lt;p>The window isn&amp;rsquo;t just in the pipeline. The window is in the standard you&amp;rsquo;re willing to keep.&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Quality</category><category>Data Pipelines</category><category>Technical Debt</category><category>Data Observability</category><category>dbt</category><category>Apache Airflow</category><category>Team Culture</category><category>Pipeline Architecture</category></item><item><title>The Data Quality Test: 10 Questions That Predict Pipeline Disasters</title><link>https://ghostinthedata.info/posts/2026/2026-04-11-questions-that-predict-pipeline-disasters/</link><pubDate>Sat, 11 Apr 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-04-11-questions-that-predict-pipeline-disasters/</guid><author>Chris Hillman</author><description>A pre-mortem checklist for any new data source. Before you ingest that new source system, ask these 10 questions — and learn what good and amazing answers look like for each one.</description><content:encoded>&lt;p>I&amp;rsquo;ve been writing about data quality a lot lately.&lt;/p>
&lt;p>Enough that I notice myself doing it. Enough that a small voice says: &lt;em>haven&amp;rsquo;t you made this point already?&lt;/em> Schema drift, NULL propagation, duplicate records, the whole catalogue of things that go wrong in the space between a source system and a warehouse. I keep circling back to it.&lt;/p>
&lt;p>And every time, I almost talk myself out of writing the piece. Then I reflect on what&amp;rsquo;s happened in the last few years of work. The postmortems I&amp;rsquo;ve read, the pipelines I&amp;rsquo;ve inherited — and the same pattern shows up with depressing regularity. Not exotic failures. Not edge cases. The boring stuff. The questions nobody asked before the first row hit the warehouse.&lt;/p>
&lt;p>That&amp;rsquo;s why I keep writing about this.&lt;/p>
&lt;p>Not because data quality is a trendy topic. Not because it makes for dramatic storytelling. Because I believe, with the kind of quiet conviction that comes from watching the same preventable disaster play out across different companies and different years, that &lt;strong>the measure of a data engineer isn&amp;rsquo;t how fast they respond to incidents — it&amp;rsquo;s whether those incidents needed to happen at all.&lt;/strong>&lt;/p>
&lt;p>Prevention isn&amp;rsquo;t glamorous. Nobody gets a shoutout in the all-hands for the outage that didn&amp;rsquo;t occur. There&amp;rsquo;s no adrenaline rush from a pipeline that just&amp;hellip; works. But there&amp;rsquo;s something else — something I&amp;rsquo;ve come to value more than any firefighting war story. It&amp;rsquo;s the feeling of walking into a Monday morning, seeing that a source system changed its schema over the weekend, and watching your pipeline handle it exactly the way you designed it to. No pages. No panic. No scramble. Just a notification in Teams that says: &lt;em>schema change detected in orders table, quarantined for review.&lt;/em>&lt;/p>
&lt;p>That feeling is what I want more data engineers to experience. And the path to it isn&amp;rsquo;t complicated. It starts with asking the right questions before you build anything.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-pre-mortem-nobody-wants-to-do">The pre-mortem nobody wants to do&lt;/h3>
&lt;br>
&lt;p>I have inherited alot of data pipelines over time: the ones that blow up spectacularly are almost never the ones with exotic technical problems. They&amp;rsquo;re the ones where somebody skipped the boring questions.&lt;/p>
&lt;p>Not the &amp;ldquo;what framework should we use&amp;rdquo; questions. Not the &amp;ldquo;should we go with Spark or DuckDB&amp;rdquo; questions. The uncomfortable, mundane questions that feel like they&amp;rsquo;re slowing you down when there&amp;rsquo;s pressure to ship.&lt;/p>
&lt;p>I&amp;rsquo;ve distilled these into ten questions. Ten things you should be able to answer about every single source system before a single row hits your warehouse. For each one, I&amp;rsquo;ll walk you through what a &lt;em>good&lt;/em> answer looks like — the baseline that keeps you out of trouble — and what an &lt;em>amazing&lt;/em> answer looks like — the kind of engineering that lets you sleep at night.&lt;/p>
&lt;p>If you can&amp;rsquo;t answer these questions, you&amp;rsquo;re not building a pipeline. You&amp;rsquo;re building a time bomb with a variable-length fuse.&lt;/p>
&lt;hr>
&lt;h3 id="1-what-happens-when-the-schema-changes-without-warning">1. What happens when the schema changes without warning?&lt;/h3>
&lt;br>
&lt;p>This is the one that gets everyone eventually. Not if — when.&lt;/p>
&lt;p>Source systems change. Application developers add columns, rename fields, change data types, and refactor table structures. They do this because it&amp;rsquo;s their job. And in most organisations, they have absolutely no obligation to tell you about it first. Your pipeline wakes up one morning expecting a &lt;code>VARCHAR&lt;/code> and gets handed an &lt;code>INTEGER&lt;/code>, and suddenly you&amp;rsquo;re in an incident channel explaining why the dashboard is blank.&lt;/p>
&lt;p>Research from Integrate.io found that production incidents increase by roughly 27% for each percentage point increase in schema drift incidents. That&amp;rsquo;s a compounding reliability problem — each unhandled change makes the next one more likely to cause real damage.&lt;/p>
&lt;p>The dangerous part isn&amp;rsquo;t the changes that cause immediate failures. It&amp;rsquo;s the silent ones. A column gets renamed, and your pipeline continues running because it doesn&amp;rsquo;t reference that column directly — but a downstream model does. A data type widens from &lt;code>INT&lt;/code> to &lt;code>BIGINT&lt;/code>, and everything works fine until an aggregation overflows six months later. A field that was never NULL suddenly starts arriving NULL, and your &lt;code>NOT NULL&lt;/code> constraint in the warehouse? You never actually enforced it.&lt;/p>
&lt;h4 id="what-good-looks-like">What good looks like&lt;/h4>
&lt;p>A good team validates incoming schemas before loading. At minimum, you&amp;rsquo;re comparing expected versus actual column names, data types, and nullability on every run. When something changes, the pipeline halts gracefully and sends a clear alert — not a cryptic stack trace, but a human-readable message that says &amp;ldquo;the &lt;code>user_status&lt;/code> column changed from VARCHAR(20) to VARCHAR(50) in the orders table.&amp;rdquo;&lt;/p>
&lt;p>You&amp;rsquo;re also versioning your schema expectations somewhere. Whether that&amp;rsquo;s a schema registry, a YAML file in version control, or even a documented contract in Confluence — the point is you&amp;rsquo;ve written down what you expect, so deviations are detectable.&lt;/p>
&lt;p>Good teams also quarantine unexpected changes rather than silently ingesting them. If a new column appears, it goes into a staging area or metadata log — not straight into the warehouse where it might break a &lt;code>SELECT *&lt;/code> somewhere downstream.&lt;/p>
&lt;h4 id="what-amazing-looks-like">What amazing looks like&lt;/h4>
&lt;p>If that was good, what is amazing? Well, teams that have schema contracts with source system owners. Not informal agreements — actual documented contracts that specify: these are the columns, these are the types, these are the nullability constraints, and here&amp;rsquo;s the process for changing any of them.&lt;/p>
&lt;p>Amazing means automated schema evolution handling. When a new non-breaking column appears, the pipeline can automatically accommodate it — adding the column to the target table, logging the change, and notifying the team — without human intervention. When a breaking change arrives (column removed, type narrowed, key structure changed), it&amp;rsquo;s automatically classified by severity and routed to the right person.&lt;/p>
&lt;p>Amazing also means your CI/CD pipeline includes schema validation. Before any source system deployment goes live, there&amp;rsquo;s an automated check that compares the proposed schema against all downstream consumers. The application team can&amp;rsquo;t deploy a breaking change without explicitly acknowledging the impact.&lt;/p>
&lt;p>Think of it this way: good catches the fire. Amazing prevents the spark.&lt;/p>
&lt;hr>
&lt;h3 id="2-what-happens-when-yesterdays-data-arrives-tomorrow">2. What happens when yesterday&amp;rsquo;s data arrives tomorrow?&lt;/h3>
&lt;br>
&lt;p>Late-arriving data is one of those problems that sounds simple until you actually try to solve it. A mobile app loses connectivity and batches up events. A partner system has an outage and replays three days of data on Monday morning. An IoT sensor comes back online after a firmware update and dumps a week&amp;rsquo;s worth of readings.&lt;/p>
&lt;p>Your pipeline ran for yesterday. It produced correct results based on the data available. Now more data shows up that belongs to yesterday. What happens?&lt;/p>
&lt;p>If your answer is &amp;ldquo;we&amp;rsquo;d just rerun the pipeline,&amp;rdquo; congratulations — you&amp;rsquo;ve just introduced a different set of problems. Did that rerun overwrite aggregations? Did downstream dashboards update? Did anyone notice the numbers changed?&lt;/p>
&lt;p>Late-arriving data is especially painful in dimensional modelling. The Kimball methodology specifically warns about late-arriving dimensions — facts that arrive before their corresponding dimension records exist. You end up with orphan foreign keys pointing at nothing, or worse, pointing at a placeholder &amp;ldquo;Unknown&amp;rdquo; record that never gets reconciled.&lt;/p>
&lt;h4 id="what-good-looks-like-1">What good looks like&lt;/h4>
&lt;p>Good teams have defined a tolerance window for late data. You know, explicitly, that your pipeline handles data arriving up to N hours late, and you&amp;rsquo;ve designed your partitioning and processing logic around that window.&lt;/p>
&lt;p>The most common pattern is a lookback window — your daily pipeline doesn&amp;rsquo;t just process today&amp;rsquo;s partition, it reprocesses the last N days to catch any late arrivals. It&amp;rsquo;s simple, it&amp;rsquo;s battle-tested, and for most batch workloads it&amp;rsquo;s perfectly adequate. You pair this with event-time partitioning (using the time the event actually occurred, not when it arrived) so that late data lands in the correct partition.&lt;/p>
&lt;p>Good also means you have observability on late data volumes. You&amp;rsquo;re tracking how much data arrives outside your expected window, and you&amp;rsquo;re alerting when that percentage exceeds a threshold. If 0.1% of your data normally arrives late and suddenly it&amp;rsquo;s 15%, something upstream has gone wrong.&lt;/p>
&lt;h4 id="what-amazing-looks-like-1">What amazing looks like&lt;/h4>
&lt;p>Amazing teams implement bi-temporal modelling. Every record carries two timestamps: when the event happened (event time) and when the system recorded it (processing time). This means you can always reconstruct what you knew at any point in time, and you can always find the data that arrived after a particular processing run.&lt;/p>
&lt;p>Amazing means automated reconciliation. When late data arrives, the system detects it, identifies affected downstream aggregations and models, and triggers targeted reprocessing — not a full pipeline rerun, but surgical updates to just the affected partitions and metrics. Databricks published a reconciliation pattern where unjoined records (facts without matching dimensions) get written to an error table, automatically retried on a schedule, and merged back into the target table once the dimension arrives.&lt;/p>
&lt;p>Amazing also means your SLAs explicitly account for late data. Your stakeholders know that the 7 AM dashboard reflects data up to midnight, with a reconciliation pass at 10 AM that catches stragglers. No surprises, no &amp;ldquo;the numbers changed&amp;rdquo; panic.&lt;/p>
&lt;hr>
&lt;h3 id="3-what-happens-when-a-primary-key-gets-reused">3. What happens when a primary key gets reused?&lt;/h3>
&lt;br>
&lt;p>This is more common than most engineers realise. Source systems recycle keys. It happens during migrations, database resets, test-to-production bleed, or when an auto-increment counter gets reset. I&amp;rsquo;ve seen it happen when a company acquired another business and merged customer databases — both systems had &lt;code>customer_id = 1&lt;/code>, and they meant completely different people.&lt;/p>
&lt;p>The problem is insidious because your pipeline has no way to know that &lt;code>customer_id = 47382&lt;/code> in today&amp;rsquo;s extract refers to a different entity than &lt;code>customer_id = 47382&lt;/code> from two years ago. Your MERGE statement happily updates the old record with the new data. Your historical analyses silently corrupt.&lt;/p>
&lt;h4 id="what-good-looks-like-2">What good looks like&lt;/h4>
&lt;p>Good teams never use source system keys (business keys) as their warehouse primary keys. Full stop. You generate surrogate keys in the warehouse — monotonically increasing integers that are completely independent of whatever identifier the source system uses. The source key gets stored as a natural key attribute, but it&amp;rsquo;s not what your fact tables join on.&lt;/p>
&lt;p>Good also means you have uniqueness monitoring. You&amp;rsquo;re checking that natural key + effective date combinations are unique, and you&amp;rsquo;re alerting when they&amp;rsquo;re not. If the same &lt;code>customer_id&lt;/code> appears twice in a single extract with different attributes, your pipeline flags it rather than silently picking one.&lt;/p>
&lt;p>You&amp;rsquo;re also maintaining a mapping table — source system key to warehouse surrogate key — with effective date ranges. When a key gets reused, the old mapping gets closed and a new one opens.&lt;/p>
&lt;h4 id="what-amazing-looks-like-2">What amazing looks like&lt;/h4>
&lt;p>Amazing teams establish immutable business keys that are independent of any source system. Instead of relying on &lt;code>customer_id&lt;/code> from the CRM, you create a composite business key that combines system identifier, source system name, and the natural key — something like &lt;code>CRM|customer_id|47382&lt;/code>. This makes key reuse impossible by construction, because the key includes its origin.&lt;/p>
&lt;p>Amazing means your data contracts specify key semantics. The source team has explicitly documented whether keys are permanent, recyclable, or sequential. Your pipeline&amp;rsquo;s handling logic adapts based on these documented properties.&lt;/p>
&lt;p>Amazing also means you have automated key collision detection. When a key arrives that previously belonged to a different entity (detected by comparing against historical attributes), the pipeline automatically creates a new entity, generates a new surrogate key, and flags the collision for review — all without human intervention.&lt;/p>
&lt;hr>
&lt;h3 id="4-what-happens-when-nulls-appear-in-non-nullable-fields">4. What happens when NULLs appear in non-nullable fields?&lt;/h3>
&lt;br>
&lt;p>NULL is the ghost in your data. It doesn&amp;rsquo;t fail assertions cleanly. It propagates silently through calculations, turning sums into NULLs, joining on nothing, and making counts unreliable. And the most common source of NULLs in your warehouse? Fields that were never supposed to be NULL in the source system, but started arriving that way because a developer forgot to add a &lt;code>NOT NULL&lt;/code> constraint, or an API started returning partial records during degraded performance, or a new code path doesn&amp;rsquo;t populate a field that the old one always did.&lt;/p>
&lt;p>I&amp;rsquo;ve watched a NULL in a single &lt;code>country_code&lt;/code> field cascade through an entire reporting layer, producing &amp;ldquo;Unknown&amp;rdquo; slices in regional dashboards that the leadership team took weeks to notice — and then two more weeks to trace back to the source.&lt;/p>
&lt;h4 id="what-good-looks-like-3">What good looks like&lt;/h4>
&lt;p>Good teams define explicit NULL handling rules for every column in every source. You have a document (or better, code) that says: for &lt;code>email_address&lt;/code>, NULLs are acceptable and should be stored as-is. For &lt;code>order_total&lt;/code>, NULLs indicate a data quality problem and should be quarantined. For &lt;code>created_date&lt;/code>, NULLs are logically impossible and the record should be rejected.&lt;/p>
&lt;p>Good means you&amp;rsquo;re asserting non-nullability in your transformation layer, not just hoping the source enforces it. In dbt, that&amp;rsquo;s a &lt;code>not_null&lt;/code> test on the relevant column. In raw SQL, it&amp;rsquo;s a &lt;code>WHERE column IS NOT NULL&lt;/code> predicate that filters before load, with the rejects sent somewhere visible.&lt;/p>
&lt;p>Good also means your warehouse distinguishes between &amp;ldquo;NULL because the value is genuinely unknown&amp;rdquo; and &amp;ldquo;NULL because the source didn&amp;rsquo;t send it.&amp;rdquo; A dedicated default value like -1 for numeric foreign keys, or &amp;ldquo;Not Provided&amp;rdquo; for string fields, makes this distinction queryable.&lt;/p>
&lt;h4 id="what-amazing-looks-like-3">What amazing looks like&lt;/h4>
&lt;p>Amazing teams implement NULL anomaly detection. You&amp;rsquo;re not just testing for NULLs — you&amp;rsquo;re tracking the NULL rate for every column over time and alerting when it deviates from the baseline. If &lt;code>email_address&lt;/code> is typically 2% NULL and suddenly jumps to 40%, that&amp;rsquo;s an upstream problem, and you catch it before anyone sees broken reports.&lt;/p>
&lt;p>Amazing means you have NULL semantics documented in your data contracts. The source team has committed to: this field will never be NULL under normal operation; if it&amp;rsquo;s NULL, that indicates a system error and should be treated as a defect. That turns a data quality conversation into a bug report.&lt;/p>
&lt;p>Amazing also means your data models are NULL-resilient by design. Your aggregations use &lt;code>COALESCE&lt;/code> and &lt;code>NULLIF&lt;/code> defensively. Your joins handle NULL keys without silently dropping records. Your BI layer renders &amp;ldquo;Data Unavailable&amp;rdquo; instead of showing blank cells. The NULLs might still arrive, but they can&amp;rsquo;t break anything.&lt;/p>
&lt;hr>
&lt;h3 id="5-what-happens-when-the-source-sends-duplicates">5. What happens when the source sends duplicates?&lt;/h3>
&lt;br>
&lt;p>Sources send duplicates. Constantly. Webhook retries, at-least-once delivery guarantees in message queues, CDC replay after an outage, batch extracts that overlap, operators who click the &amp;ldquo;send&amp;rdquo; button twice. If your pipeline assumes every record it receives is unique, it&amp;rsquo;s only a matter of time before your dashboards start reporting inflated numbers.&lt;/p>
&lt;p>The worst part is that duplicate records don&amp;rsquo;t announce themselves. They look exactly like legitimate data. The only way to catch them is to have already decided what &amp;ldquo;unique&amp;rdquo; means and to actively enforce it.&lt;/p>
&lt;h4 id="what-good-looks-like-4">What good looks like&lt;/h4>
&lt;p>Good teams have defined the grain of every table and enforce it. You know that a fact record is uniquely identified by &lt;code>order_id + line_number + event_timestamp&lt;/code>, and you have a uniqueness constraint or test that validates this on every load.&lt;/p>
&lt;p>Good means your ingestion layer deduplicates before writing to the warehouse. Whether that&amp;rsquo;s a &lt;code>QUALIFY ROW_NUMBER() OVER (PARTITION BY key ORDER BY timestamp DESC) = 1&lt;/code> pattern in SQL, a &lt;code>dropDuplicates()&lt;/code> in Spark, or a &lt;code>MERGE&lt;/code> statement that handles duplicates as upserts — you have a defined pattern and you apply it consistently.&lt;/p>
&lt;p>Good also means you track duplicate rates. You know that 0.3% of records from Source A arrive duplicated, and when that spikes to 5%, you investigate. Duplicates are a signal, not just a nuisance.&lt;/p>
&lt;h4 id="what-amazing-looks-like-4">What amazing looks like&lt;/h4>
&lt;p>Amazing teams deduplicate at multiple layers — a defence-in-depth approach. Raw ingestion deduplicates by event ID. The staging layer deduplicates by business key. The transformation layer validates uniqueness at the grain. Any duplicate that makes it through all three layers triggers an alert.&lt;/p>
&lt;p>Amazing means your streaming pipelines use stateful deduplication with sliding windows. You maintain a cache of recently processed event IDs (in-memory, in Redis, or in the target database) and reject duplicates within a configurable window — say, the last 24 hours. Outside that window, you trade perfect deduplication for practical performance, which is an explicit, documented tradeoff.&lt;/p>
&lt;p>Amazing also means you never delete or discard duplicates silently. You route them to a dead-letter table with metadata about when they arrived, why they were classified as duplicates, and what the primary record looks like. This gives you an audit trail and lets you spot systematic issues — like a webhook that&amp;rsquo;s been misconfigured to retry on success.&lt;/p>
&lt;hr>
&lt;h3 id="6-what-happens-when-the-extract-job-runs-twice">6. What happens when the extract job runs twice?&lt;/h3>
&lt;br>
&lt;p>This is idempotency, and it&amp;rsquo;s the single most important design principle for data pipelines that most teams get wrong.&lt;/p>
&lt;p>Your Airflow DAG fails halfway through. You hit retry. Your scheduler has a hiccup and triggers the same job twice. An engineer reruns a backfill without checking whether the original succeeded. In every one of these scenarios, your pipeline gets the same input more than once. What happens to the output?&lt;/p>
&lt;p>If you&amp;rsquo;re appending records with &lt;code>INSERT INTO&lt;/code>, you just doubled your data. If you&amp;rsquo;re overwriting a table, you might lose records that were added by another process. If you&amp;rsquo;re updating aggregations incrementally, your totals are now wrong.&lt;/p>
&lt;p>Research from Airbyte describes how a multinational bank&amp;rsquo;s payment-processing system, lacking idempotency controls, created duplicate transactions worth millions during a simple retry. The root cause wasn&amp;rsquo;t a complex failure — it was a fundamental design flaw.&lt;/p>
&lt;h4 id="what-good-looks-like-5">What good looks like&lt;/h4>
&lt;p>Good teams use the delete-write pattern. Before writing output for a given partition or date, you delete any existing data for that partition, then write the new data. Run it once, run it ten times — the result is the same. In SQL, that looks like:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">DELETE&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> target_table &lt;span style="color:#66d9ef">WHERE&lt;/span> partition_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2025-03-09&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> target_table
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> staging_table &lt;span style="color:#66d9ef">WHERE&lt;/span> partition_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2025-03-09&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Good means every pipeline can be safely rerun without manual intervention. No &amp;ldquo;before you rerun, make sure you delete the output from the last run&amp;rdquo; instructions. If your runbook has manual steps for reruns, your pipeline isn&amp;rsquo;t idempotent.&lt;/p>
&lt;p>Good also means you test for idempotency explicitly. Run the pipeline. Note the row count. Run it again with the same input. If the row count changes, you have a problem. This should be an automated test in your CI pipeline.&lt;/p>
&lt;h4 id="what-amazing-looks-like-5">What amazing looks like&lt;/h4>
&lt;p>Amazing teams use &lt;code>MERGE&lt;/code> / upsert patterns that handle inserts, updates, and deletes in a single atomic operation. Combined with table formats like Delta Lake or Iceberg that provide ACID transactions, you get idempotent writes that survive partial failures.&lt;/p>
&lt;p>Amazing means your pipeline&amp;rsquo;s idempotency extends beyond the write. If your extract hits an API, it uses idempotency keys. If it publishes events downstream, it includes deduplication IDs. The entire pipeline, from source to serving layer, produces identical output regardless of how many times it&amp;rsquo;s executed.&lt;/p>
&lt;p>Amazing also means you maintain processing metadata. Each pipeline run records its run ID, input parameters, record counts, and checksums. Before writing, the pipeline checks whether a run with identical parameters has already produced output — and if so, skips processing entirely. Not just idempotent writes — idempotent execution.&lt;/p>
&lt;hr>
&lt;h3 id="7-what-happens-when-timestamps-change-timezone">7. What happens when timestamps change timezone?&lt;/h3>
&lt;br>
&lt;p>Timezone bugs are the cockroaches of data engineering — they survive everything. They lurk in corner cases, they reproduce during daylight saving transitions, and they&amp;rsquo;re nearly impossible to fully eradicate once they&amp;rsquo;ve infested a codebase.&lt;/p>
&lt;p>Here&amp;rsquo;s a scenario I&amp;rsquo;ve seen multiple times: a source system stores timestamps in local time without timezone information. Your pipeline ingests them and assumes UTC. Everything works for months — until the clocks change and suddenly your event timestamps are off by an hour. A financial services company found that a source system migration introduced a timezone mismatch between loan default dates and origination dates. The resulting temporal inconsistency corrupted months of historical analysis.&lt;/p>
&lt;p>The sneaky part? Timezone bugs don&amp;rsquo;t always cause failures. They cause &lt;em>subtle inaccuracies&lt;/em>. Events that happened at 11:55 PM on Tuesday show up as Wednesday. Daily aggregations double-count an hour during the spring transition and miss an hour in autumn. A job scheduled for 2 AM during a daylight saving transition either runs twice or not at all.&lt;/p>
&lt;h4 id="what-good-looks-like-6">What good looks like&lt;/h4>
&lt;p>Good teams store everything in UTC, always. Not &amp;ldquo;usually UTC.&amp;rdquo; Not &amp;ldquo;UTC except for that one source that sends in Eastern.&amp;rdquo; UTC, full stop. Local time is a presentation-layer concern and has no business being in your warehouse.&lt;/p>
&lt;p>Good means you know the timezone semantics of every source. You&amp;rsquo;ve documented that System A sends timestamps in UTC, System B sends in US/Eastern with DST adjustments, and System C sends timezone-naive timestamps that are implicitly Australia/Sydney. Your pipeline converts explicitly on ingestion.&lt;/p>
&lt;p>Good also means you handle DST transitions in your scheduling. If your pipeline runs at 2 AM local time, you&amp;rsquo;ve tested what happens when that hour doesn&amp;rsquo;t exist (spring forward) or exists twice (fall back). Most orchestrators evaluate cron expressions in UTC — you&amp;rsquo;ve confirmed this for yours.&lt;/p>
&lt;h4 id="what-amazing-looks-like-6">What amazing looks like&lt;/h4>
&lt;p>Amazing teams store timestamps with timezone metadata. Not just &amp;ldquo;2025-03-09 14:30:00&amp;rdquo; — but &amp;ldquo;2025-03-09 14:30:00 UTC&amp;rdquo; alongside the original timezone identifier (like &lt;code>America/New_York&lt;/code>). This lets you reconstruct the local time accurately for any historical record, even if timezone rules change.&lt;/p>
&lt;p>Amazing means you have automated timezone consistency checks. Every timestamp column in your warehouse is tested to verify it&amp;rsquo;s in UTC. If a record arrives with a timestamp that&amp;rsquo;s suspiciously offset from its expected range (events from a US system arriving with timestamps that look like Asian business hours), it&amp;rsquo;s flagged for investigation.&lt;/p>
&lt;p>Amazing also means your data models account for the fact that &amp;ldquo;today&amp;rdquo; means different things in different timezones. A global company&amp;rsquo;s &amp;ldquo;daily active users&amp;rdquo; metric explicitly defines which timezone&amp;rsquo;s midnight marks the boundary. Your models support configurable timezone-based aggregation windows, not just UTC midnight.&lt;/p>
&lt;hr>
&lt;h3 id="8-what-happens-when-the-source-has-a-bad-day-and-sends-garbage">8. What happens when the source has a bad day and sends garbage?&lt;/h3>
&lt;br>
&lt;p>Systems degrade. APIs return malformed JSON. Database exports get truncated. Character encoding changes silently. A deployment goes wrong and half the fields come back as their default values. A logging system buffer overflows and starts concatenating records.&lt;/p>
&lt;p>The question isn&amp;rsquo;t whether you&amp;rsquo;ll receive garbage data — it&amp;rsquo;s whether your pipeline can tell the difference between garbage and legitimate outliers. A revenue figure of $0 might be a real zero-dollar transaction or it might be a failed parse. A customer name of &amp;ldquo;NULL&amp;rdquo; might be an actual person named Null (they exist) or a serialisation bug.&lt;/p>
&lt;h4 id="what-good-looks-like-7">What good looks like&lt;/h4>
&lt;p>Good teams implement data quality checks at the boundaries. Before data enters the warehouse, you validate: record counts are within expected ranges, key fields are populated, data types match expectations, and categorical fields contain only known values.&lt;/p>
&lt;p>Good means you have circuit-breakers. If a source extract is more than 30% smaller than the previous day (or 30% larger), the pipeline pauses and alerts. If more than 5% of records fail validation, the entire batch is quarantined for review rather than partially loaded.&lt;/p>
&lt;p>Good also means you maintain a data quality score for each source. Not a single binary pass/fail, but a composite metric that tracks completeness, accuracy, consistency, and timeliness over time. When the score dips below a threshold, stakeholders are notified with context, not just &amp;ldquo;data quality alert.&amp;rdquo;&lt;/p>
&lt;h4 id="what-amazing-looks-like-7">What amazing looks like&lt;/h4>
&lt;p>Amazing teams implement statistical anomaly detection. You&amp;rsquo;re not just checking against static thresholds — you&amp;rsquo;re comparing today&amp;rsquo;s data against a rolling baseline of the last 30 days. A revenue column with a standard deviation of $50 that suddenly shows values of $50,000 triggers an investigation, even though $50,000 is a perfectly valid number in isolation.&lt;/p>
&lt;p>Amazing means you have data quality rules that are co-authored with the business. The data team doesn&amp;rsquo;t decide alone what constitutes &amp;ldquo;garbage.&amp;rdquo; The finance team has specified that order totals should always be positive, that discount percentages can&amp;rsquo;t exceed 100%, and that &lt;code>order_date&lt;/code> must be within the last 90 days. These rules are version-controlled and tested automatically.&lt;/p>
&lt;p>Amazing also means you have automated rollback capabilities. When a bad batch is detected after it&amp;rsquo;s already been loaded, the system can automatically revert to the last known good state — using table time-travel features in Delta Lake or Iceberg — without manual intervention. Garbage got in, garbage gets out, and the dashboard never shows corrupted data.&lt;/p>
&lt;hr>
&lt;h3 id="9-who-gets-paged-when-this-breaks-at-2am">9. Who gets paged when this breaks at 2am?&lt;/h3>
&lt;br>
&lt;p>This is where engineering meets organisational design, and it&amp;rsquo;s where most data teams are embarrassingly underprepared.&lt;/p>
&lt;p>When a pipeline fails at 2 AM, what actually happens? In too many organisations, the answer is: nothing, until someone opens a dashboard at 9 AM and notices the numbers are stale. Or worse: the alert fires, it goes to a Teams channel that nobody&amp;rsquo;s watching, and it sits there until Monday.&lt;/p>
&lt;p>The question isn&amp;rsquo;t really about paging infrastructure. It&amp;rsquo;s about whether you&amp;rsquo;ve decided, ahead of time, who is responsible for what. Is a missing report the responsibility of the data engineers who built the pipeline, the analytics team who created the dashboard, the application team whose schema change caused the failure, or the platform team whose Kubernetes cluster ran out of memory?&lt;/p>
&lt;p>If you haven&amp;rsquo;t answered that question before the incident, you&amp;rsquo;ll waste the first hour of every incident answering it during the crisis.&lt;/p>
&lt;h4 id="what-good-looks-like-8">What good looks like&lt;/h4>
&lt;p>Good teams have on-call rotations with clear escalation paths. Pipeline failures route to a specific person, not a channel. That person has documented runbooks for the most common failure modes, and they have the access and authority to restart jobs, quarantine bad data, and communicate with stakeholders.&lt;/p>
&lt;p>Good means you&amp;rsquo;ve classified your pipelines by criticality. The CEO dashboard pipeline and the experimental marketing attribution pipeline don&amp;rsquo;t get the same SLA. Critical pipelines page immediately. Non-critical pipelines send a notification that gets triaged during business hours.&lt;/p>
&lt;p>Good also means you have defined SLAs with your stakeholders. They know that the revenue dashboard refreshes by 7 AM, that freshness is guaranteed within 4 hours, and that if something breaks, they&amp;rsquo;ll get a status update within 30 minutes. No surprises.&lt;/p>
&lt;h4 id="what-amazing-looks-like-8">What amazing looks like&lt;/h4>
&lt;p>Amazing teams practice incident response. Not just documenting it — rehearsing it. They run game days where they simulate pipeline failures and walk through the response process. They measure time-to-detection, time-to-response, and time-to-resolution, and they track these metrics over time.&lt;/p>
&lt;p>Amazing means every incident becomes a rule. After every data incident, there&amp;rsquo;s a blameless postmortem that produces concrete action items: a new test, a new alert, a new runbook entry, a new data contract clause. The goal is that no failure mode catches you twice.&lt;/p>
&lt;p>Amazing also means your monitoring is proactive, not just reactive. You&amp;rsquo;re not just detecting failures — you&amp;rsquo;re detecting the conditions that precede failures. If processing time has been creeping up by 5% per week, you&amp;rsquo;ll hit your SLA window in six weeks. Amazing teams notice that trend and fix it before it becomes an incident.&lt;/p>
&lt;hr>
&lt;h3 id="10-who-actually-owns-this-data">10. Who actually owns this data?&lt;/h3>
&lt;br>
&lt;p>This is the most important question on the list, and it&amp;rsquo;s the one that gets the vaguest answers. &amp;ldquo;The data team owns it.&amp;rdquo; &amp;ldquo;Engineering owns it.&amp;rdquo; &amp;ldquo;It&amp;rsquo;s in the data platform, so I guess&amp;hellip; platform engineering?&amp;rdquo;&lt;/p>
&lt;p>When nobody owns the data, everybody suffers. Source system teams change schemas without notification because it&amp;rsquo;s not their job to worry about downstream consumers. Data engineers treat data quality as someone else&amp;rsquo;s problem because they&amp;rsquo;re just moving bytes. Analysts blame engineers, engineers blame source systems, and source systems are blissfully unaware that anyone is even using their data.&lt;/p>
&lt;p>The question of data ownership is really three questions: Who is accountable for the data&amp;rsquo;s &lt;em>content&lt;/em> (is it correct)? Who is responsible for its &lt;em>delivery&lt;/em> (does it arrive on time)? And who decides its &lt;em>definition&lt;/em> (what does &amp;ldquo;active customer&amp;rdquo; actually mean)?&lt;/p>
&lt;h4 id="what-good-looks-like-9">What good looks like&lt;/h4>
&lt;p>Good teams have explicit ownership documented for every source. There&amp;rsquo;s a human name attached to each data source — not a team, not a Teams channel, but a person who is accountable for its quality, availability, and definition. When that person changes roles, ownership is explicitly transferred, not silently abandoned.&lt;/p>
&lt;p>Good means the data team has data contracts with source system owners. These contracts specify: what data is provided, in what format, with what SLAs, with what change notification process, and what happens when the contract is violated. It&amp;rsquo;s not adversarial — it&amp;rsquo;s professional.&lt;/p>
&lt;p>Good also means you have a data catalog that maps lineage from source to consumption. When a downstream dashboard breaks, you can trace back through every transformation to the specific source table and the specific owner. No detective work required.&lt;/p>
&lt;h4 id="what-amazing-looks-like-9">What amazing looks like&lt;/h4>
&lt;p>Amazing teams treat data as a product. Each critical dataset has a product owner who is accountable for its fitness for use — not just its technical availability. They gather requirements from consumers, they prioritise improvements, and they measure satisfaction. Data isn&amp;rsquo;t just &amp;ldquo;there.&amp;rdquo; It&amp;rsquo;s maintained, versioned, and evolved with the same rigour as an API.&lt;/p>
&lt;p>Amazing means ownership boundaries are encoded in the infrastructure. Data contracts aren&amp;rsquo;t just documents — they&amp;rsquo;re executable specifications that are validated on every pipeline run. When a source system violates its contract, the pipeline doesn&amp;rsquo;t just fail — it identifies the violation, classifies the severity, notifies the source owner, and quarantines the affected data. The accountability is automated.&lt;/p>
&lt;p>Amazing also means you conduct regular data ownership reviews. Quarterly, you audit every data source: Is the owner still accurate? Is the contract still valid? Are the SLAs still appropriate? Are there new consumers who need to be included in change notification? Data ownership isn&amp;rsquo;t a one-time assignment — it&amp;rsquo;s an ongoing practice.&lt;/p>
&lt;hr>
&lt;h3 id="the-checklist-that-saves-you">The checklist that saves you&lt;/h3>
&lt;br>
&lt;p>Here are all ten questions in a format you can bring to your next pipeline design review. Print it out. Tape it to your monitor. Make it a required section in your technical design documents.&lt;/p>
&lt;p>Before you ingest a new source system, answer these:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Schema changes&lt;/strong> — What&amp;rsquo;s our detection and response strategy?&lt;/li>
&lt;li>&lt;strong>Late-arriving data&lt;/strong> — What&amp;rsquo;s our tolerance window and reconciliation process?&lt;/li>
&lt;li>&lt;strong>Primary key reuse&lt;/strong> — How do we generate warehouse-independent identity?&lt;/li>
&lt;li>&lt;strong>NULL invasion&lt;/strong> — What are our per-column NULL handling rules?&lt;/li>
&lt;li>&lt;strong>Duplicate records&lt;/strong> — How do we define and enforce uniqueness at every layer?&lt;/li>
&lt;li>&lt;strong>Double execution&lt;/strong> — Can this pipeline run twice safely?&lt;/li>
&lt;li>&lt;strong>Timezone chaos&lt;/strong> — Are we storing UTC with source timezone metadata?&lt;/li>
&lt;li>&lt;strong>Garbage data&lt;/strong> — What are our circuit-breakers and anomaly thresholds?&lt;/li>
&lt;li>&lt;strong>Incident response&lt;/strong> — Who gets paged, and what&amp;rsquo;s the runbook?&lt;/li>
&lt;li>&lt;strong>Data ownership&lt;/strong> — Who&amp;rsquo;s accountable for content, delivery, and definition?&lt;/li>
&lt;/ol>
&lt;p>If you can answer all ten, you&amp;rsquo;ve done more due diligence than most. If you can answer them at the &amp;ldquo;amazing&amp;rdquo; level, you&amp;rsquo;ve built something that&amp;rsquo;ll survive the chaos of production for years.&lt;/p>
&lt;hr>
&lt;h3 id="why-i-keep-writing-about-this">Why I keep writing about this&lt;/h3>
&lt;br>
&lt;p>Every one of these ten questions is, at its core, about the same thing: &lt;strong>caring enough to think ahead.&lt;/strong> Not just building something that works today, but building something that handles the mess that tomorrow will inevitably deliver. That&amp;rsquo;s not a technical skill. It&amp;rsquo;s a disposition. It&amp;rsquo;s deciding that the quiet discipline of prevention matters more than the visible heroism of firefighting.&lt;/p>
&lt;p>The best data engineers I&amp;rsquo;ve worked with all share this trait. They&amp;rsquo;re not the ones with the most dramatic incident stories. They&amp;rsquo;re the ones whose systems are boring. Predictably, reliably, beautifully boring.&lt;/p>
&lt;p>That&amp;rsquo;s the work I want to celebrate.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Quality</category><category>Pipeline Design</category><category>Schema Drift</category><category>Idempotency</category><category>Data Contracts</category><category>Incident Response</category><category>Data Ownership</category></item><item><title>You Don't Need Permission to Fix Your Data</title><link>https://ghostinthedata.info/posts/2026/2026-03-21-data-quality-when-your-a-junior/</link><pubDate>Sat, 21 Mar 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-03-21-data-quality-when-your-a-junior/</guid><author>Chris Hillman</author><description>Five battle-tested tactics junior data engineers can use to improve data quality without waiting for authority, backed by real case studies from Airbnb, Google, and Warner Bros. Discovery.</description><content:encoded>&lt;p>Let me tell you about a junior engineer called Sam.&lt;/p>
&lt;p>Sam had been on the team about four months when I noticed something in a pull request. Tucked between two routine model changes was a new &lt;code>schema.yml&lt;/code> entry — five &lt;code>accepted_values&lt;/code> tests on a column called &lt;code>customer_status&lt;/code> that had been silently accumulating fourteen different spellings of &amp;ldquo;active&amp;rdquo; for the better part of a year.&lt;/p>
&lt;p>Nobody asked Sam to do this. It wasn&amp;rsquo;t in a sprint. There was no Jira ticket. Sam had just been working in that part of the warehouse, noticed the mess, and decided to clean it up on the way through.&lt;/p>
&lt;p>I left a comment on the PR: &amp;ldquo;Nice catch.&amp;rdquo; Sam&amp;rsquo;s reply was almost apologetic — &amp;ldquo;Hope it&amp;rsquo;s okay that I added these, I wasn&amp;rsquo;t sure if I was supposed to.&amp;rdquo;&lt;/p>
&lt;p>That response stuck with me. &lt;em>I wasn&amp;rsquo;t sure if I was supposed to.&lt;/em>&lt;/p>
&lt;p>Here was someone who&amp;rsquo;d spotted a real problem, built a real solution, tested it, and shipped it — and their instinct was to wonder whether they had permission. Not whether the fix was correct. Not whether the tests were well-written. Whether they were &lt;em>allowed&lt;/em>.&lt;/p>
&lt;p>I&amp;rsquo;ve thought about that moment a lot since, because it captures something I see in almost every data team I work with. The people closest to the problems — the ones running the queries, staring at the nulls, fielding the Microsoft Teams messages when dashboards look wrong — are often the last to believe they can do anything about it. Not because they lack skill. Because somewhere along the way, they absorbed the idea that fixing things is someone else&amp;rsquo;s job.&lt;/p>
&lt;p>That belief is expensive. And it&amp;rsquo;s wrong.&lt;/p>
&lt;p>I&amp;rsquo;m telling you this because data quality — the real, unglamorous, column-by-column work of making data trustworthy — is one of those things that sounds like a senior engineering problem until you see a four-month-old team member fix something a dozen experienced engineers walked past every day. It doesn&amp;rsquo;t take authority. It takes the willingness to act before you&amp;rsquo;re told to.&lt;/p>
&lt;p>This article is about how to be more like Sam. Five tactics you can start this week — no permission required — backed by real case studies from companies like Airbnb, Google, and Warner Bros. Discovery that figured this out the hard way.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="the-129-million-reason-your-quality-problem-isnt-too-small-to-fix">The $12.9 million reason your quality problem isn&amp;rsquo;t &amp;ldquo;too small&amp;rdquo; to fix&lt;/h3>
&lt;br>
&lt;p>Before we get into tactics, let&amp;rsquo;s talk about why your &amp;ldquo;small&amp;rdquo; fix matters more than you think.&lt;/p>
&lt;p>Gartner pegs the average annual cost of poor data quality at &lt;strong>$12.9 million per organisation&lt;/strong>. IBM and Harvard Business Review put the aggregate U.S. cost at $3.1 trillion annually. These numbers feel abstract until you realise what they actually represent: knowledge workers spending roughly half their time hunting for data, correcting errors, and searching for confirmatory sources instead of doing their actual jobs.&lt;/p>
&lt;p>Monte Carlo&amp;rsquo;s 2024 State of Data Quality survey found bad data now impacts &lt;strong>31% of company revenue&lt;/strong> on average — up from 26% just two years earlier. Their 2022 survey found data engineers spend &lt;strong>40% of their time&lt;/strong> evaluating or checking data quality. Soda&amp;rsquo;s research put that number even higher: 61% of data engineers spend half or more of their time handling data issues.&lt;/p>
&lt;p>And the human cost? A DataKitchen survey of 600 data engineers found &lt;strong>87% are frequently blamed&lt;/strong> when things go wrong with data. 78% wish their job came with a therapist.&lt;/p>
&lt;p>The implication for you: the problem is so large and so costly that any measurable improvement you create has real dollar value attached to it. That null check you added last Thursday? That&amp;rsquo;s not a minor fix — it&amp;rsquo;s a tiny piece of a multi-million-dollar problem that nobody else bothered to address.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="tactic-1-write-tests-for-your-own-models-without-asking-anyone">Tactic 1: write tests for your own models without asking anyone&lt;/h3>
&lt;br>
&lt;p>The single best case study for this tactic comes from Airbnb, and it didn&amp;rsquo;t start with a VP mandate or an OKR. It started with one engineer.&lt;/p>
&lt;p>Airbnb&amp;rsquo;s engineering blog documents how they went from &amp;ldquo;an engineering culture in which most new code shipped without a single test&amp;rdquo; to one where proposing a change without tests got called out immediately. The transformation wasn&amp;rsquo;t top-down. As the blog explains: &amp;ldquo;One approach is to rule by edict, but given our culture of engineer autonomy this would have been very poorly received. The other approach is to lead by example and build a movement.&amp;rdquo;&lt;/p>
&lt;p>That single champion&amp;rsquo;s playbook was remarkably tactical. They spoke in engineering meetings. They held office hours showing teammates how to write tests. They published a &lt;strong>weekly newsletter highlighting well-written test specs&lt;/strong> — giving public credit to engineers who included tests. They revamped the testing bootcamp for new hires, effectively making each new hire a champion. They improved tooling, reducing CI build times from over an hour to roughly six minutes.&lt;/p>
&lt;p>The result? Airbnb later applied the same grassroots philosophy to data quality, building a DQ Score on a 0–100 scale across their entire warehouse. Their anomaly detection system now prevents quality issues in new pipelines before they reach production. The Minerva metrics platform manages 12,000+ metrics and 4,000+ dimensions with 200+ data producers.&lt;/p>
&lt;p>One engineer. No mandate. A newsletter and some office hours.&lt;/p>
&lt;h4 id="what-this-looks-like-for-you">What this looks like for you&lt;/h4>
&lt;p>If you&amp;rsquo;re working with dbt, you already have the tools. dbt ships with four built-in generic tests — &lt;code>not_null&lt;/code>, &lt;code>unique&lt;/code>, &lt;code>accepted_values&lt;/code>, &lt;code>relationships&lt;/code> — that you define in YAML alongside your model definitions. No framework to build. No proposal to write. Just add a few lines to your &lt;code>schema.yml&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">dim_customers&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">columns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_id&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tests&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#ae81ff">not_null&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#ae81ff">unique&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_status&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tests&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">accepted_values&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">values&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;active&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;inactive&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;churned&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;suspended&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">email&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tests&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#ae81ff">not_null&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That&amp;rsquo;s it. Next time the pipeline runs, those tests execute automatically. If someone pushes a change that breaks uniqueness on &lt;code>customer_id&lt;/code>, the pipeline catches it before it reaches production.&lt;/p>
&lt;p>The &lt;code>dbt-expectations&lt;/code> package goes further, porting Great Expectations&amp;rsquo; advanced tests into dbt. Unit tests — introduced in dbt v1.8 — let you validate SQL logic with static mock inputs before transformations run. Tests integrate directly into CI/CD, blocking PRs with failures from reaching production.&lt;/p>
&lt;p>Not on dbt? You can still write assertion queries. Here&amp;rsquo;s a completeness check you can run against any SQL database — no framework required:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Quick completeness check: what percentage of each critical
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- field is actually populated?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> total_records,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- How many emails do we actually have?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> ROUND(&lt;span style="color:#ae81ff">100&lt;/span>.&lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(email) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>), &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> email_pct,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- How many phone numbers?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> ROUND(&lt;span style="color:#ae81ff">100&lt;/span>.&lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(phone) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>), &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> phone_pct,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- How many addresses?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> ROUND(&lt;span style="color:#ae81ff">100&lt;/span>.&lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(address_line_1) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>), &lt;span style="color:#ae81ff">1&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> address_pct,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Flag: are we below the 95% threshold on anything critical?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(email) &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>.&lt;span style="color:#ae81ff">95&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;BELOW THRESHOLD&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;OK&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> email_status
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> main.customers;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Schedule that to run daily. Store the results in a logging table. You&amp;rsquo;ve just built a monitoring system — and you didn&amp;rsquo;t ask anyone&amp;rsquo;s permission.&lt;/p>
&lt;h4 id="the-numbers-back-this-up">The numbers back this up&lt;/h4>
&lt;p>Monte Carlo&amp;rsquo;s 2022 survey found that organisations conducting at least three types of data tests weekly experienced &lt;strong>46 incidents per month&lt;/strong> compared to &lt;strong>61 incidents per month&lt;/strong> for less rigorous testers — a 25% reduction. The Forrester Total Economic Impact study of data observability platforms (April 2025) found 358% ROI, 6,500+ data personnel hours reclaimed annually, and $1.5M+ in avoided lost revenue from reduced data downtime.&lt;/p>
&lt;p>Daniel Beach of Data Engineering Central offers a pragmatic path for getting started: begin with end-to-end tests first when inheriting a pipeline with no tests, then add data quality checks, then unit tests. You don&amp;rsquo;t need to boil the ocean. You need to start.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="tactic-2-document-one-thing-every-time-you-touch-it">Tactic 2: document one thing every time you touch it&lt;/h3>
&lt;br>
&lt;p>The numbers on missing documentation are genuinely staggering. According to DX, developers spend &lt;strong>3–10 hours per week&lt;/strong> searching for information that should be documented. For a 100-person engineering team, that equals 300–1,000 hours weekly — the equivalent of 8–25 full-time engineers doing nothing but hunting for answers. Stack Overflow&amp;rsquo;s developer survey found 62% of developers spend over 30 minutes daily on poorly documented issues. And 38% listed poor documentation among their top reasons for leaving a company.&lt;/p>
&lt;p>One practitioner tracked &amp;ldquo;Time to First Production Commit&amp;rdquo; at &lt;strong>28 days&lt;/strong> for new engineers — versus an industry benchmark of 3–5 days. Engineers on that team spent 30% of their time rediscovering what previous engineers had already learned. 40% of sprint tickets required inter-team clarification, averaging 12 clarifications per ticket.&lt;/p>
&lt;p>This is what documentation debt looks like. Not a gap in a wiki — a tax on every single person who touches the codebase.&lt;/p>
&lt;h4 id="documentation-fridays-dont-work-this-does">&amp;ldquo;Documentation Fridays&amp;rdquo; don&amp;rsquo;t work. This does.&lt;/h4>
&lt;p>Multiple sources confirm that big-bang documentation efforts fail. As one case study puts it: &amp;ldquo;&amp;lsquo;Okay team, let&amp;rsquo;s spend Friday afternoon documenting our processes!&amp;rsquo; Nobody wants to do that. And it doesn&amp;rsquo;t work.&amp;rdquo;&lt;/p>
&lt;p>The successful pattern is document-as-you-go. Every time you encounter an issue, you write it down. Shopify Engineering treats documentation as a product, explicitly encouraging new team members to fill documentation gaps from day one. Their philosophy: &amp;ldquo;The most effective way to encourage others to write is to actually write.&amp;rdquo;&lt;/p>
&lt;p>For data teams specifically, the approach is even simpler. If you&amp;rsquo;re using dbt, column descriptions live in &lt;code>schema.yml&lt;/code> alongside your model definitions. dbt&amp;rsquo;s column-level lineage means passthrough and renamed columns &lt;strong>automatically inherit descriptions from upstream models&lt;/strong>. Write a description once, and it flows downstream through every model that references that column.&lt;/p>
&lt;p>Here&amp;rsquo;s what this looks like in practice:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">dim_customers&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &amp;gt;&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> One row per customer. Grain: customer_id.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Source: CRM extract via Fivetran, refreshed daily at 03:00 UTC.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Owner: data-eng-team@company.com&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">columns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_id&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Primary key. Sourced from CRM.contacts.id. Unique, never null.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">customer_status&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &amp;gt;&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Current lifecycle status. Valid values: active, inactive, 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> churned, suspended. Updated by CRM webhook on status change.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Known issue: legacy records from pre-2023 may contain 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#39;Active&amp;#39; (capitalised) — normalised in this model.&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">lifetime_value&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &amp;gt;&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Cumulative revenue attributed to this customer, in AUD.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Calculated as SUM(order_total) from fct_orders.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Excludes refunded orders. Updated daily.&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That &amp;ldquo;Known issue&amp;rdquo; note on &lt;code>customer_status&lt;/code>? That&amp;rsquo;s worth its weight in gold to the next person who touches this model. It took thirty seconds to write and it&amp;rsquo;ll save hours of confusion.&lt;/p>
&lt;p>Microsoft&amp;rsquo;s engineering blogs describe the alternative bluntly: &amp;ldquo;When tribal knowledge holders leave, reverse engineering is the only way to understand what was done.&amp;rdquo; Enboarder&amp;rsquo;s 2025 research found &lt;strong>47% of organisations&lt;/strong> cite institutional knowledge loss as their top offboarding challenge. Atlan&amp;rsquo;s Modern Data Survey found data professionals waste &lt;strong>one full day per week&lt;/strong> simply trying to figure out what data to use.&lt;/p>
&lt;p>You don&amp;rsquo;t need to document the entire warehouse. You need to document the thing you touched today. Tomorrow, document the next thing. In three months, you&amp;rsquo;ll have built something genuinely valuable — and you&amp;rsquo;ll have done it one commit at a time.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="tactic-3-build-a-quality-dashboard-nobody-asked-for">Tactic 3: build a quality dashboard nobody asked for&lt;/h3>
&lt;br>
&lt;p>At Warner Bros. Discovery, the DICE team built data quality management frameworks and created a &amp;ldquo;Data Quality Forum&amp;rdquo; that became a bridge between teams. During the Olympics livestream, they used custom SQL checks to detect missing content metadata — fixing issues before they broke reporting. The forum evolved into what Pam Zirpoli called a &amp;ldquo;cultural flywheel.&amp;rdquo; Patterns surfaced in the forum were incorporated into onboarding, translated into new monitors, and used to refine their priority matrix. Teams shifted from reactive — reacting after dashboards broke — to proactive.&lt;/p>
&lt;p>At Tempus, a data engineer documented how adopting Elementary (an open-source dbt package) transformed their monitoring. Before: &amp;ldquo;test alerting relied on a dashboard in Data Studio built on top of a log sink, and the &amp;lsquo;alerts&amp;rsquo; were a daily email that you actually had to open.&amp;rdquo; After: Microsoft Teams alerts that &amp;ldquo;@-ed a user and continued to do so until they&amp;rsquo;re turned off.&amp;rdquo; The key outcome: tests no longer fail silently.&lt;/p>
&lt;h4 id="the-six-metrics-that-actually-matter">The six metrics that actually matter&lt;/h4>
&lt;p>Drawing from IBM, dbt Labs, Databricks, and Alation, the consensus metrics for data quality monitoring are:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Freshness&lt;/strong>: time since last data update. If your orders table hasn&amp;rsquo;t been updated since yesterday, something&amp;rsquo;s broken.&lt;/li>
&lt;li>&lt;strong>Volume&lt;/strong>: expected versus actual row counts. Did today&amp;rsquo;s load bring in 50,000 rows when you normally get 48,000–52,000? Fine. Did it bring 12? Problem.&lt;/li>
&lt;li>&lt;strong>Schema changes&lt;/strong>: added, deleted, or altered columns. These should never surprise you.&lt;/li>
&lt;li>&lt;strong>Completeness&lt;/strong>: null rates per column. Simple &lt;code>COUNT(*)&lt;/code> vs &lt;code>COUNT(column)&lt;/code> tells you a lot.&lt;/li>
&lt;li>&lt;strong>Distribution shifts&lt;/strong>: statistical changes in data values. If the average order value suddenly doubles, that&amp;rsquo;s worth investigating.&lt;/li>
&lt;li>&lt;strong>Test pass/fail rates&lt;/strong>: the percentage of your quality assertions that are passing. Track this over time.&lt;/li>
&lt;/ul>
&lt;p>Here&amp;rsquo;s a freshness check you can run right now — no tools, no frameworks, just SQL:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- How stale is each of your critical tables?
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Run this daily and store results in a monitoring table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;fct_orders&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">table_name&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at) &lt;span style="color:#66d9ef">AS&lt;/span> last_record_timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> checked_at,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXTRACT&lt;/span>(EPOCH &lt;span style="color:#66d9ef">FROM&lt;/span> (&lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span> &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at))) &lt;span style="color:#f92672">/&lt;/span> &lt;span style="color:#ae81ff">3600&lt;/span>.&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">AS&lt;/span> hours_since_update,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at) &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CRITICAL - No data&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span> &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at) &lt;span style="color:#f92672">&amp;gt;&lt;/span> INTERVAL &lt;span style="color:#e6db74">&amp;#39;24 hours&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;STALE - Exceeds 24hr SLA&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span> &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">MAX&lt;/span>(updated_at) &lt;span style="color:#f92672">&amp;gt;&lt;/span> INTERVAL &lt;span style="color:#e6db74">&amp;#39;12 hours&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;WARNING - Approaching SLA&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FRESH&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> freshness_status
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> analytics.fct_orders;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Store that in a &lt;code>data_quality_log&lt;/code> table, connect it to Looker Studio or Metabase (both free), and you&amp;rsquo;ve got a monitoring dashboard. That&amp;rsquo;s it. No Kubernetes cluster. No proposal document.&lt;/p>
&lt;h4 id="visibility-changes-behaviour">Visibility changes behaviour&lt;/h4>
&lt;p>Anna Geller documented how sending data quality alerts to a shared Microsoft Teams channel transforms team behaviour. Her core insight: &amp;ldquo;As social creatures, we are more motivated to fix issues if other people can see the effort we put into it.&amp;rdquo; Data owners reply with root cause analysis, add checkmarks, or link tickets — creating a social proof loop that issues are no longer ignored.&lt;/p>
&lt;p>This isn&amp;rsquo;t just anecdotal. Research on the Hawthorne effect shows healthcare hand hygiene compliance increases by &lt;strong>up to 70%&lt;/strong> when workers know they&amp;rsquo;re observed, and employees increase productivity by an average of 13% when activity is monitored. Making data quality visible doesn&amp;rsquo;t just inform people — it changes their behaviour.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="tactic-4-create-a-data-bugs-channel">Tactic 4: create a #data-bugs channel&lt;/h3>
&lt;br>
&lt;p>MIT Sloan Management Review&amp;rsquo;s HelloFresh case study documents one of the best examples of how visibility catalyses organisational change. HelloFresh transitioned through three modes of data quality management. In Mode 1, issues were dealt with reactively by whoever happened to be affected. In Mode 2, a central engineering team handled cleanup but lacked understanding of how data was actually used — &amp;ldquo;data customers grew increasingly frustrated as they continued to spend large fractions of their workdays dealing with data that did not meet their needs.&amp;rdquo;&lt;/p>
&lt;p>The transition to Mode 3 — proactive, collaborative quality management — happened precisely because &amp;ldquo;quality issues became increasingly visible.&amp;rdquo; Not because someone wrote a strategy doc. Not because a VP mandated it. Because the problems became impossible to ignore.&lt;/p>
&lt;p>You can accelerate that transition. Create a Microsoft Teams channel. Call it &lt;code>#data-bugs&lt;/code> or &lt;code>#data-quality&lt;/code> or whatever feels natural. Start posting issues you find — with specifics. Not &amp;ldquo;the data looks wrong&amp;rdquo; but &amp;ldquo;fct_orders has 3,247 rows with null customer_id from the 2026-02-15 load, affecting the weekly revenue dashboard.&amp;rdquo;&lt;/p>
&lt;p>That&amp;rsquo;s it. You&amp;rsquo;ve just created the visibility mechanism that HelloFresh spent years arriving at.&lt;/p>
&lt;h4 id="the-broken-windows-effect-is-real--and-its-been-measured">The broken windows effect is real — and it&amp;rsquo;s been measured&lt;/h4>
&lt;p>The Pragmatic Programmer introduced the &amp;ldquo;broken windows&amp;rdquo; software analogy decades ago: &amp;ldquo;Don&amp;rsquo;t leave &amp;lsquo;broken windows&amp;rsquo; (bad designs, wrong decisions, or poor code) unrepaired. Fix each one as soon as it is discovered.&amp;rdquo;&lt;/p>
&lt;p>This isn&amp;rsquo;t just metaphor anymore. A 2024 arXiv paper by Diomidis Spinellis et al. analysed &lt;strong>2 million code commits&lt;/strong> across &lt;strong>122 projects&lt;/strong> comprising &lt;strong>5.5 million lines of code&lt;/strong>. The key finding: &amp;ldquo;History matters — developers behave differently depending on some aspects of the code quality they encounter.&amp;rdquo; Developers tailor the quality of their commits based on the quality of the file they&amp;rsquo;re committing to. Low quality begets lower quality. High quality attracts higher quality.&lt;/p>
&lt;p>Mat Ryer extends this to everyday engineering: &amp;ldquo;If you work on a project that has flaky tests, then you&amp;rsquo;re more likely to add more flaky tests. If there is a hacky design, you&amp;rsquo;re more likely to hack more in.&amp;rdquo;&lt;/p>
&lt;p>Fixing one visible data quality issue — or even just making it visible — can interrupt the decay spiral.&lt;/p>
&lt;h4 id="why-juniors-dont-speak-up-and-why-thats-exactly-the-problem">Why juniors don&amp;rsquo;t speak up (and why that&amp;rsquo;s exactly the problem)&lt;/h4>
&lt;p>Here&amp;rsquo;s the uncomfortable part. A Culture Shift UK survey found &lt;strong>junior colleagues were twice as likely (54%) as senior leaders (27%)&lt;/strong> to say speaking up about issues is &amp;ldquo;pointless.&amp;rdquo; 37% said speaking up &amp;ldquo;isn&amp;rsquo;t worth the personal risk.&amp;rdquo; A Blind survey found 58% of tech workers experience imposter syndrome.&lt;/p>
&lt;p>Google&amp;rsquo;s Project Aristotle studied 180+ teams over two years and found psychological safety was &amp;ldquo;by far the most important&amp;rdquo; of five dynamics for team effectiveness. Amy Edmondson&amp;rsquo;s foundational research contains a striking finding: while studying hospital teams, she expected high-performing teams to report fewer errors. Instead, &lt;strong>better teams reported MORE errors&lt;/strong>. Her insight: &amp;ldquo;Maybe the good teams don&amp;rsquo;t make more mistakes, maybe they report more.&amp;rdquo;&lt;/p>
&lt;p>This is the paradox. The teams that look like they have the most data quality issues are often the healthiest — because they&amp;rsquo;ve created an environment where problems surface instead of festering.&lt;/p>
&lt;p>Etsy pioneered blameless postmortems under CTO John Allspaw. Engineers send company-wide emails confessing mistakes. Etsy even gives out an annual &amp;ldquo;three-armed sweater&amp;rdquo; award to the employee who made the most surprising error. As a Hadley Wickham–curated Stanford reading list notes, while blameless postmortems are standard in DevOps, they &amp;ldquo;have not yet infiltrated standard data science practices.&amp;rdquo; That&amp;rsquo;s a gap you can help fill.&lt;/p>
&lt;p>Creating a &lt;code>#data-bugs&lt;/code> channel isn&amp;rsquo;t just about tracking issues. It&amp;rsquo;s about establishing the norm that &lt;strong>finding problems is good work&lt;/strong>, not a sign that something&amp;rsquo;s wrong with you.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="tactic-5-ship-an-example-dont-write-a-proposal">Tactic 5: ship an example, don&amp;rsquo;t write a proposal&lt;/h3>
&lt;br>
&lt;p>Marcus Blankenship&amp;rsquo;s article on learned helplessness in software engineering — 96,000+ reads, 900+ Reddit comments — contains a story about a coworker called Milind. Milind wasn&amp;rsquo;t a manager. He didn&amp;rsquo;t have a fancy title. But he was an informal leader through pure action. He asked questions about why particular approaches were taken. He admitted ignorance in group settings. He insisted the team understand root causes before moving on.&lt;/p>
&lt;p>Blankenship&amp;rsquo;s observation: &amp;ldquo;Milind changed our environment by his actions&amp;hellip; He showed me that I had much more power than I thought, if I would only stop expecting to be spoon-fed everything. His actions showed me that &lt;strong>true leadership isn&amp;rsquo;t granted, it&amp;rsquo;s grasped.&lt;/strong>&amp;rdquo;&lt;/p>
&lt;p>A junior engineer on Irina Stanescu&amp;rsquo;s newsletter shared something similar: &amp;ldquo;Even as a junior engineer, I remember influencing my chief architect and brought Terraform into my organisation 7 years ago by simply pointing out a common pain point and how much longer things took without managing our infrastructure in code.&amp;rdquo; They added: &amp;ldquo;Being able to influence without authority is a very underrated skill, and I believe this is what helped me get promoted to senior engineer even more than my technical skills.&amp;rdquo;&lt;/p>
&lt;p>The pattern is consistent. Don&amp;rsquo;t propose a data quality framework. Don&amp;rsquo;t write a 15-page RFC. Build the thing. Make it work. Show people.&lt;/p>
&lt;h4 id="why-working-examples-spread-faster-than-proposals">Why working examples spread faster than proposals&lt;/h4>
&lt;p>Everett Rogers&amp;rsquo; Diffusion of Innovations theory identifies five characteristics that determine how fast an idea spreads: relative advantage, compatibility, complexity (lower is better), &lt;strong>trialability&lt;/strong>, and &lt;strong>observability&lt;/strong>. A working example maximises both trialability and observability — the two factors most under your control as a junior engineer.&lt;/p>
&lt;p>A proposal says &amp;ldquo;we should do this.&amp;rdquo; A working example says &amp;ldquo;look, I already did this, and here&amp;rsquo;s what happened.&amp;rdquo; One requires people to imagine the benefit. The other puts the benefit directly in front of them.&lt;/p>
&lt;p>Addy Osmani, after 14 years at Google, captured the complementary lesson: &amp;ldquo;Early in my career, I believed great work would speak for itself. I was wrong. Code sits silently in a repository.&amp;rdquo; You need to build the example &lt;em>and&lt;/em> make its impact visible. Post in Microsoft Teams. Show the before and after. Quantify what changed.&lt;/p>
&lt;h4 id="the-junior-advantage-nobody-talks-about">The junior advantage nobody talks about&lt;/h4>
&lt;p>Ant Weiss&amp;rsquo;s widely-read article on organisational learned helplessness identifies common symptoms: &amp;ldquo;This is how we were told to work,&amp;rdquo; &amp;ldquo;These are the limitations of the system,&amp;rdquo; &amp;ldquo;We don&amp;rsquo;t have the resources.&amp;rdquo; His core observation: &amp;ldquo;No engineer starts their career willing to be unproductive, pessimistic and stressed out. But the companies they work at teach them to.&amp;rdquo;&lt;/p>
&lt;p>The cure, drawn from Seligman&amp;rsquo;s original research, is critical: &amp;ldquo;Threats, rewards, and observed demonstrations had no effect on the &amp;lsquo;helpless&amp;rsquo; dogs. The dogs had to be physically moved through the escape action at least twice before they would do it themselves.&amp;rdquo; You can&amp;rsquo;t just tell engineers things can be better. You have to walk them through the experience of making something better.&lt;/p>
&lt;p>Here&amp;rsquo;s the part that should give you confidence. JazzTeam reports a remarkable finding: &amp;ldquo;To prove that any problem can be solved, our team many times gave a task to Junior and even Intern Engineers who did not have psychological fears and tunnel thinking.&amp;rdquo; Juniors, unencumbered by years of &amp;ldquo;that&amp;rsquo;s impossible&amp;rdquo; conditioning, were able to tackle problems seniors had declared unsolvable.&lt;/p>
&lt;p>Your fresh eyes aren&amp;rsquo;t a weakness. They&amp;rsquo;re an asset. You haven&amp;rsquo;t yet learned that &amp;ldquo;it can&amp;rsquo;t be done.&amp;rdquo; So go do it.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="putting-it-all-together">Putting it all together&lt;/h3>
&lt;br>
&lt;p>None of these tactics require a committee meeting. None require architectural approval. None require you to be senior.&lt;/p>
&lt;p>Here&amp;rsquo;s what they do require: the willingness to act before you&amp;rsquo;re told to.&lt;/p>
&lt;p>Write a test this week. Add a column description tomorrow. Build a freshness check and schedule it. Create a Slack channel and post the first issue. Turn one of your quality checks into a reusable example your team can copy.&lt;/p>
&lt;p>DORA&amp;rsquo;s research — surveying 39,000+ professionals — finds that with focused effort and the right tooling, teams typically see measurable improvements within &lt;strong>1–3 months&lt;/strong>. A GitHub engineer who went from junior to mid-level in 2.5 years described the compound effect: &amp;ldquo;If I got stuck on some undocumented functionality, I made sure to update the docs and let the team know. Tackling a tricky bug requiring cross-team collaboration? I&amp;rsquo;d summarise everything we discovered so that it would be easier for others later. Post about it in Microsoft Teams and highlight its impact.&amp;rdquo;&lt;/p>
&lt;p>Kevin Workman, a former Google engineer, captured the mindset that makes all of this work: &amp;ldquo;Realising that I had agency and even a responsibility to advocate for my ideas is one of the most important lessons I learned on my journey to becoming a &amp;lsquo;senior&amp;rsquo; software engineer&amp;hellip; Some of the most interesting and most successful moments of my career have come from advocating for changes I thought nobody would agree to.&amp;rdquo;&lt;/p>
&lt;p>I think about Sam sometimes — that four-month-old team member whose instinct was to apologise for fixing something. The tests Sam added that day caught three data quality issues in the following month. Other engineers started adding their own tests alongside model changes. Six months later, the team had 200+ automated quality checks running on every pipeline, and the weekly &amp;ldquo;data looks wrong&amp;rdquo; Microsoft Teams messages had dropped to nearly zero.&lt;/p>
&lt;p>Sam never got a mandate. Never wrote a proposal. Never waited for Q3.&lt;/p>
&lt;p>The data quality problem in your organisation is a $12.9 million issue wearing a &amp;ldquo;we&amp;rsquo;ll get to it in Q3&amp;rdquo; disguise. And you — yes, you, the person who&amp;rsquo;s been here five months and noticed fourteen spellings of &amp;ldquo;active&amp;rdquo; — are exactly the right person to start fixing it.&lt;/p>
&lt;p>Not because someone gave you permission. Because the data doesn&amp;rsquo;t care about your title.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Quality</category><category>dbt</category><category>SQL</category><category>Testing</category><category>Documentation</category><category>Junior Engineer</category><category>Career Development</category><category>Psychological Safety</category></item><item><title>12 Steps to Better Data Engineering</title><link>https://ghostinthedata.info/posts/2026/2026-03-07-twelve-steps-to-better-data-engineering/</link><pubDate>Sat, 07 Mar 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-03-07-twelve-steps-to-better-data-engineering/</guid><author>Chris Hillman</author><description>A quick scoring framework to assess your data team's engineering maturity. Twelve yes-or-no questions, each with concrete benchmarks for what good and amazing look like using dbt, Snowflake, GitHub Actions, and AWS.</description><content:encoded>&lt;p>Let me tell you about the moment I stopped trusting architecture diagrams.&lt;/p>
&lt;p>I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you&amp;rsquo;d see in a data conference.&lt;/p>
&lt;p>Then I asked a question that changed everything: &amp;ldquo;Can you rebuild your finance table from scratch right now?&amp;rdquo;&lt;/p>
&lt;p>The room went quiet. One person started explaining that it was &amp;ldquo;mostly possible&amp;rdquo; but there were &amp;ldquo;a few manual steps&amp;rdquo; and &amp;ldquo;some seeds that someone uploads&amp;rdquo;. Someone else mentioned an incremental model that &amp;ldquo;hasn&amp;rsquo;t been full-refreshed in about eight months because last time it broke.&amp;rdquo;&lt;/p>
&lt;p>That gap — between the diagram on the wall and the reality in the warehouse — is where I&amp;rsquo;ve spent most of my career. And over the years, working across team after team, I started noticing something. The teams that were struggling weren&amp;rsquo;t missing knowledge. They had smart engineers who knew about testing, CI/CD, documentation, data contracts. They&amp;rsquo;d read the blog posts. They&amp;rsquo;d watched the conference talks. What they didn&amp;rsquo;t have was a way to honestly assess where they stood and what to fix first.&lt;/p>
&lt;p>I kept wishing someone would build a simple diagnostic. Maybe they just needed a handful of honest questions that would tell you, in ten minutes, whether your team was engineering or firefighting.&lt;/p>
&lt;p>The moment that turned wishing into doing came about six months later, with a team I was leading. We implemented just two practices — CI testing on pull requests and mandatory code review — and I watched the compound effect over twelve weeks. Our incidents dropped. The analysts stopped asking &amp;ldquo;which table do I use?&amp;rdquo; A new hire shipped a model on day four. One of the senior engineers pulled me aside and said, &amp;ldquo;I wish we&amp;rsquo;d known how bad things were before you got here. We thought we were fine because nothing was on fire.&amp;rdquo;&lt;/p>
&lt;p>Nothing was on fire. But everything was smouldering.&lt;/p>
&lt;p>That&amp;rsquo;s why I built this test. Not because the world needs another framework — it doesn&amp;rsquo;t. But because the gap between &lt;em>thinking you&amp;rsquo;re fine&lt;/em> and &lt;em>knowing where you stand&lt;/em> is where data teams lose months of momentum. And the only way to close that gap is to ask questions honest enough that the answers sting a little.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="the-twelve-questions">The twelve questions&lt;/h3>
&lt;br>
&lt;p>Before we get into the detail, here they are. Score yourself. One point for each &amp;ldquo;yes.&amp;rdquo;&lt;/p>
&lt;ol>
&lt;li>Can you rebuild any table from raw data in one command?&lt;/li>
&lt;li>Do you have a data catalog that people actually use?&lt;/li>
&lt;li>Can a new analyst find the data they need without asking you?&lt;/li>
&lt;li>Do you test transformations before deploying them?&lt;/li>
&lt;li>Do you fix data quality issues before building new pipelines?&lt;/li>
&lt;li>Do you have SLAs for your critical tables?&lt;/li>
&lt;li>Do you have a single source of truth for business definitions?&lt;/li>
&lt;li>Can you explain the lineage of any metric in under 2 minutes?&lt;/li>
&lt;li>Do data producers know when they break downstream consumers?&lt;/li>
&lt;li>Do you do code review on SQL and dbt models?&lt;/li>
&lt;li>Do new hires build a real pipeline in their first week?&lt;/li>
&lt;li>Do you regularly talk to the people who actually use your data?&lt;/li>
&lt;/ol>
&lt;p>&lt;strong>A score of 12 is unicorn territory.&lt;/strong> I&amp;rsquo;ve never seen it in the wild. Most teams I&amp;rsquo;ve worked with score 3 or 4. Anything below 6 and you&amp;rsquo;re not doing data engineering — you&amp;rsquo;re doing data triage.&lt;/p>
&lt;p>Now let&amp;rsquo;s break each one down. For every question, I&amp;rsquo;ll show you what &lt;em>good&lt;/em> looks like and what &lt;em>amazing&lt;/em> looks like — with concrete implementations using dbt, Snowflake, GitHub Actions, and AWS. Because &amp;ldquo;yes&amp;rdquo; isn&amp;rsquo;t binary in practice. There&amp;rsquo;s a massive gap between &amp;ldquo;sort of&amp;rdquo; and &amp;ldquo;absolutely.&amp;rdquo;&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="1-can-you-rebuild-any-table-from-raw-data-in-one-command">1. Can you rebuild any table from raw data in one command?&lt;/h3>
&lt;br>
&lt;p>This is the foundation. If you can&amp;rsquo;t reproduce your outputs from your inputs, you don&amp;rsquo;t have a pipeline — you have a prayer.&lt;/p>
&lt;p>The core principle here is &lt;strong>idempotency&lt;/strong>: running the same operation twice produces identical results. dbt is designed around this idea, but achieving it requires deliberate choices, especially with incremental models. Too many teams build incremental models that silently drift from what a full refresh would produce, and they don&amp;rsquo;t discover the discrepancy until something breaks spectacularly.&lt;/p>
&lt;h4 id="what-good-looks-like">What good looks like&lt;/h4>
&lt;p>Individual tables rebuild cleanly with &lt;code>dbt run --full-refresh --select model_name&lt;/code>. Your incremental models are tested against full refreshes during development. You&amp;rsquo;ve configured &lt;code>on_schema_change: sync_all_columns&lt;/code> so schema evolution doesn&amp;rsquo;t silently break things. And you&amp;rsquo;ve got a CI pipeline in GitHub Actions that runs &lt;code>dbt build&lt;/code> on pull requests.&lt;/p>
&lt;p>That&amp;rsquo;s good. That&amp;rsquo;s better than most. But it&amp;rsquo;s still table-by-table.&lt;/p>
&lt;h4 id="what-amazing-looks-like">What amazing looks like&lt;/h4>
&lt;p>Your entire warehouse rebuilds from raw data using a &lt;strong>Write-Audit-Publish (WAP) pattern&lt;/strong>. If you&amp;rsquo;re not familiar with WAP, I wrote a &lt;a href="https://ghostinthedata.info/posts/2025/2025-05-18-wap-data-pipelines/" target="_blank" rel="noopener">deep dive on implementing it with Airflow&lt;/a> — the core idea is that new data gets written to a staging environment, audited against quality checks, and only published to production once it passes. It&amp;rsquo;s the data engineering equivalent of a preflight checklist: nothing reaches consumers until it&amp;rsquo;s been verified. And if you&amp;rsquo;re on Snowflake with Iceberg tables, the game has changed — &lt;a href="https://ghostinthedata.info/posts/2026/2026-02-27-wap-iceberg-branching/" target="_blank" rel="noopener">WAP with Iceberg branching&lt;/a> gives you git-like isolation at the table level, which means you can stage, audit, and publish without creating separate schemas or databases at all.&lt;/p>
&lt;p>The practical implementation: dbt always targets a staging schema or database (or an Iceberg branch), runs all models and tests, and only on success does the data get promoted to production. Combined with Snowflake&amp;rsquo;s &lt;strong>zero-copy cloning&lt;/strong>, this becomes practical even at scale — cloning a multi-terabyte database takes seconds and costs nothing in storage. You&amp;rsquo;re not duplicating data; you&amp;rsquo;re creating metadata pointers.&lt;/p>
&lt;p>And when things &lt;em>do&lt;/em> go wrong — and they will — the question becomes how quickly you can recover. If your incremental models have drifted, or a source system silently changed its schema three weeks ago, you need a backfill strategy that doesn&amp;rsquo;t involve rebuilding everything from scratch. I wrote about the &lt;a href="https://ghostinthedata.info/posts/2026/2026-02-07-self-healing/" target="_blank" rel="noopener">pitfalls of day-by-day backfills and how to heal tables properly&lt;/a> — it&amp;rsquo;s the companion piece to idempotency, because reproducibility isn&amp;rsquo;t just about building forward. It&amp;rsquo;s about being able to go back.&lt;/p>
&lt;p>The really mature teams take it further with &lt;strong>Slim CI&lt;/strong>. Instead of rebuilding everything on every PR, they store the production &lt;code>manifest.json&lt;/code> in S3 and compare it against the new build. Only modified models and their downstream dependencies get rebuilt:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># .github/workflows/dbt-ci.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">on&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pull_request&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">types&lt;/span>: [&lt;span style="color:#ae81ff">opened, reopened, synchronize]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">steps&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Download production manifest&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">aws s3 cp s3://dbt-artifacts/manifest.json ./state/manifest.json&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Run dbt build (modified models only)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">dbt build -s &amp;#39;state:modified+&amp;#39; --defer --state ./state --target ci&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>One team I worked with reported a &lt;strong>30% reduction in compute costs&lt;/strong> just by splitting their monolithic CI job into targeted workflows.&lt;/p>
&lt;blockquote>
&lt;p>&lt;strong>Watch out for this:&lt;/strong> Protect massive tables with &lt;code>full_refresh = false&lt;/code> in your dbt config. One accidental &lt;code>--full-refresh&lt;/code> on a billion-row fact table can ruin your morning. Override it with a variable when you actually mean it: &lt;code>full_refresh = var(&amp;quot;force_full_refresh&amp;quot;, false)&lt;/code>.&lt;/p>&lt;/blockquote>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="2-do-you-have-a-data-catalog-that-people-actually-use">2. Do you have a data catalog that people actually use?&lt;/h3>
&lt;br>
&lt;p>The emphasis is on &amp;ldquo;actually use.&amp;rdquo; I&amp;rsquo;ve lost count of how many teams have shown me a data catalog, only for me to discover that nobody&amp;rsquo;s opened it in months. The data catalog is the gym membership of the data world — everyone has one, nobody goes.&lt;/p>
&lt;p>The failure mode is almost always the same: a team selects a tool, loads metadata, declares victory, and moves on. Six months later it&amp;rsquo;s a ghost town of stale descriptions and orphaned tables.&lt;/p>
&lt;h4 id="what-good-looks-like-1">What good looks like&lt;/h4>
&lt;p>dbt Docs generated and hosted somewhere accessible. Your major tables have descriptions in &lt;code>schema.yml&lt;/code>. The lineage graph exists and gets pulled up occasionally during incident response or onboarding. It&amp;rsquo;s not perfect, but it&amp;rsquo;s there.&lt;/p>
&lt;h4 id="what-amazing-looks-like-1">What amazing looks like&lt;/h4>
&lt;p>The catalog is the &lt;strong>default entry point&lt;/strong> for data questions — not Microsoft Teams, not the data engineer sitting three desks over.&lt;/p>
&lt;p>Spotify&amp;rsquo;s internal tool Lexikon pushed data scientist adoption from 75% to 95% by doing something clever: they added &lt;em>personalised dataset recommendations&lt;/em>, people/team pages showing who uses each dataset, common joins, and popular fields. It became a top-five internal tool because it was genuinely faster than asking a colleague.&lt;/p>
&lt;p>Airbnb&amp;rsquo;s Metis serves over 1,000 data users weekly. Their secret? A Google-like search interface that works for every skill level, from SQL-fluent engineers to product managers who&amp;rsquo;ve never written a query.&lt;/p>
&lt;p>The pattern that actually works for smaller teams: author descriptions in whatever UI is easiest, auto-generate a PR back to dbt&amp;rsquo;s &lt;code>schema.yml&lt;/code> daily, review in Git. This keeps your definitions version-controlled without making people learn Git just to document a table.&lt;/p>
&lt;p>And here&amp;rsquo;s the leading indicator that your catalog is working: &lt;strong>track the decline in Microsoft Teams questions about data.&lt;/strong> Productboard used exactly this metric. Fewer &amp;ldquo;which table do I use for revenue?&amp;rdquo; messages meant the catalog was earning its keep.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="3-can-a-new-analyst-find-the-data-they-need-without-asking-you">3. Can a new analyst find the data they need without asking you?&lt;/h3>
&lt;br>
&lt;p>This is the catalog question&amp;rsquo;s evil twin. A catalog helps, but self-service discovery is really about the cumulative effect of naming conventions, documentation-as-code, semantic layers, and project structure.&lt;/p>
&lt;p>dbt Labs published guidance that should be printed and taped to every data engineer&amp;rsquo;s monitor: &lt;strong>&amp;ldquo;Assume your end-user will have no other context than the model name.&amp;rdquo;&lt;/strong> Model names persist across databases, BI tools, DAGs, and docs. Folder names and schemas don&amp;rsquo;t follow the data the same way.&lt;/p>
&lt;h4 id="what-good-looks-like-2">What good looks like&lt;/h4>
&lt;p>You&amp;rsquo;re using the canonical dbt naming convention: &lt;code>stg_&lt;/code> for staging, &lt;code>int_&lt;/code> for intermediate, &lt;code>fct_&lt;/code> for facts, &lt;code>dim_&lt;/code> for dimensions. Your project follows the three-layer structure — staging (1:1 with sources, materialised as views), intermediate (composable business logic), and marts (final business-ready datasets organised by domain). A reasonably technical person can navigate the DAG and find what they need within ten minutes.&lt;/p>
&lt;h4 id="what-amazing-looks-like-2">What amazing looks like&lt;/h4>
&lt;p>A new analyst finds and understands any dataset &lt;strong>within minutes&lt;/strong>, without messaging anyone.&lt;/p>
&lt;p>Every model has both table-level and column-level descriptions. Schema field search works across all tables — &amp;ldquo;find every table with &lt;code>customer_id&lt;/code>&amp;rdquo; returns results instantly. Personalised dataset recommendations surface relevant tables based on role.&lt;/p>
&lt;p>And the crown jewel: a &lt;strong>semantic layer&lt;/strong> that lets business users query metrics in plain language without writing SQL. The dbt Semantic Layer (powered by MetricFlow) centralises metric definitions in YAML alongside your models — entities, dimensions, and measures declared once, then consumed via API in Tableau, Power BI, Looker, Python notebooks, and increasingly, AI agents.&lt;/p>
&lt;p>Bilt Rewards reported an 80% decrease in data costs after implementing it for embedded analytics. And dbt Labs&amp;rsquo; testing showed 83% of natural-language questions answered correctly when routed through the semantic layer. That last number matters more than you&amp;rsquo;d think — it&amp;rsquo;s the difference between AI tooling that actually works and AI tooling that confidently gives the wrong answer.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="4-do-you-test-transformations-before-deploying-them">4. Do you test transformations before deploying them?&lt;/h3>
&lt;br>
&lt;p>This is probably the single highest-impact practice on the list. If I could only get a team to adopt one of these twelve, it would be this one.&lt;/p>
&lt;p>But here&amp;rsquo;s what most teams get wrong: they think &amp;ldquo;testing&amp;rdquo; means slapping &lt;code>not_null&lt;/code> on a few columns and calling it done. Real testing means understanding the &lt;a href="https://ghostinthedata.info/posts/2025/2025-11-24-data-quality-framework/" target="_blank" rel="noopener">dimensions of data quality&lt;/a> — completeness, uniqueness, timeliness, validity, accuracy, consistency — and building checks that cover each one deliberately. I&amp;rsquo;ve written about &lt;a href="https://ghostinthedata.info/posts/2025/2025-11-22-data-quality/" target="_blank" rel="noopener">why data quality is a deeper problem than most teams realise&lt;/a>, and the short version is this: testing in CI is where quality becomes a habit rather than a hope. But only if your tests are actually measuring the things that matter.&lt;/p>
&lt;p>The consensus across every practitioner I&amp;rsquo;ve studied is clear: &lt;strong>CI for dbt is the most impactful thing you can do for data quality.&lt;/strong> Three complementary testing layers form a comprehensive strategy: generic dbt tests (&lt;code>not_null&lt;/code>, &lt;code>unique&lt;/code>, &lt;code>accepted_values&lt;/code>, &lt;code>relationships&lt;/code>), unit tests (introduced in dbt v1.8 for testing complex business logic with static inputs), and data diffing for value-level comparison between production and development.&lt;/p>
&lt;h4 id="what-good-looks-like-3">What good looks like&lt;/h4>
&lt;p>Basic dbt tests on key models. Tests run as part of scheduled production jobs using &lt;code>dbt build&lt;/code> (which runs tests immediately after each model, not in a separate pass). Pull requests exist for dbt changes, and someone eyeballs them before merging.&lt;/p>
&lt;h4 id="what-amazing-looks-like-3">What amazing looks like&lt;/h4>
&lt;p>Full Slim CI with per-PR isolated Snowflake schemas, data diffing integrated into PR comments, and automated linting that catches style issues before a human ever looks at the code.&lt;/p>
&lt;p>The numbers from real teams are staggering. Thumbtack — 50-plus analysts, five data engineers, over 100 PRs per month — previously spent one to two hours manually validating each pull request with SQL queries and spreadsheets. After integrating data diffing into their GitHub CI pipeline, they saved over 200 hours per month. That&amp;rsquo;s not a rounding error. That&amp;rsquo;s a full-time engineer&amp;rsquo;s worth of capacity recovered.&lt;/p>
&lt;p>Dutchie caught timezone corruption on &lt;code>created_at&lt;/code> fields and case-when logic errors that were silently shifting 20% of data between columns. Nutrafol caught a transformation that would have shown net revenue plummeting — before it reached production and before the CFO&amp;rsquo;s Monday morning dashboard refreshed.&lt;/p>
&lt;p>Zscaler went even further. They built PRISM, a multi-agent AI PR review system that reduced manual review time by 90% — auto-approving conformant PRs and posting targeted comments on complex logic changes.&lt;/p>
&lt;p>Here&amp;rsquo;s what the mature pipeline looks like in practice:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># On every PR: lint, compile, build modified models, diff against prod&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">steps&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">SQLFluff lint&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">sqlfluff lint models/ --dialect snowflake&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">dbt compile&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">dbt compile&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Slim CI build&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">dbt build -s &amp;#39;state:modified+&amp;#39; --defer --state ./state --target ci&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Data Diff&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: &lt;span style="color:#ae81ff">datafold ci submit --ci-config-id ${{ secrets.DATAFOLD_CI_CONFIG }}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="5-do-you-fix-data-quality-issues-before-building-new-pipelines">5. Do you fix data quality issues before building new pipelines?&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s an uncomfortable stat: Monte Carlo&amp;rsquo;s 2023 State of Data Quality report found that &lt;strong>74% of organisations reported that business stakeholders identify data quality issues first&lt;/strong> — up from 47% the prior year. Most data teams learn about broken data from the people who are supposed to trust it. That&amp;rsquo;s not a technology problem. That&amp;rsquo;s a culture problem.&lt;/p>
&lt;h4 id="what-good-looks-like-4">What good looks like&lt;/h4>
&lt;p>Basic dbt tests catch obvious issues. Tests run in production jobs. The team has an informal sense of which data matters most, usually earned through painful experience — someone got burned by bad revenue numbers in an executive review, and now those tables have tests.&lt;/p>
&lt;h4 id="what-amazing-looks-like-4">What amazing looks like&lt;/h4>
&lt;p>Data quality SLAs enforced with &lt;strong>error budgets&lt;/strong>, borrowed from the SRE playbook. A 99.5% data availability target allows approximately 3.6 hours of acceptable downtime per month. When the error budget is consumed, reliability work takes priority over feature delivery. Full stop. No new pipelines until you&amp;rsquo;ve earned back your quality margin.&lt;/p>
&lt;p>Warner Bros. Discovery created something they call a &lt;strong>Data Quality Forum&lt;/strong> — not a reactive incident-review meeting, but an operational bridge between teams. They used observability tooling to surface anomalies early, developed a priority matrix (P0/P1/P2), and made the forum&amp;rsquo;s patterns part of new hire onboarding. During Olympics livestreaming, custom SQL checks detected missing content metadata before it could break reporting.&lt;/p>
&lt;p>Snowflake now offers native &lt;strong>Data Metric Functions&lt;/strong> — &lt;code>FRESHNESS&lt;/code>, &lt;code>NULL_COUNT&lt;/code>, &lt;code>DUPLICATE_COUNT&lt;/code>, &lt;code>UNIQUE_COUNT&lt;/code>, &lt;code>ROW_COUNT&lt;/code> — that can be scheduled to run on DML changes or time intervals. Results land in &lt;code>SNOWFLAKE.LOCAL.DATA_QUALITY_MONITORING_RESULTS&lt;/code>. It&amp;rsquo;s not a replacement for dbt tests, but it provides a warehouse-level safety net that catches issues your transformation layer might miss.&lt;/p>
&lt;p>The cultural shift matters more than the tooling. HelloFresh&amp;rsquo;s VP of Data drove a transformation through three stages: ad-hoc individual fixes, organised cleanup by a central team, and finally proactive quality at the source. The final stage required embedding data product owners within business domains and running data literacy programs. It&amp;rsquo;s not glamorous work. But it&amp;rsquo;s the work that changes outcomes.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="6-do-you-have-slas-for-your-critical-tables">6. Do you have SLAs for your critical tables?&lt;/h3>
&lt;br>
&lt;p>&amp;ldquo;The dashboard is stale&amp;rdquo; is the Slack message that launches a thousand fire drills. SLAs turn that reactive scramble into a measured, prioritised response.&lt;/p>
&lt;p>The breakthrough insight from practitioners: &lt;strong>don&amp;rsquo;t aim for 100%.&lt;/strong> A 99.5% target gives you a realistic buffer for maintenance, edge cases, and the occasional Snowflake service hiccup. Different data products warrant different targets — which means you need a tiered classification system.&lt;/p>
&lt;h4 id="what-good-looks-like-5">What good looks like&lt;/h4>
&lt;p>Your team knows which tables are critical, usually because they&amp;rsquo;ve been burned before. Some &lt;code>dbt source freshness&lt;/code> checks are running. Basic Slack alerts fire when jobs fail. It&amp;rsquo;s reactive, but at least you know when things break.&lt;/p>
&lt;h4 id="what-amazing-looks-like-5">What amazing looks like&lt;/h4>
&lt;p>A formal tiered classification published in the data catalog with specific, measurable commitments:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Tier 1 (Gold):&lt;/strong> ML systems, revenue reporting → PagerDuty on-call, must have &lt;code>unique&lt;/code> + &lt;code>not_null&lt;/code> tests + assigned owner, freshness SLA under 4 hours&lt;/li>
&lt;li>&lt;strong>Tier 2 (Silver):&lt;/strong> Executive dashboards, KPI reports → Slack team channel alerts, owner assigned, freshness SLA under 12 hours&lt;/li>
&lt;li>&lt;strong>Tier 3 (Bronze):&lt;/strong> Ad-hoc analytics, exploration tables → weekly digest, freshness SLA under 24 hours&lt;/li>
&lt;/ul>
&lt;p>The implementation lives in dbt &lt;code>meta&lt;/code> config:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">fct_revenue&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">meta&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">owner&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;finance-data-team&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">criticality&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;tier_1&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">sla_freshness&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;4 hours&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tests&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">unique&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">column_name&lt;/span>: &lt;span style="color:#ae81ff">order_id&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">not_null&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">column_name&lt;/span>: &lt;span style="color:#ae81ff">order_id&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>SLA compliance dashboards track attainment percentage, breach counts, and trends over time. When the error budget is exhausted, feature development freezes — same principle as practice #5. Dedicated Snowflake warehouses isolate critical workloads from exploratory queries so that someone&amp;rsquo;s &lt;code>SELECT *&lt;/code> on a billion-row table doesn&amp;rsquo;t cause your revenue report to miss its 8 AM deadline.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="7-do-you-have-a-single-source-of-truth-for-business-definitions">7. Do you have a single source of truth for business definitions?&lt;/h3>
&lt;br>
&lt;p>Everyone&amp;rsquo;s had this conversation: &amp;ldquo;Which revenue number is right?&amp;rdquo; Two dashboards, two numbers, two teams who each think the other is wrong. This is what happens when metric definitions live inside BI tools, tribal knowledge, and the head of that one analyst who&amp;rsquo;s been here since 2019. If you&amp;rsquo;ve ever worked with dimensional models, you&amp;rsquo;ll recognise this as the conformed dimension problem — I covered why &lt;a href="https://ghostinthedata.info/posts/2025/2025-11-07-effective-data-modelling/" target="_blank" rel="noopener">dimensional modelling still matters&lt;/a> and how conformed dimensions are the integration backbone that prevents exactly this kind of mess.&lt;/p>
&lt;h4 id="what-good-looks-like-6">What good looks like&lt;/h4>
&lt;p>Key metrics are defined in dbt docs or a wiki. The team knows which mart table is the canonical source for revenue, orders, or whatever your core entities are. When someone asks &amp;ldquo;which table do I use?&amp;rdquo;, there&amp;rsquo;s a consistent answer — even if it&amp;rsquo;s only communicated verbally.&lt;/p>
&lt;h4 id="what-amazing-looks-like-6">What amazing looks like&lt;/h4>
&lt;p>A centralised &lt;strong>semantic layer&lt;/strong> enforced as the only path to metrics. All definitions are version-controlled in dbt YAML, reviewed via PR, and validated in CI.&lt;/p>
&lt;p>Here&amp;rsquo;s what a metric definition looks like in MetricFlow:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">metrics&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">cancellation_rate&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">description&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Percentage of orders cancelled within 24 hours of placement&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">type&lt;/span>: &lt;span style="color:#ae81ff">ratio&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">type_params&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">numerator&lt;/span>: &lt;span style="color:#ae81ff">cancellations&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">denominator&lt;/span>: &lt;span style="color:#ae81ff">order_total&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">filter&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> {{ TimeDimension(&amp;#39;metric_time&amp;#39;, &amp;#39;day&amp;#39;) }}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That definition gets consumed identically whether someone queries it from Tableau, Power BI, a Python notebook, or an AI agent. One definition. One answer. Everywhere.&lt;/p>
&lt;p>Whatnot — the livestream marketplace growing at breakneck speed — solved a different version of this problem. They used Protobuf schemas as a single source of truth for event definitions, consolidating hundreds of chaotic Snowflake tables down to two clean &amp;ldquo;exposure&amp;rdquo; tables: &lt;code>backend_events&lt;/code> and &lt;code>frontend_events&lt;/code>. Brutal simplification. But it worked because everyone could find the data.&lt;/p>
&lt;p>&lt;code>dbt sl validate&lt;/code> in CI catches breaking changes to semantic definitions before merge. That&amp;rsquo;s the key differentiator — your metric definitions aren&amp;rsquo;t just documented, they&amp;rsquo;re &lt;em>enforced&lt;/em>.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="8-can-you-explain-the-lineage-of-any-metric-in-under-2-minutes">8. Can you explain the lineage of any metric in under 2 minutes?&lt;/h3>
&lt;br>
&lt;p>When your CFO asks &amp;ldquo;where does this number come from?&amp;rdquo;, you need an answer faster than &amp;ldquo;let me check and get back to you.&amp;rdquo; Two minutes is generous. In most incident situations, you&amp;rsquo;ve got about thirty seconds before people start making assumptions.&lt;/p>
&lt;h4 id="what-good-looks-like-7">What good looks like&lt;/h4>
&lt;p>The dbt DAG is viewable, hosted on S3 or an internal site. Your team can trace major mart tables back to their sources. During incident response, someone pulls up the lineage graph and walks through the chain. It takes some squinting, but the information is there.&lt;/p>
&lt;h4 id="what-amazing-looks-like-7">What amazing looks like&lt;/h4>
&lt;p>&lt;strong>Column-level lineage&lt;/strong> automated across the full stack — from source systems through dbt transformations to the BI dashboards consumers see.&lt;/p>
&lt;p>dbt Cloud provides column-level lineage showing whether each column is &amp;ldquo;transformed&amp;rdquo; versus &amp;ldquo;passthrough/rename.&amp;rdquo; On the Snowflake side, the &lt;code>ACCESS_HISTORY&lt;/code> view (Enterprise Edition) tracks both read and write operations with column-level mappings. Snowflake Labs published an open-source adapter that converts &lt;code>ACCESS_HISTORY&lt;/code> data into OpenLineage JSON format, making cross-platform lineage possible.&lt;/p>
&lt;p>But here&amp;rsquo;s where it gets practical. The open-source tool &lt;strong>Recce&lt;/strong> compares two dbt environments and produces a &amp;ldquo;Lineage Diff&amp;rdquo; in CI. It categorises changes as breaking, partial-breaking, or non-breaking. PR reviewers get an instant risk assessment: &amp;ldquo;this change affects 47 downstream models, including three Tier 1 dashboards&amp;rdquo; versus &amp;ldquo;this change is isolated to a staging model with no downstream consumers.&amp;rdquo;&lt;/p>
&lt;p>PRs get annotated automatically with impact analysis. Cross-project lineage via dbt Mesh connects multiple dbt projects. Lineage isn&amp;rsquo;t a pretty graph that nobody looks at — it&amp;rsquo;s used daily for onboarding, incident response, impact analysis, and cost optimisation.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="9-do-data-producers-know-when-they-break-downstream-consumers">9. Do data producers know when they break downstream consumers?&lt;/h3>
&lt;br>
&lt;p>This is where data engineering starts borrowing seriously from software engineering. In a microservices world, you wouldn&amp;rsquo;t deploy an API change without knowing who calls your endpoint. But in data, teams routinely change source schemas, modify column semantics, or sunset tables with zero awareness of who&amp;rsquo;s consuming them downstream.&lt;/p>
&lt;p>&lt;strong>Data contracts&lt;/strong> change that dynamic.&lt;/p>
&lt;h4 id="what-good-looks-like-8">What good looks like&lt;/h4>
&lt;p>dbt sources are defined with &lt;code>dbt source freshness&lt;/code> running and alerting. The team is aware of major upstream dependencies. When Fivetran&amp;rsquo;s sync breaks or a source system changes its schema, someone notices within a few hours — usually because a test fails.&lt;/p>
&lt;h4 id="what-amazing-looks-like-8">What amazing looks like&lt;/h4>
&lt;p>dbt &lt;strong>model contracts&lt;/strong> (v1.5+) enforced on all gold-layer and public models:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">models&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">dim_users&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">config&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">contract&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">enforced&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">columns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">user_id&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">data_type&lt;/span>: &lt;span style="color:#ae81ff">int&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">constraints&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">type&lt;/span>: &lt;span style="color:#ae81ff">primary_key&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">email&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">data_type&lt;/span>: &lt;span style="color:#ae81ff">varchar&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">constraints&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">type&lt;/span>: &lt;span style="color:#ae81ff">not_null&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This is a &lt;strong>preflight check&lt;/strong>: dbt verifies the model&amp;rsquo;s compiled SQL returns columns matching the contract before building. Breaking changes — removed columns, changed data types, modified constraints — are caught by &lt;code>state:modified&lt;/code> in CI. Model access levels (&lt;code>public&lt;/code>, &lt;code>protected&lt;/code>, &lt;code>private&lt;/code>) control cross-project visibility.&lt;/p>
&lt;p>Whatnot went further with Protobuf schemas enforced via Buf linter in CI. Every event producer runs against a common testing harness. Post-deployment monitoring catches semantic drift — when a field still exists but its meaning has changed.&lt;/p>
&lt;blockquote>
&lt;p>&lt;strong>Gotcha worth knowing:&lt;/strong> dbt contracts validate the compiled SQL, not the actual Snowflake table. If tools like Fivetran add columns directly to your raw layer, dbt won&amp;rsquo;t know until the next run. Monitor schema changes externally using Snowflake&amp;rsquo;s &lt;code>INFORMATION_SCHEMA&lt;/code> or &lt;code>ACCESS_HISTORY&lt;/code> to catch drift between runs.&lt;/p>&lt;/blockquote>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="10-do-you-do-code-review-on-sql-and-dbt-models">10. Do you do code review on SQL and dbt models?&lt;/h3>
&lt;br>
&lt;p>I&amp;rsquo;m constantly surprised by how many data teams still deploy SQL changes without review. In software engineering, unreviewed code going to production would be considered reckless. In data engineering, it&amp;rsquo;s Tuesday.&lt;/p>
&lt;h4 id="what-good-looks-like-9">What good looks like&lt;/h4>
&lt;p>Pull requests are required for all dbt changes — branch protection is enabled on main. At least one human reviewer looks at each PR. There&amp;rsquo;s a basic PR description explaining what changed and why.&lt;/p>
&lt;h4 id="what-amazing-looks-like-9">What amazing looks like&lt;/h4>
&lt;p>Automated CI running SQLFluff lint + &lt;code>dbt compile&lt;/code> + &lt;code>dbt build&lt;/code> + data diff on every PR. Pre-commit hooks catch issues before code is even committed. PR templates with structured sections: description, linked tickets, impact zone, testing evidence.&lt;/p>
&lt;p>The recommended pre-commit stack combines SQLFluff for linting with dbt-checkpoint for structural validation:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># .pre-commit-config.yaml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">repos&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">repo&lt;/span>: &lt;span style="color:#ae81ff">https://github.com/sqlfluff/sqlfluff&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">hooks&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">id&lt;/span>: &lt;span style="color:#ae81ff">sqlfluff-lint&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">args&lt;/span>: [--&lt;span style="color:#ae81ff">dialect, snowflake]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">repo&lt;/span>: &lt;span style="color:#ae81ff">https://github.com/dbt-checkpoint/dbt-checkpoint&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">hooks&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">id&lt;/span>: &lt;span style="color:#ae81ff">check-model-has-tests&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">id&lt;/span>: &lt;span style="color:#ae81ff">check-model-columns-have-desc&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">id&lt;/span>: &lt;span style="color:#ae81ff">check-source-has-freshness&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That last hook — &lt;code>check-model-columns-have-desc&lt;/code> — is quietly revolutionary. It means documentation isn&amp;rsquo;t optional. You literally cannot merge a model without column descriptions. The &amp;ldquo;we&amp;rsquo;ll document it later&amp;rdquo; excuse dies right there in the CI pipeline.&lt;/p>
&lt;p>Surfline (700+ dbt models) reported that after integrating SQLFluff, SQL consistency improved dramatically and reviewer burden dropped. New engineers learned &amp;ldquo;good SQL&amp;rdquo; from day one because the linter enforced it automatically. Another team, Markerr, found that automated style enforcement freed up review time to focus on deeper logic questions rather than arguing about capitalisation and trailing commas.&lt;/p>
&lt;p>The &lt;code>dbt_project_evaluator&lt;/code> package is worth mentioning too — it audits your entire DAG structure against dbt Labs&amp;rsquo; published best practices. Run it in CI and it&amp;rsquo;ll flag things like models that reference sources directly (bypassing staging), duplicate sources, or models with no tests.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="11-do-new-hires-build-a-real-pipeline-in-their-first-week">11. Do new hires build a real pipeline in their first week?&lt;/h3>
&lt;br>
&lt;p>The speed at which a new hire becomes productive tells you everything about the state of your documentation, tooling, and team culture. If it takes three weeks before someone can make a meaningful contribution, you don&amp;rsquo;t have an onboarding problem — you have a platform problem. I wrote a &lt;a href="https://ghostinthedata.info/posts/2025/2025-02-23-landed-the-role/" target="_blank" rel="noopener">guide to navigating your first 90 days as a data engineer&lt;/a> — and the teams that make those first 90 days count are almost always the ones who&amp;rsquo;ve invested in the infrastructure described here.&lt;/p>
&lt;h4 id="what-good-looks-like-10">What good looks like&lt;/h4>
&lt;p>New hire has access to tools within a day or two. Some documentation exists. A buddy or mentor is assigned. They can run the dbt project locally by end of week one, even if they haven&amp;rsquo;t contributed anything yet.&lt;/p>
&lt;h4 id="what-amazing-looks-like-10">What amazing looks like&lt;/h4>
&lt;p>Pre-arrival setup is complete before Day 1: machine provisioned, logins ready, Snowflake roles assigned, dbt Cloud account active. No engineer should spend their first morning installing things.&lt;/p>
&lt;p>Each developer gets a &lt;strong>personal Snowflake sandbox&lt;/strong> — &lt;code>dbt_&amp;lt;username&amp;gt;&lt;/code> schema in the dev database, with write access only to dev and read-only access to raw/staging. Snowflake&amp;rsquo;s zero-copy cloning makes production data available instantly without duplicating storage costs. Productboard open-sourced &lt;code>dbt-snowflake-sandbox&lt;/code> — a set of dbt macros that create isolated sandboxes by cloning only the specific model dependencies a developer needs.&lt;/p>
&lt;p>The first-week project is structured and progressive:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Day 1-2:&lt;/strong> Run the existing dbt project locally. Explore the DAG. Read the key model documentation.&lt;/li>
&lt;li>&lt;strong>Day 3:&lt;/strong> Add a new source or staging model for a real (but low-risk) dataset.&lt;/li>
&lt;li>&lt;strong>Day 4:&lt;/strong> Build a staging model with tests and documentation.&lt;/li>
&lt;li>&lt;strong>Day 5:&lt;/strong> Open a PR, go through the review process, and get it merged.&lt;/li>
&lt;/ol>
&lt;p>By Friday, they&amp;rsquo;ve shipped something real. They understand the workflow. They&amp;rsquo;ve experienced CI, code review, and deployment. And critically, they &lt;em>feel&lt;/em> like a contributor rather than a tourist.&lt;/p>
&lt;p>7shifts estimated cutting onboarding time by over a week per new hire just by having documentation and data questions accessible in their data catalog.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="12-do-you-regularly-talk-to-the-people-who-actually-use-your-data">12. Do you regularly talk to the people who actually use your data?&lt;/h3>
&lt;br>
&lt;p>This is the question that separates data teams who build for their portfolio from data teams who build for their business. You can nail every technical practice on this list and still fail if you&amp;rsquo;re building the wrong things. I&amp;rsquo;ve written before about &lt;a href="https://ghostinthedata.info/posts/2025/2025-02-15-data-impact/" target="_blank" rel="noopener">how to maximise your data team&amp;rsquo;s impact&lt;/a> — and the through-line is always the same: the teams that create real value are the ones who stay close to the people consuming their work.&lt;/p>
&lt;h4 id="what-good-looks-like-11">What good looks like&lt;/h4>
&lt;p>A Slack channel exists for data questions. Communication happens when issues arise. There are occasional stakeholder meetings, usually prompted by something breaking or a new request coming in.&lt;/p>
&lt;h4 id="what-amazing-looks-like-11">What amazing looks like&lt;/h4>
&lt;p>&lt;strong>Data office hours.&lt;/strong> Weekly or bi-weekly open sessions where anyone in the organisation can bring data questions. Not a presentation. Not a status update. An open door.&lt;/p>
&lt;p>Holistics published a detailed playbook for running what they call &amp;ldquo;data clinics.&amp;rdquo; The core principles: teach, don&amp;rsquo;t serve. Show business users how to self-serve rather than just answering their question and sending them on their way. Montreal Analytics reported that after implementing data clinics, the number of self-serve business users on their BI tool grew tenfold. &lt;em>Tenfold.&lt;/em>&lt;/p>
&lt;p>Pair that with a monthly &lt;strong>Data NPS survey&lt;/strong> — a single question: &amp;ldquo;How likely are you to recommend our data team&amp;rsquo;s products to a colleague?&amp;rdquo; It sounds corporate, but without this metric, you have no way to quantify whether your consumers are satisfied or just silently building workarounds in Excel.&lt;/p>
&lt;p>dbt &lt;strong>exposures&lt;/strong> formalise the connection between data models and their consumers:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">exposures&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">weekly_revenue_dashboard&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">type&lt;/span>: &lt;span style="color:#ae81ff">dashboard&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">owner&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Finance Team&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">email&lt;/span>: &lt;span style="color:#ae81ff">finance@company.com&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">depends_on&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#ae81ff">ref(&amp;#39;fct_revenue&amp;#39;)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#ae81ff">ref(&amp;#39;dim_date&amp;#39;)&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>When a model changes, the PR shows which exposures are affected. Stakeholder impact becomes visible in code review. You can proactively notify the finance team that their revenue dashboard&amp;rsquo;s upstream model is changing before they discover it themselves.&lt;/p>
&lt;p>Frame your communication in business terms. Not &amp;ldquo;model precision is up 12%&amp;rdquo; but &amp;ldquo;the sales team can now identify high-value leads 40% faster.&amp;rdquo; Nobody outside your team cares about your DAG. They care about whether they can trust the numbers. If you&amp;rsquo;re not sure how to bridge that gap, I wrote a guide on &lt;a href="https://ghostinthedata.info/posts/2025/2025-02-08-breaking-down-business-context/" target="_blank" rel="noopener">breaking down business context&lt;/a> — because the hardest part of talking to stakeholders isn&amp;rsquo;t the talking, it&amp;rsquo;s knowing what they actually need to hear.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="scoring-your-team">Scoring your team&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s the scoring framework, and I want you to be ruthless with yourself:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Score&lt;/th>
 &lt;th>What it means&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>&lt;strong>10-12&lt;/strong>&lt;/td>
 &lt;td>You&amp;rsquo;re in elite territory. Your data platform is a competitive advantage.&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>7-9&lt;/strong>&lt;/td>
 &lt;td>You&amp;rsquo;ve got strong foundations with clear areas to improve.&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>4-6&lt;/strong>&lt;/td>
 &lt;td>You&amp;rsquo;re functional but fragile. One bad incident away from a crisis of trust.&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>1-3&lt;/strong>&lt;/td>
 &lt;td>You&amp;rsquo;re firefighting, not engineering. The business tolerates your team — it doesn&amp;rsquo;t trust it.&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>Most teams land at 3-4. That&amp;rsquo;s not a criticism — it&amp;rsquo;s where the industry is. The gap between knowing these practices exist and actually implementing them is where the real work lives.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="where-to-start">Where to start&lt;/h3>
&lt;br>
&lt;p>If you&amp;rsquo;re staring at a low score and feeling overwhelmed, here&amp;rsquo;s what I&amp;rsquo;d suggest: &lt;strong>start with practices 4 and 10.&lt;/strong> CI testing and code review.&lt;/p>
&lt;p>Here&amp;rsquo;s why. Once dbt changes go through pull requests with automated checks, you have the infrastructure to enforce everything else. Contracts? They&amp;rsquo;re a CI check. Quality gates? CI check. Documentation requirements? CI check. SLA validation? CI check. You&amp;rsquo;re not adopting twelve practices — you&amp;rsquo;re building one pipeline that gates on twelve things.&lt;/p>
&lt;p>The canonical architecture appears across virtually every mature team I&amp;rsquo;ve studied: GitHub Actions triggering Slim CI with &lt;code>dbt build -s state:modified+&lt;/code> against per-PR Snowflake schemas, with production manifests stored in S3 for state comparison. Start there. Layer on the remaining practices as your team&amp;rsquo;s maturity grows.&lt;/p>
&lt;p>Three patterns emerged from the research that I think are worth calling out explicitly:&lt;/p>
&lt;p>&lt;strong>&amp;ldquo;As code&amp;rdquo; wins everywhere.&lt;/strong> Documentation-as-code, style-guides-as-code, permissions-as-code, quality-checks-as-code, contracts-as-code. Every manual process that gets codified becomes version-controlled, reviewable, and enforceable. Every one that stays manual eventually drifts.&lt;/p>
&lt;p>&lt;strong>Snowflake&amp;rsquo;s zero-copy cloning is the force multiplier.&lt;/strong> Isolated PR environments, sandbox onboarding, rebuild testing — all of these become cheap and instant with cloning. If you&amp;rsquo;re on Snowflake and not using this feature aggressively, you&amp;rsquo;re leaving the most powerful tool in the shed.&lt;/p>
&lt;p>&lt;strong>Culture matters more than tooling.&lt;/strong> Catalog adoption fails without meeting users in their existing workflows. Data quality requires executive sponsorship and error budgets, not just more tests. Producer-consumer contracts are first and foremost a cultural change. You can buy every tool on this list and still score a 3 if the organisation doesn&amp;rsquo;t value the practices behind them.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="the-diagram-and-the-warehouse">The diagram and the warehouse&lt;/h3>
&lt;br>
&lt;p>Remember that team — the one with the beautiful diagram and the revenue table nobody could rebuild?&lt;/p>
&lt;p>I caught up with one of the engineers about four months after I&amp;rsquo;d moved on. They&amp;rsquo;d started with exactly what I&amp;rsquo;d pushed for: CI testing and code review. Then they added contracts on their gold-layer models. Then SLAs on their Tier 1 tables. Then a first-week onboarding project for new hires.&lt;/p>
&lt;p>Their score when I first asked these twelve questions? Three. Four months later? Eight. Not perfect. But the difference wasn&amp;rsquo;t really the number. The difference was that when they pulled up that architecture diagram now, it matched what was actually running in Snowflake. The gap between the wall and the warehouse had closed.&lt;/p>
&lt;p>That&amp;rsquo;s what this test is really measuring. Not whether you have the right tools — you probably do. Not whether you know the right practices — you&amp;rsquo;re reading this, so clearly you care. It&amp;rsquo;s measuring whether there&amp;rsquo;s a gap between what you think your data platform looks like and what it actually looks like.&lt;/p>
&lt;p>These twelve questions just make you say it out loud.&lt;/p>
&lt;p>So print it. Score yourself honestly. Share it with your team. And the next time someone asks you how your data platform is doing, give them a number between 1 and 12.&lt;/p>
&lt;p>That number is worth more than any architecture diagram.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Engineering</category><category>dbt</category><category>Snowflake</category><category>GitHub Actions</category><category>AWS</category><category>Data Quality</category><category>CI/CD</category><category>Data Contracts</category></item><item><title>The CSV Test Suite Nobody Writes</title><link>https://ghostinthedata.info/posts/2026/2026-03-04-csv-test-suite/</link><pubDate>Wed, 04 Mar 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-03-04-csv-test-suite/</guid><author>Chris Hillman</author><description>How to turn RFC 4180 standards and real-world edge cases into a concrete test suite that catches CSV problems before your pipeline does. A practical guide for producers and consumers.</description><content:encoded>&lt;p>In October 2020, roughly 16,000 positive COVID-19 test results vanished from the UK&amp;rsquo;s public health reporting for nearly a week. Not because the tests weren&amp;rsquo;t run. Not because the labs didn&amp;rsquo;t report them. The results were collected, transmitted, and received — inside CSV files.&lt;/p>
&lt;p>The problem? Public Health England was importing those CSV files into Microsoft Excel&amp;rsquo;s legacy &lt;code>.xls&lt;/code> format. The format has a hard row limit of 65,536. When the files grew past that limit, Excel didn&amp;rsquo;t throw an error. It didn&amp;rsquo;t warn anyone. It just silently dropped the extra rows. Sixteen thousand people who tested positive for a deadly virus during a second wave went untraced. An estimated 50,000 of their contacts were never notified. And the system this happened in? Part of a £12 billion Test and Trace programme.&lt;/p>
&lt;p>I remember reading that story and feeling something beyond frustration. I felt recognition. Because I&amp;rsquo;d seen this pattern before — not at that scale, not with those stakes — but the same fundamental failure. Someone produced a CSV. Someone else consumed it. And nobody, at any point in the chain, had written a test that asked: &lt;em>&amp;ldquo;What happens when this file gets bigger than we expect?&amp;rdquo;&lt;/em>&lt;/p>
&lt;p>That&amp;rsquo;s the moment I became a bit obsessive about CSV testing. Not because CSV is complicated — it&amp;rsquo;s deceptively simple. But because that simplicity breeds a dangerous confidence. Everyone thinks they know how to handle a CSV. Everyone thinks the format is too basic to need a test suite. And that assumption has cost organisations billions of dollars, corrupted scientific research, and — in the case of COVID — put actual lives at risk.&lt;/p>
&lt;/br>
&lt;p>I&amp;rsquo;m telling you this because CSV testing is one of those things that sounds unnecessary until you&amp;rsquo;ve spent a weekend rebuilding a pipeline because a vendor changed their quoting convention and nobody noticed for three weeks. The engineers I&amp;rsquo;ve worked with who do this well — who write proper CSV test suites — don&amp;rsquo;t do it because they&amp;rsquo;re cautious by nature. They do it because they&amp;rsquo;ve been burned. And they decided it wouldn&amp;rsquo;t happen again.&lt;/p>
&lt;p>This article is the test suite I wish someone had handed me years ago. We&amp;rsquo;re going to take RFC 4180, the closest thing CSV has to a standard, and turn it into concrete test cases. We&amp;rsquo;ll build manifest files that make file delivery verifiable. And we&amp;rsquo;ll walk through how to systematically break down everything you know about CSV edge cases — embedded commas, encoding traps, null representation chaos, the lot — into tests you can run during development, not discoveries you make in production at 2am.&lt;/p>
&lt;p>Whether you&amp;rsquo;re producing CSVs for a downstream consumer or receiving them from a vendor who swears the file is &amp;ldquo;fine,&amp;rdquo; this is your playbook.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="why-i-know-how-to-test-isnt-enough">Why &amp;ldquo;I Know How to Test&amp;rdquo; Isn&amp;rsquo;t Enough&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s what I&amp;rsquo;ve noticed when I ask data engineers how they test their CSV files: most of them describe &lt;em>validation&lt;/em>, not &lt;em>testing&lt;/em>. They&amp;rsquo;ll say things like &amp;ldquo;we check the row count&amp;rdquo; or &amp;ldquo;we verify the columns match the schema.&amp;rdquo; And those are good things to do. But they&amp;rsquo;re runtime checks — they catch problems after they&amp;rsquo;ve already arrived at your door.&lt;/p>
&lt;p>Testing is different. Testing happens during development. Testing means deliberately creating malformed files and confirming your system handles them correctly. Testing means asking &amp;ldquo;what&amp;rsquo;s the worst thing a vendor could send me?&amp;rdquo; and then actually building that worst-case file and feeding it through your pipeline.&lt;/p>
&lt;p>The gap between validation and testing is where the expensive surprises live. Validation tells you &lt;em>this specific file&lt;/em> has a problem. Testing tells you &lt;em>your system&lt;/em> has a problem — before any real data gets near it.&lt;/p>
&lt;p>So how do you build a proper CSV test suite? You need a specification to test against. And despite its limitations, that specification is RFC 4180.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="rfc-4180-your-test-specification-and-its-gaps">RFC 4180: Your Test Specification (and Its Gaps)&lt;/h3>
&lt;br>
&lt;p>RFC 4180 was published in October 2005 by Yakov Shafranovich. It&amp;rsquo;s universally treated as &amp;ldquo;the CSV standard,&amp;rdquo; but its opening line tells a different story: it&amp;rsquo;s an &amp;ldquo;Informational&amp;rdquo; memo that &amp;ldquo;does not specify an Internet standard of any kind.&amp;rdquo; It was written to document common practice and register the &lt;code>text/csv&lt;/code> MIME type. That&amp;rsquo;s it.&lt;/p>
&lt;p>But here&amp;rsquo;s the thing — even a flawed specification is better than no specification. RFC 4180 gives us seven concrete rules, and each one translates directly into test cases. Let&amp;rsquo;s walk through them.&lt;/p>
&lt;/br>
&lt;h4 id="rule-1-records-are-separated-by-crlf">Rule 1: Records Are Separated by CRLF&lt;/h4>
&lt;p>The spec mandates &lt;code>\r\n&lt;/code> (carriage return + line feed) as the line ending. Not just &lt;code>\n&lt;/code> (Unix/macOS default). Not just &lt;code>\r&lt;/code> (old Mac).&lt;/p>
&lt;p>&lt;strong>Test cases for producers:&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_line_endings_are_crlf&lt;/span>(csv_content: bytes):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Every record must end with CRLF, not LF-only or CR-only.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> lines &lt;span style="color:#f92672">=&lt;/span> csv_content&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check no bare LF exists outside of quoted fields&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, line &lt;span style="color:#f92672">in&lt;/span> enumerate(lines):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#f92672">+&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: Found bare LF (Unix line ending). &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;RFC 4180 requires CRLF.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check no bare CR exists outside of quoted fields&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, line &lt;span style="color:#f92672">in&lt;/span> enumerate(lines):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#f92672">+&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: Found bare CR (old Mac line ending). &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;RFC 4180 requires CRLF.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Test cases for consumers:&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_consumer_handles_mixed_line_endings&lt;/span>(parser):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Consumer should handle LF, CR, and CRLF gracefully.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> test_cases &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;unix_lf&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">Alice,30&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">Bob,25&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;windows_crlf&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Alice,30&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Bob,25&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;old_mac_cr&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\r&lt;/span>&lt;span style="color:#e6db74">Alice,30&lt;/span>&lt;span style="color:#ae81ff">\r&lt;/span>&lt;span style="color:#e6db74">Bob,25&lt;/span>&lt;span style="color:#ae81ff">\r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;mixed&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Alice,30&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">Bob,25&lt;/span>&lt;span style="color:#ae81ff">\r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> label, content &lt;span style="color:#f92672">in&lt;/span> test_cases&lt;span style="color:#f92672">.&lt;/span>items():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result &lt;span style="color:#f92672">=&lt;/span> parser(content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> len(result) &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Line ending style &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>label&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;: Expected 2 data rows, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;got &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(result)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This distinction matters. If you&amp;rsquo;re a producer, your job is to emit compliant files. If you&amp;rsquo;re a consumer, your job is to survive whatever arrives. The test mindset is different for each role — producers test for correctness, consumers test for resilience.&lt;/p>
&lt;/br>
&lt;h4 id="rule-2-the-last-record-may-or-may-not-have-a-trailing-crlf">Rule 2: The Last Record May or May Not Have a Trailing CRLF&lt;/h4>
&lt;p>This sounds trivial, but it catches people. Some systems emit a final newline, others don&amp;rsquo;t. Your parser needs to handle both without creating a phantom empty row at the end.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_trailing_newline_does_not_create_empty_row&lt;/span>(parser):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;A trailing CRLF should not produce an extra empty record.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> with_trailing &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Alice,30&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> without_trailing &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#34;name,age&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Alice,30&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> len(parser(with_trailing)) &lt;span style="color:#f92672">==&lt;/span> len(parser(without_trailing))
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;h4 id="rule-3-there-should-be-a-header-row">Rule 3: There Should Be a Header Row&lt;/h4>
&lt;p>The RFC says &amp;ldquo;should&amp;rdquo; — not &amp;ldquo;must.&amp;rdquo; This ambiguity is a source of real problems. Some vendors send headers, some don&amp;rsquo;t, and some send headers that change between deliveries.&lt;/p>
&lt;p>Not having headers isn&amp;rsquo;t always a mistake. From a &lt;strong>security perspective, omitting headers is actually defensible&lt;/strong>. Think about what a header row really is: it&amp;rsquo;s a plaintext schema sitting right there in the first line of your file. If someone intercepts a CSV with columns named &lt;code>ssn&lt;/code>, &lt;code>credit_score&lt;/code>, &lt;code>annual_income&lt;/code>, and &lt;code>account_balance&lt;/code>, they don&amp;rsquo;t need to decode anything. The header tells them exactly what they&amp;rsquo;ve stolen and which column to exploit first.&lt;/p>
&lt;p>Stripping headers from files in transit — especially files containing PII, financial data, or health records — reduces the value of an intercepted file to an attacker. A file full of numbers and strings without column names is significantly harder to weaponise than one with a helpful schema attached.&lt;/p>
&lt;p>But here&amp;rsquo;s the catch: &lt;strong>headerless files only work if you have a reliable mechanism to reconstruct the schema at the consumer end&lt;/strong>. The column definitions need to live somewhere — and that somewhere should be your manifest file or your data contract. The manifest we build later in this article includes a &lt;code>columns&lt;/code> array with names, types, and ordering for exactly this reason. The producer strips the headers, the manifest carries the schema, and the consumer rebuilds the structure on ingestion.&lt;/p>
&lt;p>This means your tests need to handle both scenarios:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_header_row_present_and_matches_schema&lt;/span>(csv_file, expected_columns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;When headers are expected, verify they match the schema exactly.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> csv_file&lt;span style="color:#f92672">.&lt;/span>readline()&lt;span style="color:#f92672">.&lt;/span>strip()&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> header &lt;span style="color:#f92672">==&lt;/span> expected_columns, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Header mismatch.&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Expected: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected_columns&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Got: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>header&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_no_duplicate_column_names&lt;/span>(csv_file):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Duplicate column names cause silent column overwrites in pandas.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> csv_file&lt;span style="color:#f92672">.&lt;/span>readline()&lt;span style="color:#f92672">.&lt;/span>strip()&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> duplicates &lt;span style="color:#f92672">=&lt;/span> [h &lt;span style="color:#66d9ef">for&lt;/span> h &lt;span style="color:#f92672">in&lt;/span> header &lt;span style="color:#66d9ef">if&lt;/span> header&lt;span style="color:#f92672">.&lt;/span>count(h) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> duplicates, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Duplicate column names found: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>set(duplicates)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Pandas will silently rename these to name.1, name.2, etc.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_headerless_file_matches_manifest_schema&lt;/span>(csv_file, manifest):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;When headers are intentionally omitted for security,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> the first data row must match the manifest&amp;#39;s column count
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> and the manifest must define column names and types.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> first_row &lt;span style="color:#f92672">=&lt;/span> csv_file&lt;span style="color:#f92672">.&lt;/span>readline()&lt;span style="color:#f92672">.&lt;/span>strip()&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected_cols &lt;span style="color:#f92672">=&lt;/span> manifest[&lt;span style="color:#e6db74">&amp;#39;files&amp;#39;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;columns&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> len(first_row) &lt;span style="color:#f92672">==&lt;/span> len(expected_cols), (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Headerless file has &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(first_row)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> fields but manifest &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;defines &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(expected_cols)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> columns. Without headers, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;column alignment is entirely dependent on the manifest.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Verify manifest actually defines names (not just count)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_def &lt;span style="color:#f92672">in&lt;/span> expected_cols:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#e6db74">&amp;#39;name&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col_def, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Manifest column definitions must include &amp;#39;name&amp;#39; — &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;headerless files are only safe if the schema is &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;fully specified in the manifest.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Whether you include headers or not, &lt;strong>the decision must be explicit and documented in your data contract&lt;/strong>. The worst scenario isn&amp;rsquo;t a headerless file or a headered file — it&amp;rsquo;s a file where nobody agreed which one it should be, and the answer changes depending on who ran the export.&lt;/p>
&lt;/br>
&lt;h4 id="rule-4-each-row-should-contain-the-same-number-of-fields">Rule 4: Each Row Should Contain the Same Number of Fields&lt;/h4>
&lt;p>This is where &amp;ldquo;ragged rows&amp;rdquo; come from — and they&amp;rsquo;re surprisingly common. A free-text field containing an unescaped comma silently creates an extra column in that row, shifting everything to the right.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_consistent_column_count&lt;/span>(csv_reader):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Every row must have the same number of fields as the header.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> next(csv_reader)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected &lt;span style="color:#f92672">=&lt;/span> len(header)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ragged_rows &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, row &lt;span style="color:#f92672">in&lt;/span> enumerate(csv_reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(row) &lt;span style="color:#f92672">!=&lt;/span> expected:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ragged_rows&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;row&amp;#39;&lt;/span>: i,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;expected&amp;#39;&lt;/span>: expected,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;got&amp;#39;&lt;/span>: len(row),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;content&amp;#39;&lt;/span>: row[:&lt;span style="color:#ae81ff">3&lt;/span>] &lt;span style="color:#75715e"># First 3 fields for debugging&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> ragged_rows, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(ragged_rows)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> ragged rows. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;First: row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ragged_rows[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;row&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> has &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ragged_rows[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;got&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> fields (expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">). &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;This usually means an unescaped comma or newline in the data.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This test is the single most valuable check you can run against a CSV file. If it passes, most of the structural problems aren&amp;rsquo;t present. If it fails, you know exactly where to look.&lt;/p>
&lt;/br>
&lt;h4 id="rules-5-7-quoting-and-escaping">Rules 5-7: Quoting and Escaping&lt;/h4>
&lt;p>These three rules govern how fields handle special characters:&lt;/p>
&lt;ul>
&lt;li>Fields containing commas, double-quotes, or line breaks &lt;strong>must&lt;/strong> be enclosed in double-quotes&lt;/li>
&lt;li>Double-quotes inside a quoted field are escaped by doubling them: &lt;code>&amp;quot;&amp;quot;&lt;/code>&lt;/li>
&lt;li>Spaces are part of field data and must not be trimmed&lt;/li>
&lt;/ul>
&lt;p>These rules are where the most insidious bugs hide:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># -- Producer tests: Do we generate valid quoted fields? --&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_fields_with_commas_are_quoted&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;A field containing a comma must be wrapped in double-quotes.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> record &lt;span style="color:#f92672">=&lt;/span> generate_csv_row(address&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;456 Oak Ave, Apt 2B&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> record &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;id,&amp;#34;456 Oak Ave, Apt 2B&amp;#34;,other_field&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_fields_with_quotes_are_escaped&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Double-quotes in data must be doubled, and field must be quoted.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> record &lt;span style="color:#f92672">=&lt;/span> generate_csv_row(company&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;Acme &amp;#34;Holdings&amp;#34; Ltd&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#e6db74">&amp;#39;&amp;#34;Acme &amp;#34;&amp;#34;Holdings&amp;#34;&amp;#34; Ltd&amp;#34;&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> record
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_fields_with_newlines_are_quoted&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;A field containing a line break must be wrapped in double-quotes.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> record &lt;span style="color:#f92672">=&lt;/span> generate_csv_row(notes&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Line one&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Line two&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># The entire field, including the newline, should be inside quotes&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#e6db74">&amp;#39;&amp;#34;Line one&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Line two&amp;#34;&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> record
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># -- Consumer tests: Can we parse valid-but-tricky fields? --&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_parse_embedded_comma&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;name,address&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Alice&amp;#34;,&amp;#34;456 Oak Ave, Apt 2B&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result &lt;span style="color:#f92672">=&lt;/span> parse(content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> result[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;address&amp;#39;&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;456 Oak Ave, Apt 2B&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_parse_embedded_newline&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;name,notes&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Alice&amp;#34;,&amp;#34;Line one&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Line two&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result &lt;span style="color:#f92672">=&lt;/span> parse(content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> result[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;notes&amp;#39;&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Line one&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">Line two&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_parse_escaped_quotes&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;name,company&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Alice&amp;#34;,&amp;#34;Acme &amp;#34;&amp;#34;Holdings&amp;#34;&amp;#34; Ltd&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\r\n&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result &lt;span style="color:#f92672">=&lt;/span> parse(content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> result[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;company&amp;#39;&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Acme &amp;#34;Holdings&amp;#34; Ltd&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That covers the seven RFC rules. But here&amp;rsquo;s the catch — the RFC is only half the story. The really nasty problems come from everything the RFC &lt;em>doesn&amp;rsquo;t&lt;/em> cover.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="testing-what-the-rfc-forgot">Testing What the RFC Forgot&lt;/h3>
&lt;br>
&lt;p>RFC 4180 says nothing about character encoding, data types, null values, or how Excel will mangle your data. These are the gaps where production failures breed. Each one becomes a test category.&lt;/p>
&lt;h4 id="encoding-tests">Encoding Tests&lt;/h4>
&lt;p>The most common encoding problem is invisible: the &lt;strong>UTF-8 BOM&lt;/strong> (Byte Order Mark). Excel on Windows prepends three bytes (&lt;code>EF BB BF&lt;/code>) to CSV files saved as UTF-8. Those bytes become an invisible character attached to your first column name. You try &lt;code>df['Name']&lt;/code> and get a &lt;code>KeyError&lt;/code> because the actual column is &lt;code>'\ufeffName'&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_no_bom_prefix&lt;/span>(csv_path):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;UTF-8 BOM causes invisible characters in the first column name.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> first_bytes &lt;span style="color:#f92672">=&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>read(&lt;span style="color:#ae81ff">3&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> first_bytes &lt;span style="color:#f92672">!=&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\xef\xbb\xbf&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;File contains UTF-8 BOM. This will prepend an invisible &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;character to the first column name and break column lookups.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_file_is_valid_utf8&lt;/span>(csv_path):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Catch encoding mismatches before they become mojibake.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>read()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">UnicodeDecodeError&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Try to identify the actual encoding&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> chardet
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> detected &lt;span style="color:#f92672">=&lt;/span> chardet&lt;span style="color:#f92672">.&lt;/span>detect(content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">AssertionError&lt;/span>(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;File is not valid UTF-8. Decode error at byte &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>e&lt;span style="color:#f92672">.&lt;/span>start&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Detected encoding: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>detected[&lt;span style="color:#e6db74">&amp;#39;encoding&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;(confidence: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>detected[&lt;span style="color:#e6db74">&amp;#39;confidence&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">.0%&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">). &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;This is likely a Latin-1 or Windows-1252 file.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_no_null_bytes&lt;/span>(csv_path):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Null bytes (0x00) indicate binary data or a UTF-16 file
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> being read as UTF-8.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>read()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> null_positions &lt;span style="color:#f92672">=&lt;/span> [i &lt;span style="color:#66d9ef">for&lt;/span> i, b &lt;span style="color:#f92672">in&lt;/span> enumerate(content) &lt;span style="color:#66d9ef">if&lt;/span> b &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> null_positions, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(null_positions)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> null bytes. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;First at position &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>null_positions[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;This might be a UTF-16 encoded file.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If you&amp;rsquo;re a &lt;strong>consumer&lt;/strong>, you should also test that your parser survives non-UTF-8 input gracefully — does it raise a clear error, or does it silently produce mojibake? Build test fixtures with Latin-1 encoded files containing accented characters (&lt;code>café&lt;/code>, &lt;code>naïve&lt;/code>, &lt;code>München&lt;/code>) and verify your system either decodes correctly or fails explicitly.&lt;/p>
&lt;h4 id="null-value-tests">Null Value Tests&lt;/h4>
&lt;p>RFC 4180 has no concept of null. An empty field (&lt;code>,,&lt;/code>) is just an empty string. But every system represents null differently, and when you&amp;rsquo;re receiving files from multiple vendors, the chaos compounds.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># These are all real null representations I&amp;#39;ve seen in production&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>NULL_VARIANTS &lt;span style="color:#f92672">=&lt;/span> [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Empty string (most common)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;NULL&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># MySQL Workbench default&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;null&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># JavaScript/JSON convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;None&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Python convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;\N&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># PostgreSQL COPY convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;N/A&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Business user convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;n/a&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Lowercase variant&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;NA&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># R convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;#N/A&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Excel formula error&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;-&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Dashboard export convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;NaN&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Pandas/NumPy convention&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;(null)&amp;#39;&lt;/span>, &lt;span style="color:#75715e"># Some BI tools&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_null_representation_is_consistent&lt;/span>(csv_path, null_convention&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;All null values should use the agreed convention.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Mixed conventions cause type inference failures.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> csv
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, newline&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>reader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> next(reader)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row_num, row &lt;span style="color:#f92672">in&lt;/span> enumerate(reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_idx, value &lt;span style="color:#f92672">in&lt;/span> enumerate(row):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> value &lt;span style="color:#f92672">in&lt;/span> NULL_VARIANTS &lt;span style="color:#f92672">and&lt;/span> value &lt;span style="color:#f92672">!=&lt;/span> null_convention:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;row&amp;#39;&lt;/span>: row_num,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;column&amp;#39;&lt;/span>: header[col_idx],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;value&amp;#39;&lt;/span>: repr(value),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;expected&amp;#39;&lt;/span>: repr(null_convention)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> violations, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(violations)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> non-standard null representations. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;First: row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;row&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;column &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;column&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; contains &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;value&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> (expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#e6db74">&amp;#39;expected&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This is a test you should agree on with your vendor as part of a &lt;strong>data contract&lt;/strong> — which we&amp;rsquo;ll get to shortly. Without explicit agreement, you&amp;rsquo;ll spend more time debugging null handling than you will on actual analysis.&lt;/p>
&lt;h4 id="the-leading-zeros-problem">The Leading Zeros Problem&lt;/h4>
&lt;p>Excel silently strips leading zeros from any field it interprets as numeric. ZIP code &lt;code>07102&lt;/code> becomes &lt;code>7102&lt;/code>. Product code &lt;code>00456&lt;/code> becomes &lt;code>456&lt;/code>. Numbers over 15 digits lose precision permanently. This is not a CSV problem — it&amp;rsquo;s an Excel problem — but it affects every CSV that passes through Excel, which is most of them.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_leading_zeros_preserved&lt;/span>(csv_path, zero_padded_columns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Fields that should retain leading zeros still have them.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Excel strips these silently on open-and-save.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> csv
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, newline&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>DictReader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row_num, row &lt;span style="color:#f92672">in&lt;/span> enumerate(reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> zero_padded_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> value &lt;span style="color:#f92672">=&lt;/span> row[col]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> value &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> value[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>isdigit():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">continue&lt;/span> &lt;span style="color:#75715e"># Skip non-numeric values&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> value &lt;span style="color:#f92672">and&lt;/span> value[&lt;span style="color:#ae81ff">0&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#39;0&amp;#39;&lt;/span> &lt;span style="color:#f92672">and&lt;/span> len(value) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">pass&lt;/span> &lt;span style="color:#75715e"># Good — leading zero preserved&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> value &lt;span style="color:#f92672">and&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> zero_padded_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check against expected length&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># e.g., ZIP should be 5 digits&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">pass&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_no_scientific_notation&lt;/span>(csv_path, numeric_columns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Numbers should not be in scientific notation (1.23E+11).
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> This happens when Excel converts long numbers.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> csv&lt;span style="color:#f92672">,&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sci_notation &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>compile(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;^-?\d+\.?\d*[eE][+-]?\d+$&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, newline&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>DictReader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row_num, row &lt;span style="color:#f92672">in&lt;/span> enumerate(reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> numeric_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sci_notation&lt;span style="color:#f92672">.&lt;/span>&lt;span style="color:#66d9ef">match&lt;/span>(row[col] &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations&lt;span style="color:#f92672">.&lt;/span>append((row_num, col, row[col]))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> violations, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Scientific notation found — data may have been &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;corrupted by Excel. First: row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;column &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;, value &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">2&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="date-format-tests">Date Format Tests&lt;/h4>
&lt;p>Is &lt;code>03/04/2024&lt;/code> March 4th or April 3rd? Unless you know the source system&amp;rsquo;s locale, you literally cannot tell for dates where both day and month values are 12 or below. The only safe format is ISO 8601: &lt;code>YYYY-MM-DD&lt;/code>.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_dates_are_iso8601&lt;/span>(csv_path, date_columns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Date fields should use ISO 8601 (YYYY-MM-DD) format.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Ambiguous formats like MM/DD/YYYY vs DD/MM/YYYY
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> are programmatically unresolvable.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> csv&lt;span style="color:#f92672">,&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> iso_pattern &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>compile(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;^\d&lt;/span>&lt;span style="color:#e6db74">{4}&lt;/span>&lt;span style="color:#e6db74">-\d&lt;/span>&lt;span style="color:#e6db74">{2}&lt;/span>&lt;span style="color:#e6db74">-\d&lt;/span>&lt;span style="color:#e6db74">{2}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, newline&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>DictReader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row_num, row &lt;span style="color:#f92672">in&lt;/span> enumerate(reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> date_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> val &lt;span style="color:#f92672">=&lt;/span> row&lt;span style="color:#f92672">.&lt;/span>get(col, &lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> val &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> iso_pattern&lt;span style="color:#f92672">.&lt;/span>&lt;span style="color:#66d9ef">match&lt;/span>(val):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> violations&lt;span style="color:#f92672">.&lt;/span>append((row_num, col, val))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> violations, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Non-ISO date format found. First: row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;column &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;, value &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>violations[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">2&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Ambiguous date formats cannot be safely parsed.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="invisible-character-tests">Invisible Character Tests&lt;/h4>
&lt;p>These are the ones that make you question your sanity. Non-breaking spaces (&lt;code>U+00A0&lt;/code>), zero-width spaces (&lt;code>U+200B&lt;/code>), zero-width joiners — they look identical to normal characters (or invisible entirely) but break numeric conversion, string matching, and joins.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>INVISIBLE_CHARS &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u00a0&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Non-breaking space&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u200b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Zero-width space&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u200c&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Zero-width non-joiner&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u200d&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Zero-width joiner&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\ufeff&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;BOM / Zero-width no-break space&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u2028&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Line separator&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u2029&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Paragraph separator&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u00ad&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Soft hyphen&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_no_invisible_characters&lt;/span>(csv_path):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Invisible Unicode characters break joins, type casting,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> and string comparisons while being impossible to spot visually.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(csv_path, encoding&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> content &lt;span style="color:#f92672">=&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>read()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> found &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> char, name &lt;span style="color:#f92672">in&lt;/span> INVISIBLE_CHARS&lt;span style="color:#f92672">.&lt;/span>items():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> positions &lt;span style="color:#f92672">=&lt;/span> [i &lt;span style="color:#66d9ef">for&lt;/span> i, c &lt;span style="color:#f92672">in&lt;/span> enumerate(content) &lt;span style="color:#66d9ef">if&lt;/span> c &lt;span style="color:#f92672">==&lt;/span> char]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> positions:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> found&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> (U+&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ord(char)&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">04X&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">): &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(positions)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> occurrences&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">assert&lt;/span> &lt;span style="color:#f92672">not&lt;/span> found, (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Invisible characters detected:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#f92672">+&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#f92672">.&lt;/span>join(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; - &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>f&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span> &lt;span style="color:#66d9ef">for&lt;/span> f &lt;span style="color:#f92672">in&lt;/span> found)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="manifest-files-testing-secure-delivery">Manifest Files: Testing Secure Delivery&lt;/h3>
&lt;br>
&lt;p>All of those tests assume you&amp;rsquo;ve received the file intact. But how do you know? A file could be truncated during transfer, silently corrupted on disk, or swapped out entirely. This is where manifest files come in — they&amp;rsquo;re the contract between producer and consumer.&lt;/p>
&lt;p>A manifest is metadata &lt;em>about&lt;/em> the file that travels alongside it. At minimum, it should contain the filename, a cryptographic checksum, the row count, and the file size. NIST has formally deprecated MD5 and SHA-1, so use SHA-256.&lt;/p>
&lt;p>Here&amp;rsquo;s what a solid manifest looks like:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;manifest_version&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;1.0&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;generated_at&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;2026-03-04T08:00:00Z&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;source_system&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;vendor-crm&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;batch_id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;batch-20260304-001&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;files&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;filename&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;customers_20260304.csv&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;sha256&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;a3f2b8c9d1e4f5a6b7c8d9e0f1a2b3c4d5e6f7...&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;row_count&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">50432&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;row_count_includes_header&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;file_size_bytes&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">8234567&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;column_count&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">12&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;encoding&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;UTF-8&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;delimiter&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;,&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;quote_char&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;\&amp;#34;&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;has_header&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;columns&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> { &lt;span style="color:#f92672">&amp;#34;name&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>, &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;integer&amp;#34;&lt;/span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> { &lt;span style="color:#f92672">&amp;#34;name&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;name&amp;#34;&lt;/span>, &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;string&amp;#34;&lt;/span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> { &lt;span style="color:#f92672">&amp;#34;name&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;email&amp;#34;&lt;/span>, &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;string&amp;#34;&lt;/span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> { &lt;span style="color:#f92672">&amp;#34;name&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;created_date&amp;#34;&lt;/span>, &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span>, &lt;span style="color:#f92672">&amp;#34;format&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;YYYY-MM-DD&amp;#34;&lt;/span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now here&amp;rsquo;s the test suite that validates a delivery against its manifest:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> hashlib
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">validate_delivery&lt;/span>(manifest_path, data_dir):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Validate a CSV delivery against its manifest.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Returns list of errors. Empty list = valid delivery.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 1. Parse manifest&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(manifest_path) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> manifest &lt;span style="color:#f92672">=&lt;/span> json&lt;span style="color:#f92672">.&lt;/span>load(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file_spec &lt;span style="color:#f92672">in&lt;/span> manifest[&lt;span style="color:#e6db74">&amp;#39;files&amp;#39;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> filepath &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>join(data_dir, file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 2. File existence&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>exists(filepath):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;MISSING: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> not found&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">continue&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 3. File size&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> actual_size &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>getsize(filepath)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> actual_size &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;EMPTY: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> is zero bytes&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">continue&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> actual_size &lt;span style="color:#f92672">!=&lt;/span> file_spec[&lt;span style="color:#e6db74">&amp;#39;file_size_bytes&amp;#39;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SIZE MISMATCH: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;file_size_bytes&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> bytes, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;got &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>actual_size&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 4. Checksum&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sha256 &lt;span style="color:#f92672">=&lt;/span> hashlib&lt;span style="color:#f92672">.&lt;/span>sha256()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(filepath, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> chunk &lt;span style="color:#f92672">in&lt;/span> iter(&lt;span style="color:#66d9ef">lambda&lt;/span>: f&lt;span style="color:#f92672">.&lt;/span>read(&lt;span style="color:#ae81ff">8192&lt;/span>), &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sha256&lt;span style="color:#f92672">.&lt;/span>update(chunk)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> actual_hash &lt;span style="color:#f92672">=&lt;/span> sha256&lt;span style="color:#f92672">.&lt;/span>hexdigest()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> actual_hash &lt;span style="color:#f92672">!=&lt;/span> file_spec[&lt;span style="color:#e6db74">&amp;#39;sha256&amp;#39;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;CHECKSUM MISMATCH: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;has been modified or corrupted in transit&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 5. Row count&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(filepath, encoding&lt;span style="color:#f92672">=&lt;/span>file_spec&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#39;encoding&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>)) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> actual_rows &lt;span style="color:#f92672">=&lt;/span> sum(&lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">for&lt;/span> _ &lt;span style="color:#f92672">in&lt;/span> f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected &lt;span style="color:#f92672">=&lt;/span> file_spec[&lt;span style="color:#e6db74">&amp;#39;row_count&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> actual_rows &lt;span style="color:#f92672">!=&lt;/span> expected:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;ROW COUNT MISMATCH: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows, got &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>actual_rows&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># 6. Column count and names&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(filepath, encoding&lt;span style="color:#f92672">=&lt;/span>file_spec&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#39;encoding&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>)) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> csv
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>reader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> next(reader)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(header) &lt;span style="color:#f92672">!=&lt;/span> file_spec[&lt;span style="color:#e6db74">&amp;#39;column_count&amp;#39;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;COLUMN COUNT MISMATCH: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;column_count&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> columns, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;got &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(header)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;columns&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_spec:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected_names &lt;span style="color:#f92672">=&lt;/span> [c[&lt;span style="color:#e6db74">&amp;#39;name&amp;#39;&lt;/span>] &lt;span style="color:#66d9ef">for&lt;/span> c &lt;span style="color:#f92672">in&lt;/span> file_spec[&lt;span style="color:#e6db74">&amp;#39;columns&amp;#39;&lt;/span>]]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> header &lt;span style="color:#f92672">!=&lt;/span> expected_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors&lt;span style="color:#f92672">.&lt;/span>append(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;COLUMN NAME MISMATCH: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_spec[&lt;span style="color:#e6db74">&amp;#39;filename&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; Expected: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected_names&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; Got: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>header&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> errors
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The key insight about manifest validation is that it should be a &lt;strong>gate&lt;/strong>, not a check. If the manifest fails, you don&amp;rsquo;t process the file. Period. No &amp;ldquo;well, the row count is close enough.&amp;rdquo; No &amp;ldquo;the checksum is probably fine.&amp;rdquo; Either the delivery matches the contract or it gets quarantined.&lt;/p>
&lt;p>For orchestration tools like Airflow, the pattern looks like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Airflow DAG pattern: manifest-gated processing&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>wait_for_manifest &lt;span style="color:#f92672">=&lt;/span> FileSensor(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;wait_for_manifest&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> filepath&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;/landing/{{ ds_nodash }}/manifest.json&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> poke_interval&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">120&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> timeout&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">7200&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>validate &lt;span style="color:#f92672">=&lt;/span> BranchPythonOperator(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> task_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;validate_delivery&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> python_callable&lt;span style="color:#f92672">=&lt;/span>validate_and_route,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Routes to either &amp;#39;load_data&amp;#39; or &amp;#39;quarantine_and_alert&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>wait_for_manifest &lt;span style="color:#f92672">&amp;gt;&amp;gt;&lt;/span> validate &lt;span style="color:#f92672">&amp;gt;&amp;gt;&lt;/span> [load_data, quarantine_and_alert]
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>As a producer&lt;/strong>, you should generate the manifest &lt;em>after&lt;/em> writing the CSV file — computing the checksum on the exact bytes that were written. Ship the manifest alongside the data file. If you&amp;rsquo;re using SFTP or S3, upload the data file first, then the manifest. The consumer watches for the manifest as a &amp;ldquo;ready&amp;rdquo; signal — its presence means the data file is complete.&lt;/p>
&lt;p>&lt;strong>As a consumer&lt;/strong>, you should test your manifest validation against deliberately corrupted scenarios:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_detects_truncated_file&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Simulate a file that was cut short during transfer.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Write a file, then truncate it&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">...&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_detects_missing_file&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Manifest references a file that doesn&amp;#39;t exist.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">...&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_detects_extra_rows&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;File has more rows than manifest claims —
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> could indicate duplicate data or appended garbage.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">...&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_detects_wrong_checksum&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;File was modified after manifest was generated.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">...&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_rejects_zero_byte_file&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Empty files should never pass validation.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">...&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="the-csv-test-matrix-thinking-like-a-tester">The CSV Test Matrix: Thinking Like a Tester&lt;/h3>
&lt;br>
&lt;p>Most people know about edge cases — embedded commas, encoding problems, null ambiguity — but they don&amp;rsquo;t know how to systematically translate that knowledge into a test suite. They end up with ad hoc checks scattered across their codebase, catching some things and missing others.&lt;/p>
&lt;p>The trick is to think in &lt;strong>categories&lt;/strong>, not individual cases. Every CSV problem falls into one of five categories, and each category has a predictable set of test cases.&lt;/p>
&lt;h4 id="category-1-structural-integrity">Category 1: Structural Integrity&lt;/h4>
&lt;p>&lt;em>Does the file conform to the expected shape?&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Test&lt;/th>
 &lt;th>What It Catches&lt;/th>
 &lt;th>Producer/Consumer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Consistent column count across all rows&lt;/td>
 &lt;td>Unescaped delimiters in data&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Header row present and matches schema (when expected)&lt;/td>
 &lt;td>Schema drift, renamed columns&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Headerless file matches manifest column count&lt;/td>
 &lt;td>Misaligned data when headers stripped for security&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No duplicate column names&lt;/td>
 &lt;td>Silent column overwrites&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No trailing delimiter creating phantom column&lt;/td>
 &lt;td>Off-by-one column counts&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>File is not empty / not just a header&lt;/td>
 &lt;td>Failed exports, empty result sets&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Row count within expected range&lt;/td>
 &lt;td>Truncation, duplication, data loss&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;h4 id="category-2-encoding-and-character-safety">Category 2: Encoding and Character Safety&lt;/h4>
&lt;p>&lt;em>Are the bytes what we think they are?&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Test&lt;/th>
 &lt;th>What It Catches&lt;/th>
 &lt;th>Producer/Consumer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>File is valid UTF-8&lt;/td>
 &lt;td>Latin-1/Windows-1252 masquerading as UTF-8&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No BOM prefix&lt;/td>
 &lt;td>Invisible first-column corruption&lt;/td>
 &lt;td>Producer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No null bytes&lt;/td>
 &lt;td>Binary data or wrong encoding family&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No invisible Unicode characters&lt;/td>
 &lt;td>Broken joins, failed type casts&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No smart quotes or em-dashes&lt;/td>
 &lt;td>Copy-paste from Word/email&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;h4 id="category-3-quoting-and-escaping">Category 3: Quoting and Escaping&lt;/h4>
&lt;p>&lt;em>Are special characters handled correctly?&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Test&lt;/th>
 &lt;th>What It Catches&lt;/th>
 &lt;th>Producer/Consumer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Fields with commas are quoted&lt;/td>
 &lt;td>Row splitting, column misalignment&lt;/td>
 &lt;td>Producer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Fields with double-quotes use &lt;code>&amp;quot;&amp;quot;&lt;/code> escaping&lt;/td>
 &lt;td>Unclosed quotes consuming multiple rows&lt;/td>
 &lt;td>Producer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Fields with newlines are quoted&lt;/td>
 &lt;td>Row count mismatches, multi-line chaos&lt;/td>
 &lt;td>Producer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Consumer survives unclosed quotes gracefully&lt;/td>
 &lt;td>Vendor bugs eating your pipeline&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No backslash escaping (non-standard)&lt;/td>
 &lt;td>Interop failures with strict parsers&lt;/td>
 &lt;td>Producer&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;h4 id="category-4-data-type-safety">Category 4: Data Type Safety&lt;/h4>
&lt;p>&lt;em>Will downstream systems interpret values correctly?&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Test&lt;/th>
 &lt;th>What It Catches&lt;/th>
 &lt;th>Producer/Consumer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Dates in ISO 8601 format&lt;/td>
 &lt;td>Ambiguous MM/DD vs DD/MM&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Leading zeros preserved&lt;/td>
 &lt;td>Excel corruption of ZIPs, product codes&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>No scientific notation in numeric fields&lt;/td>
 &lt;td>Excel corruption of long numbers&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Consistent null representation&lt;/td>
 &lt;td>Mixed NULL/None/N/A/empty conventions&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Boolean fields use consistent values&lt;/td>
 &lt;td>TRUE/True/true/1/yes/Y chaos&lt;/td>
 &lt;td>Both&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Numeric fields contain only valid numbers&lt;/td>
 &lt;td>Currency symbols, percentage signs, commas in numbers&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;h4 id="category-5-delivery-integrity">Category 5: Delivery Integrity&lt;/h4>
&lt;p>&lt;em>Did the file arrive intact and complete?&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Test&lt;/th>
 &lt;th>What It Catches&lt;/th>
 &lt;th>Producer/Consumer&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>SHA-256 checksum matches manifest&lt;/td>
 &lt;td>Corruption in transit&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>File size matches manifest&lt;/td>
 &lt;td>Truncation, wrong file&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Row count matches manifest&lt;/td>
 &lt;td>Silent data loss (the COVID problem)&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Column count matches manifest&lt;/td>
 &lt;td>Schema changes without notice&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>File is not a duplicate of previous delivery&lt;/td>
 &lt;td>Accidental re-sends&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Manifest is present and valid JSON&lt;/td>
 &lt;td>Incomplete delivery&lt;/td>
 &lt;td>Consumer&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>This matrix is your test plan. Print it out. Stick it on the wall next to your desk. When you&amp;rsquo;re building a new CSV integration, walk through each category and ask: &amp;ldquo;Do I have a test for this?&amp;rdquo;&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="building-test-fixtures-the-art-of-deliberate-breakage">Building Test Fixtures: The Art of Deliberate Breakage&lt;/h3>
&lt;br>
&lt;p>A test suite is only as good as its fixtures. And for CSV testing, your fixtures need to be deliberately, creatively broken. Here&amp;rsquo;s how I build mine.&lt;/p>
&lt;p>Start with a &lt;strong>golden file&lt;/strong> — a perfectly compliant CSV that passes every test. This is your baseline. Then create variations that each break exactly one thing:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 568 553"
 >
 &lt;g transform='translate(8,16)'>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='0' y='20' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='36' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='52' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='68' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='84' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='100' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='116' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='132' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='148' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='164' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='180' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='196' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='212' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='228' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='244' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='260' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='276' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='292' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='308' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='324' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='340' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='356' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='372' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='388' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='404' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='420' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='436' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='452' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='468' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='8' y='20' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='52' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='180' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='276' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='372' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='468' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='16' y='20' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='52' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='180' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='276' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='372' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='468' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='32' y='52' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='32' y='68' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='84' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='100' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='116' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='132' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='148' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='164' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='180' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='32' y='196' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='212' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='228' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='244' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='260' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='276' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='32' y='292' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='308' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='324' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='340' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='356' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='372' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='32' y='388' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='404' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='420' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='436' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='452' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='468' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='32' y='484' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='500' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='516' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='532' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='40' y='20' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='40' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='40' y='68' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='84' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='116' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='132' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='148' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='180' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='40' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='212' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='228' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='244' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='276' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='40' y='292' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='308' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='324' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='340' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='356' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='372' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='40' y='388' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='404' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='420' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='436' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='452' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='468' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='40' y='484' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='500' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='516' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='532' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='48' y='20' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='48' y='52' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='48' y='68' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='84' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='116' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='132' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='148' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='180' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='48' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='212' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='228' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='244' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='276' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='48' y='292' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='308' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='324' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='340' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='356' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='372' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='48' y='388' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='404' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='420' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='436' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='452' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='468' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='48' y='484' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='500' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='516' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='532' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='56' y='20' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='56' y='36' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='56' y='52' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='56' y='180' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='56' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='56' y='372' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='56' y='468' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='64' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='52' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='64' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='64' y='84' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='64' y='100' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='64' y='116' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='64' y='132' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='64' y='148' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='164' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='180' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='64' y='196' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='64' y='212' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='64' y='228' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='64' y='244' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='64' y='260' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='64' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='64' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='324' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='64' y='340' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='356' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='64' y='372' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='64' y='388' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='64' y='404' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='64' y='420' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='64' y='436' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='64' y='452' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='64' y='468' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='64' y='484' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='500' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='64' y='516' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='64' y='532' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='72' y='20' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='72' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='72' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='72' y='84' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='72' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='72' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='72' y='148' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='72' y='164' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='72' y='180' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='72' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='72' y='228' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='244' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='72' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='276' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='72' y='292' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='72' y='308' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='72' y='324' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='72' y='340' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='72' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='72' y='372' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='72' y='388' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='72' y='404' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='72' y='420' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='72' y='436' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='452' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='72' y='468' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='72' y='484' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='72' y='500' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='72' y='516' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='72' y='532' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='80' y='20' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='80' y='52' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='80' y='68' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='80' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='80' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='80' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='80' y='132' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='80' y='148' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='80' y='164' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='80' y='180' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='80' y='196' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='80' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='80' y='228' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='80' y='244' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='80' y='260' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='80' y='276' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='80' y='292' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='80' y='308' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='80' y='324' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='80' y='340' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='80' y='356' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='80' y='372' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='80' y='388' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='80' y='404' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='80' y='420' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='80' y='436' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='80' y='452' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='80' y='468' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='80' y='484' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='80' y='500' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='80' y='516' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='80' y='532' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='88' y='20' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='88' y='52' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='88' y='68' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='88' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='88' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='88' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='88' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='88' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='88' y='164' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='88' y='180' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='88' y='196' fill='currentColor' style='font-size:1em'>8&lt;/text>
&lt;text text-anchor='middle' x='88' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='88' y='228' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='88' y='244' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='88' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='276' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='88' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='324' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='88' y='340' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='88' y='356' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='88' y='372' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='88' y='388' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='88' y='404' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='88' y='420' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='436' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='88' y='452' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='88' y='468' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='88' y='484' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='88' y='500' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='88' y='516' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='88' y='532' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='96' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='96' y='52' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='96' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='84' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='96' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='132' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='96' y='148' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='96' y='164' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='96' y='180' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='96' y='196' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='96' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='96' y='228' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='96' y='244' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='96' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='96' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='96' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='96' y='324' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='96' y='340' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='96' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='372' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='388' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='96' y='404' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='96' y='420' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='96' y='436' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='96' y='452' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='468' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='96' y='484' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='96' y='500' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='96' y='516' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='96' y='532' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='104' y='20' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='104' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='52' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='104' y='68' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='104' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='100' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='104' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='104' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='104' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='104' y='164' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='104' y='196' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='104' y='212' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='104' y='228' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='104' y='244' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='104' y='260' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='104' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='104' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='104' y='324' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='104' y='340' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='104' y='356' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='104' y='372' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='104' y='388' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='104' y='404' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='104' y='420' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='104' y='436' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='104' y='452' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='104' y='484' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='104' y='500' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='104' y='516' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='104' y='532' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='112' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='112' y='52' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='112' y='68' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='112' y='84' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='112' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='112' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='112' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='148' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='112' y='164' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='112' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='112' y='212' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='112' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='112' y='244' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='112' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='340' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='112' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='372' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='112' y='388' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='112' y='404' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='420' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='112' y='436' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='112' y='452' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='112' y='484' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='112' y='500' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='112' y='516' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='112' y='532' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='120' y='36' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='120' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='120' y='84' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='120' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='120' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='120' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='120' y='148' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='120' y='164' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='120' y='196' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='120' y='212' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='120' y='228' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='120' y='244' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='120' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='120' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='120' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='120' y='324' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='120' y='340' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='120' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='388' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='120' y='404' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='120' y='420' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='120' y='436' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='120' y='452' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='484' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='120' y='500' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='120' y='516' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='120' y='532' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='128' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='84' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='128' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='128' y='148' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='128' y='164' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='128' y='196' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='128' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='228' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='128' y='244' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='128' y='260' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='128' y='292' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='128' y='308' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='128' y='324' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='128' y='340' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='128' y='356' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='128' y='388' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='404' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='128' y='420' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='128' y='436' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='128' y='452' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='128' y='484' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='128' y='500' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='128' y='516' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='128' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='136' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='136' y='68' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='136' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='136' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='136' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='136' y='132' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='136' y='148' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='136' y='164' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='136' y='196' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='136' y='212' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='136' y='228' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='136' y='244' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='136' y='260' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='136' y='292' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='136' y='308' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='136' y='324' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='136' y='340' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='136' y='356' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='136' y='388' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='136' y='404' fill='currentColor' style='font-size:1em'>z&lt;/text>
&lt;text text-anchor='middle' x='136' y='420' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='136' y='436' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='136' y='452' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='136' y='484' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='136' y='500' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='136' y='516' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='136' y='532' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='144' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='144' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='144' y='100' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='144' y='116' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='144' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='144' y='148' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='144' y='164' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='144' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='228' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='144' y='244' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='144' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='144' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='144' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='144' y='324' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='144' y='340' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='144' y='356' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='144' y='388' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='144' y='404' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='144' y='420' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='144' y='436' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='452' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='144' y='484' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='144' y='500' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='144' y='516' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='144' y='532' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='152' y='36' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='152' y='68' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='152' y='84' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='152' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='116' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='152' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='152' y='148' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='152' y='164' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='152' y='196' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='152' y='228' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='152' y='244' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='152' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='152' y='292' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='152' y='308' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='152' y='324' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='152' y='340' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='152' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='388' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='152' y='404' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='152' y='420' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='152' y='436' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='152' y='452' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='152' y='484' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='500' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='516' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='152' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='36' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='160' y='68' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='160' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='160' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='160' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='160' y='164' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='160' y='228' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='160' y='244' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='160' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='160' y='292' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='160' y='308' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='160' y='324' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='160' y='340' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='356' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='160' y='388' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='160' y='404' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='160' y='420' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='160' y='436' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='160' y='452' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='160' y='484' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='160' y='500' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='160' y='516' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='160' y='532' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='168' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='84' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='168' y='100' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='168' y='116' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='168' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='168' y='148' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='168' y='164' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='168' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='244' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='168' y='260' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='168' y='292' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='168' y='308' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='168' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='340' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='168' y='388' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='404' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='420' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='168' y='436' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='452' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='500' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='168' y='516' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='168' y='532' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='176' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='176' y='68' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='176' y='84' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='100' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='176' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='176' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='176' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='176' y='228' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='176' y='260' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='292' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='176' y='308' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='176' y='324' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='340' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='356' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='176' y='388' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='176' y='404' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='420' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='176' y='436' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='176' y='452' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='500' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='516' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='176' y='532' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='184' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='184' y='84' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='100' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='184' y='116' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='184' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='184' y='148' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='184' y='260' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='292' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='184' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='184' y='324' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='340' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='356' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='184' y='388' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='184' y='404' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='420' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='184' y='452' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='184' y='500' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='184' y='516' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='184' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='148' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='260' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='292' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='192' y='308' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='324' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='340' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='356' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='192' y='388' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='192' y='404' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='420' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='192' y='452' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='500' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='516' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='192' y='532' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='200' y='84' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='100' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='200' y='132' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='148' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='200' y='260' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='292' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='200' y='308' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='324' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='340' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='356' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='388' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='200' y='404' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='420' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='200' y='452' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='200' y='500' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='200' y='516' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='200' y='532' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='208' y='100' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='208' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='208' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='208' y='148' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='208' y='292' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='208' y='308' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='208' y='356' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='208' y='388' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='208' y='420' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='208' y='500' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='208' y='516' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='208' y='532' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='216' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='116' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='216' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='148' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='216' y='308' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='420' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='216' y='532' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='224' y='100' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='224' y='116' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='224' y='132' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='224' y='148' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='224' y='308' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='224' y='356' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='224' y='420' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='224' y='532' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='232' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='232' y='148' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='232' y='420' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='232' y='532' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='240' y='116' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='240' y='420' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='248' y='116' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='264' y='260' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='20' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='36' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='68' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='84' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='100' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='116' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='132' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='148' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='164' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='196' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='212' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='228' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='244' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='292' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='308' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='324' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='340' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='356' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='388' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='404' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='420' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='436' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='452' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='484' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='500' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='516' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='272' y='532' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='280' y='260' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='288' y='20' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='288' y='36' fill='currentColor' style='font-size:1em'>M&lt;/text>
&lt;text text-anchor='middle' x='288' y='68' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='288' y='84' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='288' y='100' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='288' y='116' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='288' y='132' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='288' y='148' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='288' y='164' fill='currentColor' style='font-size:1em'>Z&lt;/text>
&lt;text text-anchor='middle' x='288' y='196' fill='currentColor' style='font-size:1em'>B&lt;/text>
&lt;text text-anchor='middle' x='288' y='212' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='288' y='228' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='288' y='244' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='288' y='260' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='288' y='292' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='288' y='308' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='288' y='324' fill='currentColor' style='font-size:1em'>Q&lt;/text>
&lt;text text-anchor='middle' x='288' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='288' y='356' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='288' y='388' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='288' y='404' fill='currentColor' style='font-size:1em'>Z&lt;/text>
&lt;text text-anchor='middle' x='288' y='420' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='288' y='436' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='288' y='452' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='288' y='484' fill='currentColor' style='font-size:1em'>F&lt;/text>
&lt;text text-anchor='middle' x='288' y='500' fill='currentColor' style='font-size:1em'>M&lt;/text>
&lt;text text-anchor='middle' x='288' y='516' fill='currentColor' style='font-size:1em'>M&lt;/text>
&lt;text text-anchor='middle' x='288' y='532' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='296' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='296' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='296' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='296' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='296' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='296' y='116' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='296' y='132' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='296' y='148' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='296' y='164' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='296' y='196' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='296' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='296' y='228' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='296' y='244' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='296' y='260' fill='currentColor' style='font-size:1em'>F&lt;/text>
&lt;text text-anchor='middle' x='296' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='296' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='296' y='324' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='296' y='340' fill='currentColor' style='font-size:1em'>H&lt;/text>
&lt;text text-anchor='middle' x='296' y='356' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='296' y='388' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='296' y='404' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='296' y='420' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='296' y='436' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='296' y='452' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='296' y='484' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='296' y='500' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='296' y='516' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='296' y='532' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='304' y='20' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='304' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='68' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='304' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='304' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='304' y='116' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='304' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='304' y='148' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='304' y='164' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='304' y='196' fill='currentColor' style='font-size:1em'>M&lt;/text>
&lt;text text-anchor='middle' x='304' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='304' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='304' y='244' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='260' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='304' y='292' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='304' y='308' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='304' y='324' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='304' y='340' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='304' y='356' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='388' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='304' y='404' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='304' y='420' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='304' y='436' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='304' y='452' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='304' y='484' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='304' y='500' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='516' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='20' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='312' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='312' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='312' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='148' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='312' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='312' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='312' y='228' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='312' y='244' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='312' y='260' fill='currentColor' style='font-size:1em'>8&lt;/text>
&lt;text text-anchor='middle' x='312' y='292' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='312' y='308' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='312' y='324' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='312' y='356' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='312' y='388' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='312' y='420' fill='currentColor' style='font-size:1em'>4&lt;/text>
&lt;text text-anchor='middle' x='312' y='436' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='312' y='452' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='312' y='484' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='500' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='312' y='516' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='312' y='532' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='320' y='36' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='320' y='68' fill='currentColor' style='font-size:1em'>4&lt;/text>
&lt;text text-anchor='middle' x='320' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='116' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='320' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='320' y='148' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='320' y='196' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='320' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='228' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='320' y='244' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='292' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='308' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='320' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='320' y='340' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='320' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='320' y='388' fill='currentColor' style='font-size:1em'>4&lt;/text>
&lt;text text-anchor='middle' x='320' y='404' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='320' y='420' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='320' y='436' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='320' y='452' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='320' y='500' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='320' y='516' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='320' y='532' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='328' y='20' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='328' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='68' fill='currentColor' style='font-size:1em'>7&lt;/text>
&lt;text text-anchor='middle' x='328' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='328' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='328' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='328' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='328' y='164' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='328' y='196' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='328' y='212' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='328' y='244' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='328' y='260' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='328' y='292' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='328' y='308' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='328' y='340' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='328' y='356' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='328' y='388' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='328' y='404' fill='currentColor' style='font-size:1em'>7&lt;/text>
&lt;text text-anchor='middle' x='328' y='420' fill='currentColor' style='font-size:1em'>6&lt;/text>
&lt;text text-anchor='middle' x='328' y='452' fill='currentColor' style='font-size:1em'>4&lt;/text>
&lt;text text-anchor='middle' x='328' y='484' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='328' y='500' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='516' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='532' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='336' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='336' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='336' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='336' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='336' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='336' y='148' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='336' y='164' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='336' y='196' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='336' y='212' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='336' y='228' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='336' y='244' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='336' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='336' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='336' y='324' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='336' y='340' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='336' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='336' y='388' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='336' y='404' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='336' y='420' fill='currentColor' style='font-size:1em'>7&lt;/text>
&lt;text text-anchor='middle' x='336' y='436' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='336' y='452' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='336' y='484' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='336' y='500' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='336' y='516' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='336' y='532' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='344' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='68' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='344' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='344' y='100' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='116' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='344' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='344' y='148' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='164' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='196' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='344' y='228' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='344' y='244' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='344' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='344' y='292' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='344' y='308' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='344' y='324' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='344' y='340' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='344' y='356' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='344' y='388' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='344' y='404' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='344' y='420' fill='currentColor' style='font-size:1em'>8&lt;/text>
&lt;text text-anchor='middle' x='344' y='436' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='452' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='344' y='484' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='500' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='516' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='532' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='352' y='20' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='352' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='352' y='84' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='352' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='352' y='116' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='352' y='148' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='352' y='164' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='352' y='196' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='352' y='212' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='352' y='228' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='352' y='292' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='352' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='352' y='356' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='352' y='388' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='352' y='404' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='352' y='420' fill='currentColor' style='font-size:1em'>9&lt;/text>
&lt;text text-anchor='middle' x='352' y='436' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='352' y='452' fill='currentColor' style='font-size:1em'>6&lt;/text>
&lt;text text-anchor='middle' x='352' y='532' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='360' y='20' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='360' y='36' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='360' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='360' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='360' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='360' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='360' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='164' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='360' y='196' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='360' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='360' y='228' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='360' y='244' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='360' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='360' y='292' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='360' y='308' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='360' y='324' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='360' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='360' y='388' fill='currentColor' style='font-size:1em'>4&lt;/text>
&lt;text text-anchor='middle' x='360' y='420' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='360' y='436' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='360' y='452' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='360' y='484' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='360' y='500' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='360' y='516' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='368' y='20' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='368' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='368' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='368' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='368' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='368' y='148' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='368' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='244' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='368' y='260' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='368' y='292' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='368' y='308' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='368' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='368' y='356' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='368' y='404' fill='currentColor' style='font-size:1em'>→&lt;/text>
&lt;text text-anchor='middle' x='368' y='420' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='368' y='436' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='368' y='484' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='368' y='500' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='368' y='516' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='368' y='532' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='376' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='376' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='376' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='376' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='376' y='116' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='376' y='148' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='376' y='212' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='376' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='376' y='244' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='376' y='260' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='376' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='376' y='308' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='376' y='324' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='376' y='340' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='376' y='356' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='376' y='388' fill='currentColor' style='font-size:1em'>—&lt;/text>
&lt;text text-anchor='middle' x='376' y='420' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='376' y='452' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='376' y='484' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='376' y='500' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='376' y='516' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='376' y='532' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='384' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='84' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='384' y='100' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='384' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='384' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='148' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='384' y='244' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='384' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='384' y='308' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='384' y='340' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='384' y='404' fill='currentColor' style='font-size:1em'>7&lt;/text>
&lt;text text-anchor='middle' x='384' y='436' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='384' y='452' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='384' y='484' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='384' y='500' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='384' y='516' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='392' y='20' fill='currentColor' style='font-size:1em'>—&lt;/text>
&lt;text text-anchor='middle' x='392' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='392' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='392' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='392' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='392' y='148' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='392' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='392' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='392' y='260' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='392' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='392' y='324' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='392' y='340' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='392' y='356' fill='currentColor' style='font-size:1em'>\&lt;/text>
&lt;text text-anchor='middle' x='392' y='388' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='392' y='404' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='392' y='420' fill='currentColor' style='font-size:1em'>→&lt;/text>
&lt;text text-anchor='middle' x='392' y='436' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='392' y='452' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='392' y='484' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='392' y='516' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='392' y='532' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='400' y='36' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='400' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='400' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='400' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='400' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='400' y='212' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='400' y='228' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='400' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='400' y='292' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='400' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='400' y='324' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='400' y='340' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='400' y='356' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='400' y='388' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='400' y='404' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='400' y='436' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='400' y='452' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='400' y='484' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='400' y='500' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='400' y='516' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='400' y='532' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='408' y='20' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='408' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='408' y='68' fill='currentColor' style='font-size:1em'>x&lt;/text>
&lt;text text-anchor='middle' x='408' y='84' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='408' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='408' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='408' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='408' y='148' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='408' y='212' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='408' y='228' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='408' y='260' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='408' y='292' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='408' y='324' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='408' y='340' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='408' y='388' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='408' y='404' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='408' y='420' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='408' y='436' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='408' y='484' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='408' y='500' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='408' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='20' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='416' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='416' y='84' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='416' y='100' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='416' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='416' y='132' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='416' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='416' y='212' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='416' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='416' y='308' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='416' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='416' y='356' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='388' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='416' y='420' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='416' y='452' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='500' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='416' y='516' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='416' y='532' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='424' y='20' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='424' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='424' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='424' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='424' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='424' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='424' y='148' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='424' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='424' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='424' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='424' y='292' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='424' y='308' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='424' y='324' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='424' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='424' y='356' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='424' y='388' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='424' y='420' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='424' y='436' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='424' y='452' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='424' y='516' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='424' y='532' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='432' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='432' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='432' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='432' y='100' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='432' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='432' y='132' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='432' y='148' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='432' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='432' y='228' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='432' y='260' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='432' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='432' y='308' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='432' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='432' y='340' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='432' y='356' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='432' y='420' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='432' y='436' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='432' y='452' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='432' y='500' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='432' y='516' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='432' y='532' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='440' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='440' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='440' y='100' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='440' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='440' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='440' y='212' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='440' y='228' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='440' y='260' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='440' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='440' y='308' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='440' y='324' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='440' y='356' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='440' y='388' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='440' y='420' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='440' y='436' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='440' y='452' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='440' y='500' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='440' y='516' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='440' y='532' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='448' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='448' y='36' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='448' y='68' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='448' y='84' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='448' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='448' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='448' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='448' y='148' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='448' y='212' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='448' y='228' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='448' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='448' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='448' y='356' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='448' y='388' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='448' y='420' fill='currentColor' style='font-size:1em'>+&lt;/text>
&lt;text text-anchor='middle' x='448' y='436' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='448' y='452' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='448' y='500' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='448' y='516' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='448' y='532' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='456' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='456' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='456' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='456' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='456' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='456' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='456' y='260' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='456' y='292' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='456' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='456' y='324' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='456' y='356' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='456' y='388' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='456' y='420' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='456' y='436' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='456' y='452' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='456' y='500' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='464' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='464' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='464' y='68' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='464' y='116' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='464' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='464' y='148' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='464' y='212' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='464' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='464' y='260' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='464' y='292' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='464' y='356' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='464' y='388' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='464' y='420' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='464' y='452' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='464' y='516' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='464' y='532' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='472' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='472' y='68' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='472' y='100' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='472' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='472' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='472' y='148' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='472' y='212' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='472' y='228' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='472' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='472' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='472' y='308' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='472' y='324' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='472' y='356' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='472' y='388' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='472' y='452' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='472' y='516' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='472' y='532' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='480' y='36' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='480' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='480' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='480' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='480' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='480' y='148' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='480' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='480' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='480' y='260' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='480' y='292' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='480' y='308' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='480' y='324' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='480' y='388' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='480' y='516' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='480' y='532' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='488' y='20' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='488' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='488' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='488' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='488' y='132' fill='currentColor' style='font-size:1em'>"&lt;/text>
&lt;text text-anchor='middle' x='488' y='148' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='488' y='212' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='488' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='488' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='488' y='308' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='488' y='324' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='488' y='388' fill='currentColor' style='font-size:1em'>?&lt;/text>
&lt;text text-anchor='middle' x='488' y='452' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='488' y='516' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='488' y='532' fill='currentColor' style='font-size:1em'>'&lt;/text>
&lt;text text-anchor='middle' x='496' y='20' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='496' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='496' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='496' y='148' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='496' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='496' y='292' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='496' y='308' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='496' y='324' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='496' y='452' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='496' y='532' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='504' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='504' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='504' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='504' y='308' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='504' y='324' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='504' y='452' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='504' y='516' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='512' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='512' y='100' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='512' y='308' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='512' y='324' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='512' y='452' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='512' y='516' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='512' y='532' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='520' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='520' y='452' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='520' y='516' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='520' y='532' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='528' y='532' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='536' y='516' fill='currentColor' style='font-size:1em'>9&lt;/text>
&lt;text text-anchor='middle' x='536' y='532' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='544' y='516' fill='currentColor' style='font-size:1em'>9&lt;/text>
&lt;text text-anchor='middle' x='552' y='516' fill='currentColor' style='font-size:1em'>9&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>Each fixture is a test case. Your producer tests verify you never generate files that look like the broken fixtures. Your consumer tests verify you detect and handle each type of breakage correctly.&lt;/p>
&lt;p>The most valuable fixture? &lt;code>unclosed_quote.csv&lt;/code>. A single unclosed double-quote can cause a parser to consume every subsequent row as part of one giant field — potentially the entire rest of the file. I&amp;rsquo;ve seen this eat 2 million rows. Test for it explicitly.&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="what-to-put-in-your-data-contract">What to Put in Your Data Contract&lt;/h3>
&lt;br>
&lt;p>Tests tell you whether a specific file is good or bad. A &lt;strong>data contract&lt;/strong> tells you what &amp;ldquo;good&amp;rdquo; means. Without one, you&amp;rsquo;re testing against assumptions — and your assumptions will eventually differ from your vendor&amp;rsquo;s.&lt;/p>
&lt;p>Here&amp;rsquo;s what a data contract for CSV exchange should specify. Think of it as the RFC that RFC 4180 forgot to write:&lt;/p>
&lt;p>&lt;strong>File format:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Delimiter (comma, pipe, tab — be explicit)&lt;/li>
&lt;li>Quote character (double-quote)&lt;/li>
&lt;li>Escape method (doubled quotes, per RFC 4180)&lt;/li>
&lt;li>Line ending (CRLF)&lt;/li>
&lt;li>Encoding (UTF-8, no BOM)&lt;/li>
&lt;li>Header row policy (present, omitted for security, or conditional — and if omitted, where the schema lives)&lt;/li>
&lt;li>Column names and order (in manifest if headers are stripped)&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Data type conventions:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Date format (ISO 8601: &lt;code>YYYY-MM-DD&lt;/code>)&lt;/li>
&lt;li>Timestamp format (ISO 8601: &lt;code>YYYY-MM-DDTHH:MM:SSZ&lt;/code>)&lt;/li>
&lt;li>Null representation (empty field — no &amp;ldquo;NULL&amp;rdquo;, no &amp;ldquo;N/A&amp;rdquo;)&lt;/li>
&lt;li>Boolean representation (&lt;code>true&lt;/code>/&lt;code>false&lt;/code>)&lt;/li>
&lt;li>Numeric precision (no scientific notation, no thousands separators)&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Delivery protocol:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Transfer method (SFTP, S3, etc.)&lt;/li>
&lt;li>Manifest file required (JSON, SHA-256 checksum)&lt;/li>
&lt;li>Signal file or completion marker&lt;/li>
&lt;li>Delivery schedule and SLA&lt;/li>
&lt;li>Who to contact when something breaks&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Change management:&lt;/strong>&lt;/p>
&lt;ul>
&lt;li>Minimum notice period for schema changes&lt;/li>
&lt;li>Versioning strategy (new columns appended, never reordered)&lt;/li>
&lt;li>Breaking vs non-breaking change definitions&lt;/li>
&lt;/ul>
&lt;p>You don&amp;rsquo;t need a 50-page legal document. A one-page markdown file committed to a shared repository works. The point isn&amp;rsquo;t formality — it&amp;rsquo;s explicitness. When the vendor&amp;rsquo;s export changes and your pipeline breaks at 3am, you want something to point to that says &amp;ldquo;this is what we agreed.&amp;rdquo;&lt;/p>
&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="putting-it-all-together-a-reusable-validation-function">Putting It All Together: A Reusable Validation Function&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s a practical, copy-paste-ready validation function that covers the most critical checks. This isn&amp;rsquo;t a framework — it&amp;rsquo;s a function you can drop into any Python project and start using immediately.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> csv
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> hashlib
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> dataclasses &lt;span style="color:#f92672">import&lt;/span> dataclass, field
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> typing &lt;span style="color:#f92672">import&lt;/span> Optional
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#a6e22e">@dataclass&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">class&lt;/span> &lt;span style="color:#a6e22e">CSVValidationResult&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Results from validating a CSV file.&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> filepath: str
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_valid: bool &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">True&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> errors: list &lt;span style="color:#f92672">=&lt;/span> field(default_factory&lt;span style="color:#f92672">=&lt;/span>list)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> warnings: list &lt;span style="color:#f92672">=&lt;/span> field(default_factory&lt;span style="color:#f92672">=&lt;/span>list)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> stats: dict &lt;span style="color:#f92672">=&lt;/span> field(default_factory&lt;span style="color:#f92672">=&lt;/span>dict)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">add_error&lt;/span>(self, message: str):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> self&lt;span style="color:#f92672">.&lt;/span>errors&lt;span style="color:#f92672">.&lt;/span>append(message)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> self&lt;span style="color:#f92672">.&lt;/span>is_valid &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">False&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">add_warning&lt;/span>(self, message: str):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> self&lt;span style="color:#f92672">.&lt;/span>warnings&lt;span style="color:#f92672">.&lt;/span>append(message)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">validate_csv&lt;/span>(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> filepath: str,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected_columns: Optional[list] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected_encoding: str &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;utf-8&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_columns: Optional[list] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> zero_padded_columns: Optional[list] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> null_convention: str &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_row_count: Optional[int] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> min_row_count: Optional[int] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#f92672">-&amp;gt;&lt;/span> CSVValidationResult:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Validate a CSV file against common quality checks.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Works for both producers (checking your own output)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> and consumers (checking vendor deliveries).
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result &lt;span style="color:#f92672">=&lt;/span> CSVValidationResult(filepath&lt;span style="color:#f92672">=&lt;/span>filepath)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># --- Check 1: File exists and isn&amp;#39;t empty ---&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>exists(filepath):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;File not found: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>filepath&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_size &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>getsize(filepath)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file_size &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(&lt;span style="color:#e6db74">&amp;#34;File is empty (zero bytes)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;file_size_bytes&amp;#39;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> file_size
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># --- Check 2: Encoding ---&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(filepath, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> raw_head &lt;span style="color:#f92672">=&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>read(&lt;span style="color:#ae81ff">10000&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># BOM check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> raw_head[:&lt;span style="color:#ae81ff">3&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\xef\xbb\xbf&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_warning(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;File contains UTF-8 BOM — this will corrupt &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;the first column name in most parsers&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Null byte check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\x00&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> raw_head:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;File contains null bytes — likely wrong encoding &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;(e.g., UTF-16 read as UTF-8) or binary data&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Encoding validity&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> raw_full &lt;span style="color:#f92672">=&lt;/span> open(filepath, &lt;span style="color:#e6db74">&amp;#39;rb&amp;#39;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>read()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> raw_full&lt;span style="color:#f92672">.&lt;/span>decode(expected_encoding)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">UnicodeDecodeError&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;File is not valid &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected_encoding&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Decode error at byte position &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>e&lt;span style="color:#f92672">.&lt;/span>start&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># --- Check 3: Structural integrity ---&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> encoding &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;utf-8-sig&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> raw_head[:&lt;span style="color:#ae81ff">3&lt;/span>] &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\xef\xbb\xbf&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">else&lt;/span> expected_encoding
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(filepath, newline&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>, encoding&lt;span style="color:#f92672">=&lt;/span>encoding) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> reader &lt;span style="color:#f92672">=&lt;/span> csv&lt;span style="color:#f92672">.&lt;/span>reader(f)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Header check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> header &lt;span style="color:#f92672">=&lt;/span> next(reader)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">StopIteration&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(&lt;span style="color:#e6db74">&amp;#34;File has no rows (not even a header)&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;column_count&amp;#39;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(header)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;columns&amp;#39;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> header
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Duplicate column check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> seen &lt;span style="color:#f92672">=&lt;/span> {}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> header:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> seen[col] &lt;span style="color:#f92672">=&lt;/span> seen&lt;span style="color:#f92672">.&lt;/span>get(col, &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#f92672">+&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dupes &lt;span style="color:#f92672">=&lt;/span> {k: v &lt;span style="color:#66d9ef">for&lt;/span> k, v &lt;span style="color:#f92672">in&lt;/span> seen&lt;span style="color:#f92672">.&lt;/span>items() &lt;span style="color:#66d9ef">if&lt;/span> v &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dupes:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Duplicate column names: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>dupes&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Expected columns check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> expected_columns &lt;span style="color:#f92672">and&lt;/span> header &lt;span style="color:#f92672">!=&lt;/span> expected_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column mismatch.&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; Expected: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>expected_columns&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34; Got: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>header&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Row-level checks&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> row_count &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ragged_rows &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> iso_date &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>compile(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;^\d&lt;/span>&lt;span style="color:#e6db74">{4}&lt;/span>&lt;span style="color:#e6db74">-\d&lt;/span>&lt;span style="color:#e6db74">{2}&lt;/span>&lt;span style="color:#e6db74">-\d&lt;/span>&lt;span style="color:#e6db74">{2}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> row_num, row &lt;span style="color:#f92672">in&lt;/span> enumerate(reader, start&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> row_count &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Column count consistency&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(row) &lt;span style="color:#f92672">!=&lt;/span> len(header):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ragged_rows&lt;span style="color:#f92672">.&lt;/span>append((row_num, len(row)))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Date format check&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> date_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> date_columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> idx &lt;span style="color:#f92672">=&lt;/span> header&lt;span style="color:#f92672">.&lt;/span>index(col) &lt;span style="color:#66d9ef">if&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> header &lt;span style="color:#66d9ef">else&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> idx &lt;span style="color:#f92672">is&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span> &lt;span style="color:#f92672">and&lt;/span> row[idx]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> iso_date&lt;span style="color:#f92672">.&lt;/span>&lt;span style="color:#66d9ef">match&lt;/span>(row[idx]):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_warning(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Row &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row_num&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">, column &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;: &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Non-ISO date &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row[idx]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;row_count&amp;#39;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> row_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> ragged_rows:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(ragged_rows)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows have wrong column count. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Expected &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(header)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> columns. &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;First bad row: line &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ragged_rows[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;has &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ragged_rows[&lt;span style="color:#ae81ff">0&lt;/span>][&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> fields.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Row count bounds&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> min_row_count &lt;span style="color:#f92672">and&lt;/span> row_count &lt;span style="color:#f92672">&amp;lt;&lt;/span> min_row_count:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Row count &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> is below minimum &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;threshold of &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>min_row_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> max_row_count &lt;span style="color:#f92672">and&lt;/span> row_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> max_row_count:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_error(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Row count &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>row_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> exceeds maximum &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;threshold of &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>max_row_count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># --- Check 4: Invisible characters ---&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> invisible &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u00a0&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Non-breaking space&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\u200b&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;Zero-width space&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;&lt;/span>&lt;span style="color:#ae81ff">\ufeff&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;BOM (mid-file)&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> raw_full&lt;span style="color:#f92672">.&lt;/span>decode(encoding, errors&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#39;replace&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> char, name &lt;span style="color:#f92672">in&lt;/span> invisible&lt;span style="color:#f92672">.&lt;/span>items():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> count &lt;span style="color:#f92672">=&lt;/span> text&lt;span style="color:#f92672">.&lt;/span>count(char)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>add_warning(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>count&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; characters (U+&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>ord(char)&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">04X&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">)&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># --- Compute checksum for manifest comparison ---&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sha256 &lt;span style="color:#f92672">=&lt;/span> hashlib&lt;span style="color:#f92672">.&lt;/span>sha256(raw_full)&lt;span style="color:#f92672">.&lt;/span>hexdigest()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;sha256&amp;#39;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> sha256
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> result
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Use it like this:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>result &lt;span style="color:#f92672">=&lt;/span> validate_csv(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;vendor_delivery/customers_20260304.csv&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> expected_columns&lt;span style="color:#f92672">=&lt;/span>[&lt;span style="color:#e6db74">&amp;#39;id&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;name&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;email&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;city&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;state&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;zip&amp;#39;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> date_columns&lt;span style="color:#f92672">=&lt;/span>[&lt;span style="color:#e6db74">&amp;#39;created_date&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;updated_date&amp;#39;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> min_row_count&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">1000&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_row_count&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">100000&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> result&lt;span style="color:#f92672">.&lt;/span>is_valid:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> error &lt;span style="color:#f92672">in&lt;/span> result&lt;span style="color:#f92672">.&lt;/span>errors:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;ERROR: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>error&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Route to quarantine&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Valid. &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;row_count&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> rows, &amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SHA-256: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>result&lt;span style="color:#f92672">.&lt;/span>stats[&lt;span style="color:#e6db74">&amp;#39;sha256&amp;#39;&lt;/span>][:&lt;span style="color:#ae81ff">16&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Proceed to loading&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;/br>
&lt;/br>
&lt;h3 id="what-16000-missing-records-taught-us">What 16,000 Missing Records Taught Us&lt;/h3>
&lt;br>
&lt;p>Let&amp;rsquo;s come back to where we started. Sixteen thousand COVID test results, dropped silently because nobody tested what would happen when the file exceeded a row limit. The labs did their job. The CSV was generated correctly. The file was transmitted successfully. Every individual component worked. The system failed because nobody tested the &lt;em>boundaries&lt;/em> — the places where one component&amp;rsquo;s assumptions meet another component&amp;rsquo;s limitations.&lt;/p>
&lt;p>That&amp;rsquo;s what CSV testing is really about. It&amp;rsquo;s not about whether you can parse a comma-separated file — of course you can. It&amp;rsquo;s about whether you&amp;rsquo;ve thought through every place where the producer&amp;rsquo;s assumptions and the consumer&amp;rsquo;s assumptions might diverge. The vendor who exports &lt;code>NULL&lt;/code> as the literal string &amp;ldquo;NULL.&amp;rdquo; The CRM that allows newlines in address fields. The business user who opens your carefully formatted CSV in Excel, saves it, and sends it back with stripped zeros and mangled dates.&lt;/p>
&lt;p>Every one of those is a test case. Every one of those is a boundary you can verify during development instead of discovering in production.&lt;/p>
&lt;p>The engineers who sleep well at night aren&amp;rsquo;t the ones who assume their CSV files are fine. They&amp;rsquo;re the ones who wrote the tests to prove it.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>CSV</category><category>Data Quality</category><category>Testing</category><category>RFC 4180</category><category>Python</category><category>Data Engineering</category><category>Data Pipelines</category></item><item><title>Write-Audit-Publish with Iceberg Tables in Snowflake</title><link>https://ghostinthedata.info/posts/2026/2026-02-27-wap-iceberg-branching/</link><pubDate>Fri, 27 Feb 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-02-27-wap-iceberg-branching/</guid><author>Chris Hillman</author><description>A hands-on tutorial for implementing the WAP pattern on Apache Iceberg tables in Snowflake — with synthetic data, quality gates, and the zero-I/O publish step that keeps production safe.</description><content:encoded>&lt;p>It was a Tuesday afternoon when the analyst pinged me on Microsoft Teams: &amp;ldquo;Hey, the Total Portfolio numbers just jumped 40% overnight. Did we land a whale?&amp;rdquo;&lt;/p>
&lt;p>We hadn&amp;rsquo;t.&lt;/p>
&lt;p>What actually happened was more mundane and significantly more painful. A schema change in the source system introduced a currency conversion bug. Our pipeline dutifully loaded the corrupted data into production at 3 AM, the dashboards updated by 6 AM, and the Department Head opened her morning report to numbers that looked like champagne-worthy growth.&lt;/p>
&lt;p>Here&amp;rsquo;s the thing — we had data quality tests. They ran &lt;em>after&lt;/em> the data hit production. A classic case of closing the gate after the horse has bolted, kicked over the fence, and trampled the neighbour&amp;rsquo;s garden.&lt;/p>
&lt;p>That incident changed how I think about data pipelines entirely. It led me to the &lt;strong>Write-Audit-Publish&lt;/strong> pattern — a concept championed heavily by Zach Wilson. Zach frames it simply: write to a staging area with an identical schema to production, run your data quality checks, and only publish to production if they pass. It&amp;rsquo;s deceptively simple. And it works.&lt;/p>
&lt;p>I&amp;rsquo;ve &lt;a href="https://ghostinthedata.info/posts/2025/2025-05-18-wap-data-pipelines/" target="_blank" rel="noopener">written about WAP before&lt;/a> using Airflow as the orchestration layer, with traditional staging tables and swap mechanics. That approach is solid, battle-tested, and it&amp;rsquo;ll serve most teams well. But now that Apache Iceberg tables in Snowflake have matured — there&amp;rsquo;s a more elegant way to implement WAP that eliminates one of the pattern&amp;rsquo;s biggest practical headaches: the I/O overhead.&lt;/p>
&lt;p>In this tutorial, we&amp;rsquo;re going to build a complete WAP test harness from scratch on Snowflake Iceberg tables. We&amp;rsquo;ll create synthetic data, simulate a change that introduces bad data, catch it with quality gates, and publish only the clean data to production. Along the way, I&amp;rsquo;ll show you why the zero-copy mechanics of Iceberg&amp;rsquo;s architecture make the publish step insanely fast — and why that speed changes the calculus on when WAP is worth the effort.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="why-the-publish-step-is-where-traditional-wap-hurts">Why the Publish Step Is Where Traditional WAP Hurts&lt;/h3>
&lt;br>
&lt;p>If you&amp;rsquo;ve worked in data engineering for more than a few months, you&amp;rsquo;ve probably built some version of WAP without calling it that. You load data into a staging table, run some checks, then copy the good stuff into production. That works. Sort of.&lt;/p>
&lt;p>The problem is what happens at the publish step. In a traditional staging-table WAP implementation, publishing means physically moving data. You&amp;rsquo;re running an &lt;code>INSERT INTO production SELECT * FROM staging&lt;/code>, or you&amp;rsquo;re doing a table swap that involves Snowflake creating new micro-partitions. Either way, you&amp;rsquo;re generating I/O. You&amp;rsquo;re writing data that already exists in a perfectly good format, to another location, in another format, just so it lives under the right table name.&lt;/p>
&lt;p>For a table with a few million rows, nobody notices. For a table with a few billion rows partitioned across hundreds of files? That publish step becomes a pipeline bottleneck. I&amp;rsquo;ve seen WAP implementations where the audit takes 30 seconds and the publish takes 45 minutes. The quality gate is fast. The bureaucratic data shuffle is slow.&lt;/p>
&lt;p>This is where Iceberg changes the equation.&lt;/p>
&lt;p>Iceberg tables don&amp;rsquo;t store data inside Snowflake&amp;rsquo;s proprietary micro-partition format. The data lives as Parquet files in your external storage — S3, GCS, or Azure Blob. Snowflake manages the metadata that tells it which Parquet files belong to which snapshot of the table. When you do a &lt;code>CLONE&lt;/code> of an Iceberg table in Snowflake, it creates a new metadata pointer to the same underlying Parquet files. &lt;strong>No data moves. No files are copied. No I/O is generated.&lt;/strong>&lt;/p>
&lt;p>And when you &lt;code>SWAP&lt;/code> a validated clone back into production? Again — it&amp;rsquo;s a metadata operation. Snowflake updates which metadata pointer the production table name resolves to. The Parquet files don&amp;rsquo;t move. The snapshots don&amp;rsquo;t change. The swap is essentially instant regardless of table size.&lt;/p>
&lt;p>Let me put that in concrete terms. Say you have a 500 GB Iceberg table with three years of order history. In a traditional WAP setup, publishing validated staging data means writing hundreds of gigabytes to new micro-partitions — that&amp;rsquo;s real compute cost, real wall-clock time, and a real window where downstream consumers might see partial data. With Iceberg&amp;rsquo;s zero-copy clone and swap, the same publish step takes seconds. Not minutes. Seconds. Because you&amp;rsquo;re flipping a pointer, not copying bytes.&lt;/p>
&lt;p>This isn&amp;rsquo;t a marginal improvement. It fundamentally changes when WAP is worth implementing. When publish is expensive, you think twice about adding WAP to every pipeline. When publish is free, the question flips — why &lt;em>wouldn&amp;rsquo;t&lt;/em> you gate every production write with quality checks?&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="the-conceptual-model-how-wap-works-on-iceberg">The Conceptual Model: How WAP Works on Iceberg&lt;/h3>
&lt;br>
&lt;p>Here&amp;rsquo;s the flow we&amp;rsquo;re going to build. Every step maps to actual SQL you can run in Snowflake today:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 504 313"
 >
 &lt;g transform='translate(8,16)'>
&lt;path d='M 424,208 L 432,192' fill='none' stroke='currentColor'>&lt;/path>
&lt;polygon points='128.000000,128.000000 116.000000,122.400002 116.000000,133.600006' fill='currentColor' transform='rotate(90.000000, 120.000000, 128.000000)'>&lt;/polygon>
&lt;circle cx='432' cy='192' r='6' stroke='currentColor' fill='#fff'>&lt;/circle>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='52' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='68' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='84' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='100' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='116' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='132' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='148' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='164' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>W&lt;/text>
&lt;text text-anchor='middle' x='64' y='100' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='64' y='164' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='72' y='100' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='72' y='164' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='80' y='100' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='80' y='164' fill='currentColor' style='font-size:1em'>B&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='88' y='100' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='88' y='164' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='96' y='4' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='96' y='36' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='96' y='100' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='96' y='164' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='96' y='180' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='96' y='196' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='96' y='212' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='96' y='228' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='96' y='244' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='96' y='260' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>B&lt;/text>
&lt;text text-anchor='middle' x='104' y='36' fill='currentColor' style='font-size:1em'>:&lt;/text>
&lt;text text-anchor='middle' x='104' y='100' fill='currentColor' style='font-size:1em'>:&lt;/text>
&lt;text text-anchor='middle' x='104' y='164' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='104' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='104' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='112' y='52' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='112' y='68' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='112' y='116' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='112' y='164' fill='currentColor' style='font-size:1em'>H&lt;/text>
&lt;text text-anchor='middle' x='112' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='112' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='120' y='36' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='120' y='52' fill='currentColor' style='font-size:1em'>z&lt;/text>
&lt;text text-anchor='middle' x='120' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='120' y='100' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='120' y='116' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='128' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='128' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='128' y='100' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='128' y='116' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='128' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='196' fill='currentColor' style='font-size:1em'>✅&lt;/text>
&lt;text text-anchor='middle' x='128' y='260' fill='currentColor' style='font-size:1em'>❌&lt;/text>
&lt;text text-anchor='middle' x='136' y='4' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='136' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='136' y='52' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='136' y='68' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='136' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='136' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='136' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='136' y='164' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='144' y='4' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='144' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='144' y='52' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='144' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='144' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='144' y='196' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='144' y='260' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='152' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='152' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='152' y='52' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='152' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='152' y='100' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='152' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='152' y='164' fill='currentColor' style='font-size:1em'>R&lt;/text>
&lt;text text-anchor='middle' x='152' y='196' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='152' y='212' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='152' y='228' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='152' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='152' y='276' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='152' y='292' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='160' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='160' y='52' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='160' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='100' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='160' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='160' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='164' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='160' y='196' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='160' y='212' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='160' y='228' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='160' y='260' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='160' y='276' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='160' y='292' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='168' y='4' fill='currentColor' style='font-size:1em'>_&lt;/text>
&lt;text text-anchor='middle' x='168' y='36' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='168' y='52' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='168' y='68' fill='currentColor' style='font-size:1em'>w&lt;/text>
&lt;text text-anchor='middle' x='168' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='168' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='164' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='168' y='212' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='168' y='228' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='168' y='276' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='168' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='176' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='176' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='176' y='52' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='176' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='176' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='176' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='176' y='164' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='176' y='196' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='176' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='176' y='228' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='176' y='260' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='176' y='292' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='184' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='184' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='184' y='52' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='184' y='68' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='184' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='184' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='184' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='184' y='164' fill='currentColor' style='font-size:1em'>B&lt;/text>
&lt;text text-anchor='middle' x='184' y='196' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='184' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='184' y='260' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='184' y='276' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='192' y='4' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='192' y='36' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='192' y='52' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='192' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='192' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='192' y='116' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='192' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='192' y='164' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='192' y='196' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='212' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='192' y='228' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='192' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='276' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='192' y='292' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='200' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='200' y='36' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='200' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='200' y='100' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='200' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='200' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='200' y='164' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='200' y='196' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='200' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='200' y='228' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='200' y='260' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='200' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='200' y='292' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='208' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='208' y='36' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='208' y='52' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='208' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='208' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='208' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='208' y='164' fill='currentColor' style='font-size:1em'>K&lt;/text>
&lt;text text-anchor='middle' x='208' y='196' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='208' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='208' y='228' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='208' y='260' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='208' y='276' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='208' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='216' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='216' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='216' y='100' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='216' y='116' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='216' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='216' y='164' fill='currentColor' style='font-size:1em'>:&lt;/text>
&lt;text text-anchor='middle' x='216' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='216' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='216' y='276' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='224' y='4' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;text text-anchor='middle' x='224' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='224' y='52' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='224' y='68' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='224' y='100' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='224' y='116' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='224' y='132' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='224' y='228' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='224' y='260' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='224' y='276' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='224' y='292' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='232' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='232' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='232' y='68' fill='currentColor' style='font-size:1em'>N&lt;/text>
&lt;text text-anchor='middle' x='232' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='232' y='116' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='232' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='232' y='196' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='232' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='232' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='232' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='232' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='232' y='292' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='240' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='240' y='52' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='240' y='68' fill='currentColor' style='font-size:1em'>T&lt;/text>
&lt;text text-anchor='middle' x='240' y='100' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='240' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='240' y='196' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='240' y='212' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='240' y='228' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='240' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='240' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='240' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='248' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='248' y='68' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='248' y='100' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='248' y='116' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='248' y='132' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='248' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='248' y='212' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='248' y='228' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='248' y='260' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='248' y='292' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='256' y='36' fill='currentColor' style='font-size:1em'>→&lt;/text>
&lt;text text-anchor='middle' x='256' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='256' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='256' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='256' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='256' y='212' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='256' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='256' y='260' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='256' y='276' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='256' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='264' y='52' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='264' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='264' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='264' y='132' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='264' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='264' y='276' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='272' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='272' y='68' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='272' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='272' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='272' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='272' y='196' fill='currentColor' style='font-size:1em'>→&lt;/text>
&lt;text text-anchor='middle' x='272' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='272' y='228' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='272' y='260' fill='currentColor' style='font-size:1em'>→&lt;/text>
&lt;text text-anchor='middle' x='272' y='276' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='272' y='292' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='280' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='280' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='280' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='280' y='100' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='280' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='280' y='132' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='280' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='280' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='280' y='292' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='288' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='288' y='52' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='288' y='100' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='288' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='288' y='196' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='288' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='288' y='228' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='288' y='260' fill='currentColor' style='font-size:1em'>Q&lt;/text>
&lt;text text-anchor='middle' x='288' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='288' y='292' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='296' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='296' y='68' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='296' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='296' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='296' y='132' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='296' y='196' fill='currentColor' style='font-size:1em'>W&lt;/text>
&lt;text text-anchor='middle' x='296' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='296' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='296' y='260' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='296' y='276' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='296' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='304' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='304' y='52' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='304' y='68' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='304' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='304' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='304' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='304' y='196' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='304' y='212' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='304' y='228' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='304' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='304' y='276' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='304' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='312' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='312' y='52' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='312' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='312' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='312' y='116' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='312' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='312' y='196' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='312' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='260' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='312' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='292' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='320' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='320' y='52' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='320' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='320' y='116' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='320' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='320' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='320' y='228' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='320' y='292' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='328' y='52' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;text text-anchor='middle' x='328' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='132' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='328' y='196' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='328' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='328' y='228' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='328' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='328' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='328' y='292' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='336' y='36' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='336' y='100' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='336' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='336' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='336' y='196' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='336' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='336' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='336' y='260' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='336' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='336' y='292' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='344' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='344' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='344' y='116' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='344' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='344' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='344' y='276' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='344' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='100' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='352' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='352' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='352' y='276' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='352' y='292' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='100' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='360' y='132' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='360' y='196' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='360' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='360' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='368' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='116' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='368' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='368' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='368' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='368' y='292' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='376' y='116' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='376' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='376' y='196' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='376' y='212' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='376' y='260' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='376' y='276' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='376' y='292' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='384' y='132' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='384' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='384' y='276' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='292' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='392' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='392' y='132' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='392' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='392' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='392' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='392' y='292' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='400' y='116' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='400' y='132' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='400' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='400' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='400' y='292' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='408' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='408' y='132' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='408' y='260' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='408' y='292' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='416' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='416' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='416' y='196' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='416' y='212' fill='currentColor' style='font-size:1em'>I&lt;/text>
&lt;text text-anchor='middle' x='416' y='260' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='292' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='424' y='116' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='424' y='132' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='424' y='196' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='424' y='260' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='424' y='292' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='432' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='432' y='132' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;text text-anchor='middle' x='432' y='212' fill='currentColor' style='font-size:1em'>O&lt;/text>
&lt;text text-anchor='middle' x='432' y='260' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='432' y='292' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='440' y='116' fill='currentColor' style='font-size:1em'>,&lt;/text>
&lt;text text-anchor='middle' x='440' y='196' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='440' y='212' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;text text-anchor='middle' x='440' y='260' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='440' y='292' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;text text-anchor='middle' x='448' y='196' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='448' y='260' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='456' y='196' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='456' y='260' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='464' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='472' y='196' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='480' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='488' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>The beauty of this model is that production is &lt;strong>never exposed to unvalidated data&lt;/strong>. The clone is a sandbox. Downstream consumers querying &lt;code>fct_orders&lt;/code> see the same clean data they saw yesterday, right up until the moment we swap — and the swap is atomic. There&amp;rsquo;s no window of inconsistency. No partial writes. No &amp;ldquo;oops, the pipeline failed halfway through the publish.&amp;rdquo;&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="setting-up-the-foundation">Setting Up the Foundation&lt;/h3>
&lt;br>
&lt;p>Let&amp;rsquo;s build this thing. You&amp;rsquo;ll need a Snowflake account with access to create Iceberg tables, which means an &lt;strong>external volume&lt;/strong> pointing to your cloud storage and the appropriate privileges.&lt;/p>
&lt;h4 id="prerequisites">Prerequisites&lt;/h4>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create our tutorial workspace
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">DATABASE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> wap_tutorial;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">SCHEMA&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> wap_tutorial.iceberg_wap;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>USE &lt;span style="color:#66d9ef">DATABASE&lt;/span> wap_tutorial;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>USE &lt;span style="color:#66d9ef">SCHEMA&lt;/span> iceberg_wap;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create external volume (adjust for your cloud provider)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- You&amp;#39;ll need to configure the IAM trust relationship first
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- See Snowflake docs for the cloud-specific IAM setup
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">EXTERNAL&lt;/span> VOLUME wap_ext_vol
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> STORAGE_LOCATIONS &lt;span style="color:#f92672">=&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> NAME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;wap-iceberg-storage&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> STORAGE_BASE_URL &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;s3://your-bucket/iceberg-wap/&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> STORAGE_PROVIDER &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;S3&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> STORAGE_AWS_ROLE_ARN &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;arn:aws:iam::123456789012:role/snowflake-iceberg-role&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> );
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Verify the volume is healthy
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">DESC&lt;/span> &lt;span style="color:#66d9ef">EXTERNAL&lt;/span> VOLUME wap_ext_vol;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If the external volume setup feels unfamiliar, don&amp;rsquo;t stress — the Snowflake documentation for this is thorough. The IAM configuration is the fiddly part, but it&amp;rsquo;s a one-time cost. Once the volume exists, everything downstream is straightforward SQL.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="step-1-write--creating-the-iceberg-table-and-synthetic-data">Step 1: WRITE — Creating the Iceberg Table and Synthetic Data&lt;/h3>
&lt;br>
&lt;p>We&amp;rsquo;re going to simulate an e-commerce order pipeline. This is a scenario practically every data engineer has encountered — daily order data flowing from a transactional system into the warehouse.&lt;/p>
&lt;p>First, create the Iceberg table using Snowflake as the catalog:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create the Iceberg table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Using Snowflake as the catalog means Snowflake handles
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- all lifecycle maintenance (compaction, metadata updates)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> ICEBERG &lt;span style="color:#66d9ef">TABLE&lt;/span> fct_orders (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> product_id INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quantity INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> unit_price DECIMAL(&lt;span style="color:#ae81ff">10&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> currency_code VARCHAR(&lt;span style="color:#ae81ff">3&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> total_amount DECIMAL(&lt;span style="color:#ae81ff">12&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_status VARCHAR(&lt;span style="color:#ae81ff">20&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> region VARCHAR(&lt;span style="color:#ae81ff">50&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> loaded_at TIMESTAMP_NTZ &lt;span style="color:#66d9ef">DEFAULT&lt;/span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CATALOG&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SNOWFLAKE&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> EXTERNAL_VOLUME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;wap_ext_vol&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> BASE_LOCATION &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;fct_orders/&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now let&amp;rsquo;s generate our initial &amp;ldquo;production&amp;rdquo; data. This represents the clean, trusted baseline that&amp;rsquo;s been accumulating for the past 30 days. Five thousand orders, all in USD, no nulls, no negative amounts — a pristine starting point:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Generate 30 days of clean, synthetic order data
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This is our trusted production baseline
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> fct_orders (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id, customer_id, order_date, product_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quantity, unit_price, currency_code, total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_status, region
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROW_NUMBER() OVER (&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> SEQ4()) &lt;span style="color:#f92672">+&lt;/span> &lt;span style="color:#ae81ff">10000&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> order_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">500&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATEADD(&lt;span style="color:#e6db74">&amp;#39;day&amp;#39;&lt;/span>, &lt;span style="color:#f92672">-&lt;/span>UNIFORM(&lt;span style="color:#ae81ff">0&lt;/span>, &lt;span style="color:#ae81ff">29&lt;/span>, RANDOM()), &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()) &lt;span style="color:#66d9ef">AS&lt;/span> order_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">50&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> product_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> quantity,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(UNIFORM(&lt;span style="color:#ae81ff">5&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, &lt;span style="color:#ae81ff">200&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, RANDOM())::DECIMAL(&lt;span style="color:#ae81ff">10&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>), &lt;span style="color:#ae81ff">2&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> unit_price,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> currency_code,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- total_amount = quantity * unit_price (clean, correct calculation)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> ROUND(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>, RANDOM()) &lt;span style="color:#f92672">*&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">5&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, &lt;span style="color:#ae81ff">200&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, RANDOM())::DECIMAL(&lt;span style="color:#ae81ff">12&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#ae81ff">2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">AS&lt;/span> total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PENDING&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CONFIRMED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SHIPPED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">4&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;DELIVERED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CANCELLED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> order_status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;APAC&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;EMEA&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;AMERICAS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">4&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;LATAM&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ANZ&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> region
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span>(GENERATOR(ROWCOUNT &lt;span style="color:#f92672">=&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">5000&lt;/span>));
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Let&amp;rsquo;s verify our baseline is healthy before we mess it up:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Quick sanity check on our production data
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> total_orders,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> customer_id) &lt;span style="color:#66d9ef">AS&lt;/span> unique_customers,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MIN&lt;/span>(order_date) &lt;span style="color:#66d9ef">AS&lt;/span> earliest_order,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(order_date) &lt;span style="color:#66d9ef">AS&lt;/span> latest_order,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(&lt;span style="color:#66d9ef">AVG&lt;/span>(total_amount), &lt;span style="color:#ae81ff">2&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> avg_order_value,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> negative_amounts,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#f92672">!=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> non_usd_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>You should see 5,000 orders, zero negative amounts, and zero non-USD orders. This is clean production data. Remember these numbers — because we&amp;rsquo;re about to introduce some chaos.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="simulating-the-change-when-bad-data-arrives">Simulating the Change: When Bad Data Arrives&lt;/h3>
&lt;br>
&lt;p>Now the realistic part. A new batch of 500 orders arrives from the source system. But this batch has problems — the kind of subtle, infuriating problems that slip past casual inspection but wreck downstream analytics.&lt;/p>
&lt;p>First, we create the zero-copy clone. This is our staging sandbox:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create a zero-copy clone of production
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This is a METADATA operation — no Parquet files are copied
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The clone shares the same underlying data files as production
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> ICEBERG &lt;span style="color:#66d9ef">TABLE&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> CLONE fct_orders;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That &lt;code>CLONE&lt;/code> just happened in under a second regardless of whether your table is 5,000 rows or 500 million. No data moved. The staging clone points to the same Parquet files in your external storage. Snowflake just created a new metadata entry that says &amp;ldquo;this table currently looks like &lt;em>that&lt;/em> table.&amp;rdquo;&lt;/p>
&lt;p>Now let&amp;rsquo;s load the problematic batch into our staging clone:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Simulate incoming data with realistic bugs:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- 1. Some NULL currency codes (source system migration issue)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- 2. Some invalid currency codes (mapping error)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- 3. Some negative total_amounts (conversion bug)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- 4. Some EUR amounts that weren&amp;#39;t actually converted
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> fct_orders_staging (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_id, customer_id, order_date, product_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> quantity, unit_price, currency_code, total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_status, region
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROW_NUMBER() OVER (&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> SEQ4()) &lt;span style="color:#f92672">+&lt;/span> &lt;span style="color:#ae81ff">20000&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> order_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">500&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>() &lt;span style="color:#66d9ef">AS&lt;/span> order_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">50&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> product_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>, RANDOM()) &lt;span style="color:#66d9ef">AS&lt;/span> quantity,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(UNIFORM(&lt;span style="color:#ae81ff">5&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, &lt;span style="color:#ae81ff">200&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, RANDOM())::DECIMAL(&lt;span style="color:#ae81ff">10&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>), &lt;span style="color:#ae81ff">2&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> unit_price,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- HERE&amp;#39;S THE BUG: some orders arrive with wrong currency codes
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">100&lt;/span>, RANDOM()) &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">85&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">100&lt;/span>, RANDOM()) &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">5&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#75715e">-- Null currency!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">100&lt;/span>, RANDOM()) &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">5&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;INVALID&amp;#39;&lt;/span> &lt;span style="color:#75715e">-- Invalid code!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span> &lt;span style="color:#75715e">-- EUR amount but never converted to USD
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> currency_code,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- And some totals went negative due to a sign error
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">100&lt;/span>, RANDOM()) &lt;span style="color:#f92672">&amp;lt;=&lt;/span> &lt;span style="color:#ae81ff">90&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> ROUND(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">*&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">5&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, &lt;span style="color:#ae81ff">200&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, RANDOM())::DECIMAL(&lt;span style="color:#ae81ff">12&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>), &lt;span style="color:#ae81ff">2&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> ROUND(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#f92672">*&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">*&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">5&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, &lt;span style="color:#ae81ff">200&lt;/span>.&lt;span style="color:#ae81ff">00&lt;/span>, RANDOM())::DECIMAL(&lt;span style="color:#ae81ff">12&lt;/span>,&lt;span style="color:#ae81ff">2&lt;/span>), &lt;span style="color:#ae81ff">2&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> total_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PENDING&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CONFIRMED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;SHIPPED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">4&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;DELIVERED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CANCELLED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> order_status,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> UNIFORM(&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>, RANDOM())
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;APAC&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;EMEA&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;AMERICAS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#ae81ff">4&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;LATAM&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ANZ&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> region
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">TABLE&lt;/span>(GENERATOR(ROWCOUNT &lt;span style="color:#f92672">=&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">500&lt;/span>));
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The write step is done. We have 500 new orders in our staging clone — some clean, some dirty. And critically, production is completely untouched. Anyone querying &lt;code>fct_orders&lt;/code> right now sees exactly what they saw before. No Department Head is going to message you on Teams this morning about phantom Total Portfolio growth.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="step-2-audit--the-quality-gate-battery">Step 2: AUDIT — The Quality Gate Battery&lt;/h3>
&lt;br>
&lt;p>This is where WAP earns its keep. We run every check we care about against the staging clone. If anything fails, we either fix it or drop the clone and production never sees the bad data. As Zach puts it — downstream pipelines wait on production data to be &lt;em>ready&lt;/em>. Not just &lt;em>present&lt;/em>.&lt;/p>
&lt;p>Let&amp;rsquo;s build a comprehensive audit battery. Each check targets a different failure mode:&lt;/p>
&lt;h4 id="check-1-null-detection-on-critical-fields">Check 1: Null Detection on Critical Fields&lt;/h4>
&lt;p>Nulls in currency or amount fields are deal-breakers. A null currency means we can&amp;rsquo;t convert the amount. A null amount means we can&amp;rsquo;t aggregate Total Portfolio numbers. Either one poisons everything downstream.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;NULL_CHECK&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> null_currency,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> total_amount &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> null_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> order_id &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> null_order_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> total_amount &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> order_id &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>();
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="check-2-valid-currency-codes">Check 2: Valid Currency Codes&lt;/h4>
&lt;p>Only accepted ISO currency codes should appear. Anything else means a mapping went sideways upstream.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;CURRENCY_VALIDATION&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> invalid_currency_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> LISTAGG(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> currency_code, &lt;span style="color:#e6db74">&amp;#39;, &amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> invalid_values,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="check-3-no-negative-amounts">Check 3: No Negative Amounts&lt;/h4>
&lt;p>A negative &lt;code>total_amount&lt;/code> on an order that isn&amp;rsquo;t a refund is a sign of a calculation bug. This is exactly the kind of error that inflated our Total Portfolio numbers that Tuesday morning.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;NEGATIVE_AMOUNT_CHECK&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> negative_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MIN&lt;/span>(total_amount) &lt;span style="color:#66d9ef">AS&lt;/span> worst_amount,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="check-4-duplicate-detection">Check 4: Duplicate Detection&lt;/h4>
&lt;p>Duplicate order IDs mean the source system sent the same batch twice, or our pipeline ran a retry without proper idempotency.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;DUPLICATE_CHECK&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> order_id) &lt;span style="color:#66d9ef">AS&lt;/span> duplicate_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> order_id) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>();
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="check-5-volume-anomaly-detection">Check 5: Volume Anomaly Detection&lt;/h4>
&lt;p>If today&amp;rsquo;s batch is wildly larger or smaller than the historical daily average, something structural has changed. Maybe a filter broke, maybe a new source came online, maybe someone&amp;rsquo;s test data leaked into production.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WITH&lt;/span> daily_volumes &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> daily_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>today_volume &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> today_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;VOLUME_ANOMALY&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.today_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ROUND(&lt;span style="color:#66d9ef">AVG&lt;/span>(d.daily_count), &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> avg_daily_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> t.today_count &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#66d9ef">AVG&lt;/span>(d.daily_count) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> t.today_count &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">AVG&lt;/span>(d.daily_count) &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>.&lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_result
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> daily_volumes d
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CROSS&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> today_volume t
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> t.today_count;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="aggregating-the-verdict">Aggregating the Verdict&lt;/h4>
&lt;p>In a real pipeline, you&amp;rsquo;d wrap these checks in a stored procedure or Snowflake Task. Here&amp;rsquo;s the aggregation logic that gives you a single pass/fail decision:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Aggregate all quality checks into a single verdict
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">TEMPORARY&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> audit_results &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WITH&lt;/span> checks &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Null check
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#e6db74">&amp;#39;NULL_CHECK&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> check_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> total_amount &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">result&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Negative amounts
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#e6db74">&amp;#39;NEGATIVE_AMOUNTS&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>() &lt;span style="color:#66d9ef">AND&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Invalid currency
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#e6db74">&amp;#39;INVALID_CURRENCY&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> (currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Duplicates
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#e6db74">&amp;#39;DUPLICATES&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> order_id) &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> checks;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- See the full verdict
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> audit_results;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Check if we&amp;#39;re clear to publish
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> total_checks,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">result&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PASS&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> passed,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SUM&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">result&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> failed,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">SUM&lt;/span>(&lt;span style="color:#66d9ef">CASE&lt;/span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#66d9ef">result&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;FAIL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">END&lt;/span>) &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CLEAR TO PUBLISH&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;BLOCKED — FIX REQUIRED&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> verdict
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> audit_results;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>With our deliberately corrupted data, you&amp;rsquo;ll see multiple failures here. Null currencies, invalid codes, negative amounts — exactly the problems we planted. The audit caught everything. Production is still clean.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="step-3-publish--the-happy-path-when-audits-pass">Step 3: PUBLISH — The Happy Path (When Audits Pass)&lt;/h3>
&lt;br>
&lt;p>Let&amp;rsquo;s first walk through what happens when everything is clean. This is the path your pipeline will take 95% of the time — and it&amp;rsquo;s where the Iceberg zero-copy advantage really shines.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- PUBLISH: Swap the validated clone into production
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This is a METADATA OPERATION — no Parquet files are copied or moved
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The swap is ATOMIC — downstream sees old data or new data, never partial
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> ICEBERG &lt;span style="color:#66d9ef">TABLE&lt;/span> fct_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> SWAP &lt;span style="color:#66d9ef">WITH&lt;/span> fct_orders_staging;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Clean up — the old production reference is now the staging table name
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">DROP&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> fct_orders_staging;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>That&amp;rsquo;s it. Two statements. The &lt;code>SWAP&lt;/code> updated which metadata pointer the table name &lt;code>fct_orders&lt;/code> resolves to. The Parquet files sitting in your S3 bucket didn&amp;rsquo;t budge. No new files were written. No compute was burned copying data. No warehouse credits were consumed on I/O.&lt;/p>
&lt;p>Think about what this means for pipeline design. In a traditional WAP implementation with standard Snowflake tables, the publish step is the expensive one — it&amp;rsquo;s the &lt;code>INSERT INTO ... SELECT *&lt;/code> or the &lt;code>CREATE TABLE AS SELECT&lt;/code> that physically writes data. That cost scales linearly with data volume. Double the table size, double the publish time and cost.&lt;/p>
&lt;p>With Iceberg&amp;rsquo;s zero-copy swap, publish cost is &lt;strong>constant regardless of table size&lt;/strong>. A 10 MB table and a 10 TB table swap in the same amount of time. That&amp;rsquo;s not a rounding error. That&amp;rsquo;s a fundamental shift in the economics of data quality enforcement.&lt;/p>
&lt;p>For context, I timed this on a table with ~200 million rows across several hundred Parquet files. The clone took about a second. The audit checks took about 40 seconds. The swap? Sub-second. The most expensive part of the entire WAP pipeline was the actual data quality logic — which is exactly where you &lt;em>want&lt;/em> the time to be spent.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="step-3-alternate-quarantine--when-audits-fail">Step 3 (Alternate): QUARANTINE — When Audits Fail&lt;/h3>
&lt;br>
&lt;p>Now the realistic scenario. Our audit found problems. We have options.&lt;/p>
&lt;p>&lt;strong>Option A: Drop everything, investigate, retry.&lt;/strong> This is the nuclear option. Production stays on yesterday&amp;rsquo;s data until the source is fixed and a clean batch arrives.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Nuclear option — reject the entire batch
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">DROP&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> fct_orders_staging;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Production is exactly as it was. Nothing changed.
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Option B: Quarantine the bad records, publish the clean ones.&lt;/strong> This is more practical for most teams. You log the failures for investigation, surgically remove the bad data, re-audit, and publish what&amp;rsquo;s clean.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- First, capture the failures for investigation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This gives your team evidence to fix the source system
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> quarantine_log &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">*&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CURRENT_TIMESTAMP&lt;/span>() &lt;span style="color:#66d9ef">AS&lt;/span> quarantined_at,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;NULL_CURRENCY&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;INVALID_CURRENCY&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;NEGATIVE_AMOUNT&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> failure_reason
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> );
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- See what we caught
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> failure_reason, &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">AS&lt;/span> record_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> quarantine_log
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> failure_reason
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> record_count &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now remove the bad records from the staging clone and re-validate:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Surgically remove the bad records
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">DELETE&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> fct_orders_staging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">OR&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> );
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Re-run the audit against the cleaned staging data
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- (Repeat the audit_results query from Step 2)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- If it passes now, publish the clean subset
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> ICEBERG &lt;span style="color:#66d9ef">TABLE&lt;/span> fct_orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> SWAP &lt;span style="color:#66d9ef">WITH&lt;/span> fct_orders_staging;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">DROP&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> fct_orders_staging;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The quarantine log is important. It&amp;rsquo;s not enough to just filter out bad data — you need to understand &lt;em>why&lt;/em> it was bad so you can fix the source. Every quarantined record is a symptom. The disease is upstream.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="putting-it-all-together-the-orchestrated-pipeline">Putting It All Together: The Orchestrated Pipeline&lt;/h3>
&lt;br>
&lt;p>In practice, you&amp;rsquo;d wrap this entire flow in a Snowflake Task or an external orchestrator. Here&amp;rsquo;s a stored procedure that encapsulates the full WAP pattern. This is something you can adapt and parameterise for any Iceberg table in your warehouse:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">PROCEDURE&lt;/span> run_wap_pipeline(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> target_table VARCHAR,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> staging_table VARCHAR
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">RETURNS&lt;/span> VARCHAR
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">LANGUAGE&lt;/span> &lt;span style="color:#66d9ef">SQL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">DECLARE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> fail_count INT;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result_msg VARCHAR;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">BEGIN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- WRITE: Clone production
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">IMMEDIATE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;CREATE OR REPLACE ICEBERG TABLE &amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">||&lt;/span> staging_table &lt;span style="color:#f92672">||&lt;/span> &lt;span style="color:#e6db74">&amp;#39; CLONE &amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> target_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- (Your data loading logic goes here)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#75715e">-- INSERT INTO staging_table SELECT ... FROM source
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- AUDIT: Run quality checks
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">INTO&lt;/span> fail_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Null check
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">WHERE&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> IDENTIFIER(:staging_table)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> (currency_code &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> total_amount &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Negative amount check
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">WHERE&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> IDENTIFIER(:staging_table)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> total_amount &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Invalid currency check
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">WHERE&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> IDENTIFIER(:staging_table)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> currency_code &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#e6db74">&amp;#39;USD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;EUR&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;GBP&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;AUD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;CAD&amp;#39;&lt;/span>,&lt;span style="color:#e6db74">&amp;#39;JPY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> );
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">IF&lt;/span> (fail_count &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>) &lt;span style="color:#66d9ef">THEN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- PUBLISH: Swap (zero-copy, instant)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">IMMEDIATE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;ALTER ICEBERG TABLE &amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">||&lt;/span> target_table &lt;span style="color:#f92672">||&lt;/span> &lt;span style="color:#e6db74">&amp;#39; SWAP WITH &amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> staging_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">IMMEDIATE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;DROP TABLE IF EXISTS &amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> staging_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result_msg :&lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;PUBLISHED — All checks passed. Swap completed.&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- ROLLBACK: Drop the clone
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">IMMEDIATE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;DROP TABLE IF EXISTS &amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> staging_table;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> result_msg :&lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;BLOCKED — &amp;#39;&lt;/span> &lt;span style="color:#f92672">||&lt;/span> fail_count
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">||&lt;/span> &lt;span style="color:#e6db74">&amp;#39; check(s) failed. Production unchanged.&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">RETURN&lt;/span> result_msg;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">END&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Run it
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CALL&lt;/span> run_wap_pipeline(&lt;span style="color:#e6db74">&amp;#39;fct_orders&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;fct_orders_staging&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="a-note-on-iceberg-branching--the-future-path">A Note on Iceberg Branching — The Future Path&lt;/h3>
&lt;br>
&lt;p>If you&amp;rsquo;ve been following the Iceberg ecosystem, you might be wondering about &lt;strong>branching&lt;/strong>. Apache Iceberg&amp;rsquo;s spec includes a branching feature that&amp;rsquo;s purpose-built for WAP — you create a named branch within the same table, write to it in isolation, and fast-forward the main branch when audits pass. It&amp;rsquo;s like a Git pull request for your data.&lt;/p>
&lt;p>The feature exists and it works beautifully in Spark and Flink. You can create branches with &lt;code>ALTER TABLE ... CREATE BRANCH&lt;/code>, write to them via the &lt;code>spark.wap.branch&lt;/code> session config, and publish with &lt;code>CALL system.fast_forward()&lt;/code>. The semantics are cleaner than the clone-and-swap approach because the branch &lt;em>is&lt;/em> the table — just a different lineage of snapshots within the same metadata.&lt;/p>
&lt;p>As of writing this it appears &lt;strong>Snowflake does not natively support Iceberg branching through its SQL engine today.&lt;/strong> The branching and tagging metadata operations are Iceberg spec features implemented in the Java library, with engine integrations for Spark and Flink. Snowflake&amp;rsquo;s Iceberg support gives you full DML, Time Travel, and schema evolution — but not yet the &lt;code>CREATE BRANCH&lt;/code> or &lt;code>fast_forward&lt;/code> operations.&lt;/p>
&lt;p>For teams already running Spark workloads alongside Snowflake, the branching approach is worth exploring now. You run the WAP pipeline in Spark with native Iceberg branching, and Snowflake reads the published data as an externally managed Iceberg table. It&amp;rsquo;s a composable architecture where you push quality enforcement to cheaper compute and keep the warehouse for analytics.&lt;/p>
&lt;p>For teams that are Snowflake-native, the clone-and-swap approach we&amp;rsquo;ve built in this tutorial gives you the vast majority of the benefits today. When Snowflake adds native branching support — and given the competitive landscape with Databricks, I&amp;rsquo;d expect it — we can revisit this with a cleaner implementation. The quality checks and audit battery you build now will carry forward regardless of the underlying publish mechanism.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="best-practices-thatll-save-you-pain">Best Practices That&amp;rsquo;ll Save You Pain&lt;/h3>
&lt;br>
&lt;p>After running WAP patterns in various forms across multiple teams, these are the lessons that kept surfacing.&lt;/p>
&lt;p>&lt;strong>Start with hard gates, evolve to soft gates.&lt;/strong> Initially, any check failure should block publication. Full stop. As your quality checks mature and you understand your data better, you&amp;rsquo;ll learn which checks are hard blockers — null primary keys, negative amounts, schema violations — and which are warnings worth logging but not blocking on — volume slightly above the historical average, a handful of records with an unusual but valid status. Build your audit results as structured data with severity levels so the publish decision can be nuanced.&lt;/p>
&lt;p>&lt;strong>Log quarantined records, always.&lt;/strong> Even when you drop a staging clone, capture the failing records somewhere for investigation. A quarantine table or a dead-letter S3 prefix gives your team the evidence they need to fix the source system — which is the real solution, not just filtering out bad data forever. Every quarantined record should answer three questions: what failed, which check caught it, and when it arrived.&lt;/p>
&lt;p>&lt;strong>Make your quality checks idempotent.&lt;/strong> The audit battery should be safe to run multiple times without side effects. No state mutations, no &amp;ldquo;mark as audited&amp;rdquo; flags that prevent re-runs. If the pipeline fails halfway through auditing, you should be able to restart from the top of the audit phase without creating a new clone.&lt;/p>
&lt;p>&lt;strong>Monitor your audit pass rates over time.&lt;/strong> A pipeline that fails WAP checks once a month is healthy — the pattern is catching occasional source issues. A pipeline that fails WAP checks every other day is telling you something about your upstream data source that needs fixing. Track the pass/fail ratio, the specific checks that fail most often, and the volume of quarantined records. This data is ammunition for the conversation with the source system owners.&lt;/p>
&lt;p>&lt;strong>Don&amp;rsquo;t forget the cleanup.&lt;/strong> Snowflake&amp;rsquo;s &lt;code>CLONE&lt;/code> creates metadata that references the same underlying Parquet files. If you create clones and forget to drop them, you&amp;rsquo;re not duplicating storage (the Parquet files are shared), but you are accumulating metadata that Snowflake tracks. Expired clones should be dropped promptly. Set up a cleanup task that sweeps for orphaned staging tables older than your retention threshold.&lt;/p>
&lt;p>&lt;strong>Tag your successful publishes.&lt;/strong> Not Iceberg tags (Snowflake doesn&amp;rsquo;t support those natively yet), but a metadata log. Keep a table that records every successful WAP publish — the timestamp, the table name, the row count of the new batch, and the audit results. When something goes wrong downstream three weeks from now and someone asks &amp;ldquo;what changed?&amp;rdquo;, that log is your first stop.&lt;/p>
&lt;hr>
&lt;br>
&lt;br>
&lt;h3 id="the-bigger-picture-data-quality-as-a-deploy-gate">The Bigger Picture: Data Quality as a Deploy Gate&lt;/h3>
&lt;br>
&lt;p>Zach Wilson has a line that sticks with me: &amp;ldquo;Data engineering shouldn&amp;rsquo;t have any less engineering rigor than software engineering.&amp;rdquo; He&amp;rsquo;s right. We&amp;rsquo;ve spent years building CI/CD pipelines for application code — linting, unit tests, integration tests, staging environments, approval gates — and then we load data straight into production with a prayer and a Teams message that says &amp;ldquo;run looks good.&amp;rdquo;&lt;/p>
&lt;p>WAP is the data equivalent of a deployment gate. You wouldn&amp;rsquo;t merge a pull request that fails tests. You shouldn&amp;rsquo;t publish data that fails quality checks. The mechanics are different — we&amp;rsquo;re swapping table metadata instead of deploying container images — but the principle is identical: validate before you publish, and make the publish step as cheap and reversible as possible.&lt;/p>
&lt;p>Iceberg tables make the &amp;ldquo;as cheap as possible&amp;rdquo; part genuinely achievable. When publish is a zero-copy metadata operation that completes in under a second regardless of table size, the overhead of WAP effectively disappears. The only cost is the compute for your quality checks — and that&amp;rsquo;s compute you &lt;em>want&lt;/em> to be spending.&lt;/p>
&lt;p>That Tuesday afternoon with the phantom Total Portfolio growth? It cost us three days of investigation, two days of reprocessing historical data, and an unknowable amount of credibility with the Department Head and her team. Implementing WAP on that pipeline would have cost us nothing but a few lines of SQL and a staging clone that gets created and destroyed automatically every morning.&lt;/p>
&lt;p>The pattern works. The tooling is ready. Start where you are, build the quality gates, and let the zero-copy publish mechanics handle the rest.&lt;/p>
&lt;p>Your production data — and your Department Head — will thank you for it.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Apache Iceberg</category><category>Snowflake</category><category>Write-Audit-Publish</category><category>Data Quality</category><category>SQL</category><category>Lakehouse</category><category>Data Pipelines</category><category>Best Practices</category></item><item><title>When AI Stops Treating Us Like People: The Twin Crises Nobody Wants to Discuss</title><link>https://ghostinthedata.info/posts/2025/2025-12-12-ai-bubble/</link><pubDate>Fri, 12 Dec 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-12-12-ai-bubble/</guid><author>Chris Hillman</author><description>AI companies have burned $560 billion chasing $35 billion in revenue while public trust collapses. The intersection of financial instability and human cost could trigger a correction that reshapes the industry.</description><content:encoded>&lt;h3 id="the-moment-it-stops-being-theoretical">The moment it stops being theoretical&lt;/h3>
&lt;hr>
&lt;p>A friend was telling me about their recent job interviews. Not the questions they struggled with or the technical challenge they bombed—that&amp;rsquo;s normal interview anxiety. No, they were frustrated because they&amp;rsquo;d spent forty-five minutes talking to an AI chatbot that analysed their &amp;ldquo;enthusiasm for remote work&amp;rdquo; and tracked their eye movements while asking whether they&amp;rsquo;d &amp;ldquo;misrepresent themselves 3.8x more than average candidates.&amp;rdquo;&lt;/p>
&lt;p>The speed with which I would exit an AI interview would rival even the most seasoned teenager&amp;rsquo;s alt-tab skills.&lt;/p>
&lt;p>But here&amp;rsquo;s the thing—this isn&amp;rsquo;t just another complaint about awkward technology or corporate efficiency theatre. This moment, repeated thousands of times daily across hiring processes, represents something more fundamental breaking down. AI was supposed to eliminate the boring parts of work, the repetitive drudgery that didn&amp;rsquo;t require human judgment. Instead, it&amp;rsquo;s eliminating the human parts of work—the connection, the authenticity, the actual assessment of whether someone can do the job.&lt;/p>
&lt;p>And that pattern—AI deployed to dehumanize rather than enhance, to reduce costs rather than create value—isn&amp;rsquo;t limited to hiring. It&amp;rsquo;s the story of a $560 billion investment that&amp;rsquo;s generated just $35 billion in revenue. It&amp;rsquo;s the story of public trust collapsing from cautious optimism to outright scepticism. It&amp;rsquo;s the story of two interconnected crises that could reshape the entire technology industry if either one fails.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-in-the-dystopian-hellscape-is-going-on">What in the dystopian hellscape is going on&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s stay with that hiring example for a moment, because it crystallizes the problem. Companies now use AI to screen resumes, conduct first-round interviews, analyse voice patterns for &amp;ldquo;culture fit,&amp;rdquo; track eye movements for &amp;ldquo;engagement signals,&amp;rdquo; and evaluate &amp;ldquo;authenticity&amp;rdquo; through speech pattern analysis. The justification? Candidates are supposedly committing fraud &amp;ldquo;3.8x higher rates&amp;rdquo;—though higher than what, when, or where remains conveniently unspecified.&lt;/p>
&lt;p>So AI is good when companies use it to dehumanize candidates, but it&amp;rsquo;s bad when candidates use it to stay competitive in that same dehumanizing process. Got it.&lt;/p>
&lt;p>The fundamental issue, is that &amp;ldquo;AI was supposed to do the parts of the jobs humans found boring or repetitive.&amp;rdquo; Taking meeting notes. Generating documentation. Processing expense reports. The stuff that makes you think &amp;ldquo;surely a computer could do this.&amp;rdquo; Instead, we&amp;rsquo;re using it for the parts that desperately need human judgment—assessing whether someone will thrive in a role, whether they&amp;rsquo;ll mesh with a team, whether their particular combination of skills and experience makes them the right fit for this specific challenge.&lt;/p>
&lt;p>If I&amp;rsquo;m hiring a human for a role, I don&amp;rsquo;t care about grading their every word or analysing their eye movement patterns. Doing so is fundamentally counterproductive since it encourages candidates to be as boring and uninteresting as possible while maintaining perfect eye contact and regurgitating perfect phrases, lest they don&amp;rsquo;t get the job.&lt;/p>
&lt;p>Honestly? I don&amp;rsquo;t think I want to hang out with the people who AI likes anyway.&lt;/p>
&lt;p>But this pattern—deploying AI in ways that optimise for the wrong things—appears everywhere once you start looking. McDonald&amp;rsquo;s AI drive-thru adding 260 Chicken McNuggets to orders. Klarna claiming AI &amp;ldquo;did the work of 700 agents&amp;rdquo; before quietly reversing course when quality tanked. Google&amp;rsquo;s AI Overview suggesting you add glue to pizza. These aren&amp;rsquo;t bugs. They&amp;rsquo;re symptoms of a systematic misapplication of technology by companies that haven&amp;rsquo;t figured out what problems AI actually solves.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-numbers-that-should-terrify-you">The numbers that should terrify you&lt;/h3>
&lt;hr>
&lt;p>OpenAI, the company most synonymous with the AI revolution, lost &lt;strong>$5 billion in 2024 on revenue of just $3.7 billion&lt;/strong>. Let that sink in. They&amp;rsquo;re spending $2.25 for every dollar they earn. The projected loss for 2025? $14.4 billion. Between 2023 and 2028, the company expects cumulative losses of &lt;strong>$44 billion&lt;/strong> before potentially—possibly, theoretically—reaching cash-flow positive status in 2029 or 2030.&lt;/p>
&lt;p>Despite these staggering losses, OpenAI&amp;rsquo;s valuation rocketed from $157 billion in October 2024 to somewhere between $300-500 billion today. That&amp;rsquo;s a tripling in twelve months with no profitability in sight. These aren&amp;rsquo;t valuations based on business fundamentals. They&amp;rsquo;re bets that future revenue will somehow, miraculously, justify present spending.&lt;/p>
&lt;p>Anthropic tells the same story with different numbers. Revenue jumped from $1 billion annually in 2024 to a projected $9 billion by end of 2025—impressive growth, right? Except they&amp;rsquo;re burning through &lt;strong>$2.7-3 billion annually&lt;/strong> and don&amp;rsquo;t expect to break even until 2028. The valuation inflated tenfold in eighteen months, from $18.4 billion to $183 billion. Again: not a sustainable business, but a bet on a future that may never materialise.&lt;/p>
&lt;p>The cost structure explains why profitability remains this elusive unicorn. Every ChatGPT query costs real money to compute. Every Claude conversation. Every AI-generated image. &lt;strong>Inference—running trained models—accounts for 80-90% of total AI lifetime expenses&lt;/strong>. Unlike traditional software where marginal costs approach zero as you scale, AI scales in reverse. More users mean more costs, linearly and predictably.&lt;/p>
&lt;p>Cursor, the AI coding assistant, reportedly sends 100% of its revenue to Anthropic just for model access. Perplexity AI spent &lt;strong>164% of revenue&lt;/strong> on compute providers in 2024. When your biggest customers operate at negative margins, when the companies building businesses on your platform lose money on every transaction, the entire ecosystem becomes brittle.&lt;/p>
&lt;p>The hyperscalers—Microsoft, Google, Amazon, Meta—are betting their futures on AI infrastructure with combined 2025 capital expenditures projected at &lt;strong>$325-380 billion&lt;/strong>, up 46% from 2024. These spending levels now exceed 1% of U.S. GDP quarterly. Microsoft committed $88.7 billion for fiscal 2025. Amazon projected $105-125 billion. Yet Meta&amp;rsquo;s CFO acknowledged that &amp;ldquo;genAI work is not going to be a meaningful driver of revenue this year or next year.&amp;rdquo;&lt;/p>
&lt;p>Read that again. Companies are spending hundreds of billions building infrastructure for revenue that their own financial officers admit won&amp;rsquo;t materialise for years. The gap between investment and returns has never been wider in technology history.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="when-the-guy-who-called-2008-places-a-billion-dollar-bet">When the guy who called 2008 places a billion-dollar bet&lt;/h3>
&lt;hr>
&lt;p>Michael Burry isn&amp;rsquo;t some permabear shouting about every market downturn. He&amp;rsquo;s the investor who correctly identified the 2008 subprime mortgage crisis before it happened, who placed massive bets against the housing market when everyone called him crazy, who was proven devastatingly right when the entire financial system nearly collapsed.&lt;/p>
&lt;p>In early November 2025, Burry disclosed regulatory filings showing over &lt;strong>$1 billion in put options against AI leaders&lt;/strong>: $912 million against Palantir (66% of his portfolio) and $187 million against Nvidia. His thesis is explicit and detailed, laid out in his Substack newsletter &amp;ldquo;Cassandra Unchained.&amp;rdquo;&lt;/p>
&lt;p>&amp;ldquo;I am not claiming Nvidia is Enron. It is clearly Cisco,&amp;rdquo; Burry wrote. He&amp;rsquo;s identifying Nvidia as today&amp;rsquo;s equivalent of Cisco during the dot-com bubble—the essential hardware supplier powering a massive capital investment cycle that will eventually see overbuilt supply meet far less demand than expected.&lt;/p>
&lt;p>But his most damning observation concerns accounting practices. Burry projects that tech companies are &lt;strong>understating depreciation by $176 billion between 2026-2028&lt;/strong> by extending the useful life of AI computing equipment from realistic 2-3 year product cycles to artificially long periods. This accounting treatment inflates reported earnings—Oracle by 26.9%, Meta by 20.8%—creating the illusion of profitability where losses actually exist.&lt;/p>
&lt;p>His criticism of &amp;ldquo;circular financing&amp;rdquo; cuts deeper. OpenAI owns 10% of AMD while Nvidia invested $100 billion in OpenAI. Microsoft is both a major OpenAI shareholder and one of Nvidia&amp;rsquo;s largest customers (representing roughly 20% of Nvidia revenue). CoreWeave, funded by Nvidia, serves OpenAI&amp;rsquo;s compute needs. &amp;ldquo;True end demand is ridiculously small,&amp;rdquo; Burry observed. &amp;ldquo;Almost all customers are funded by their dealers.&amp;rdquo;&lt;/p>
&lt;p>When an investor who successfully predicted the last major financial crisis places billion-dollar bets against your industry and explicitly compares your accounting practices to pre-2008 mortgage fraud, that&amp;rsquo;s not speculation. That&amp;rsquo;s pattern recognition.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-trust-collapse-nobody-wants-to-acknowledge">The trust collapse nobody wants to acknowledge&lt;/h3>
&lt;hr>
&lt;p>Here&amp;rsquo;s the part where we connect the financial instability back to that AI interview experience. Because these crises aren&amp;rsquo;t separate—they&amp;rsquo;re feeding each other in ways that threaten the entire AI business model.&lt;/p>
&lt;p>According to Pew Research, &lt;strong>only 17% of Americans believe AI will have a positive impact&lt;/strong> on the U.S. over the next 20 years. A full 51% are more concerned than excited about AI&amp;rsquo;s increased use in daily life. These figures represent dramatic worsening since 2021, when optimism still outweighed concern.&lt;/p>
&lt;p>The gap between expert opinion and public sentiment has become a chasm. While 56% of AI experts see positive impact ahead, only 17% of the general public agrees. While 73% of experts expect positive employment effects, only 23% of the public shares that view. Either the experts are completely disconnected from reality, or the public has accurately identified risks the industry refuses to acknowledge.&lt;/p>
&lt;p>The trust erosion isn&amp;rsquo;t abstract. It&amp;rsquo;s rooted in lived experience. That AI interview that made you feel reduced to &amp;ldquo;an n-dimensional matrix.&amp;rdquo; The AI-generated customer service response that completely missed your problem. The hiring algorithm that rejected your application before a human ever saw it. The chatbot that confidently gave you wrong information. The image generator that couldn&amp;rsquo;t distinguish historical accuracy from Reddit jokes.&lt;/p>
&lt;p>That &amp;ldquo;authenticity is a limited resource that is diminishing every day.&amp;rdquo; Every time someone encounters AI deployed to reduce costs rather than add value, deployed to eliminate human judgment rather than enhance it, that person becomes incrementally more sceptical. Not of technology in general—of this specific application pattern where efficiency trumps everything else.&lt;/p>
&lt;p>Consumer behaviour reflects this scepticism in ways that directly threaten revenue. A Washington State University study found that products described as &amp;ldquo;AI-powered&amp;rdquo; are consistently less popular with consumers. Gartner surveys show &lt;strong>64% of customers prefer companies NOT use AI for customer service&lt;/strong>, with 53% willing to switch to competitors that don&amp;rsquo;t. Only &lt;strong>3% of smartphone owners&lt;/strong> are willing to pay extra for AI features.&lt;/p>
&lt;p>Three percent.&lt;/p>
&lt;p>When you&amp;rsquo;re building a business model that depends on massive consumer adoption and widespread willingness to pay premium prices, 3% willingness to pay is a death sentence. It means the revenue models underpinning all those billion-dollar valuations are built on fundamentally wrong assumptions about demand.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-enterprise-adoption-crisis">The enterprise adoption crisis&lt;/h3>
&lt;hr>
&lt;p>Maybe consumer scepticism doesn&amp;rsquo;t matter, you might argue. Maybe the real money is in enterprise. Except enterprise adoption is stalling for the same reasons, just with bigger price tags attached to the disappointment.&lt;/p>
&lt;p>The MIT NANDA study found that &lt;strong>95% of enterprise AI pilots yield zero measurable business return&lt;/strong> despite $30-40 billion in GenAI investment. BCG reports 74% of companies haven&amp;rsquo;t seen tangible value from AI initiatives. McKinsey&amp;rsquo;s surveys show 80%+ of organisations haven&amp;rsquo;t realised significant bottom-line AI impact. Gartner predicts &lt;strong>30% of GenAI projects will be abandoned by end of 2025&lt;/strong> due to poor data quality, escalating costs, or unclear value.&lt;/p>
&lt;p>Monte Carlo&amp;rsquo;s 2024 survey of data professionals reveals the pressure cooker: &lt;strong>100% feel pressure&lt;/strong> from leadership to implement GenAI strategy, yet &lt;strong>90% believe their leaders do NOT have realistic expectations&lt;/strong> for what&amp;rsquo;s technically feasible or capable of driving business value. Only 12% of companies have dedicated AI teams—meaning &lt;strong>84% rely on existing data teams&lt;/strong> to figure it out.&lt;/p>
&lt;p>This is the reality on the ground for data professionals. Leadership demands AI transformation. Budgets flow to AI initiatives. But the actual work of implementation falls to teams who know the technology isn&amp;rsquo;t mature enough, the data isn&amp;rsquo;t clean enough, the use cases aren&amp;rsquo;t well-defined enough. The pressure to deploy anyway—to show &amp;ldquo;innovation,&amp;rdquo; to justify spending, to keep up with competitors—leads to implementations that fail quietly, generate minimal value, and burn credibility.&lt;/p>
&lt;p>The case studies pile up. McDonald&amp;rsquo;s ended its three-year AI drive-thru partnership with IBM in June 2024 after viral videos showed the system adding 260 Chicken McNuggets to orders and repeatedly misinterpreting requests. Klarna initially claimed AI was &amp;ldquo;doing the work of 700 agents&amp;rdquo; after cutting 22% of its workforce, then reversed course with a &amp;ldquo;major hiring initiative&amp;rdquo; in May 2025. The CEO admitted AI was actually &amp;ldquo;doing work of 700 really bad agents&amp;rdquo;—quality decline cost customers, and the rollback cost approximately $15 million with no net gain.&lt;/p>
&lt;p>IBM&amp;rsquo;s Watson Health, once touted as transformative for cancer diagnosis, never reached production use, ran over budget, and was eventually sold for approximately $1 billion after multi-billion dollar investment. The Humane Ai Pin and Rabbit R1 represent hardware category failures, with slashed prices following weak sales and critical reviews noting the products solve &amp;ldquo;problems that don&amp;rsquo;t exist.&amp;rdquo;&lt;/p>
&lt;p>Each failure compounds the trust deficit. Each over-promised and under-delivered implementation makes the next pitch harder. Each news story about AI gone wrong creates more sceptics who start questioning whether this technology delivers value or just disrupts things that were working fine.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-2008-playbook-now-in-tech">The 2008 playbook, now in tech&lt;/h3>
&lt;hr>
&lt;p>The parallels between current AI investment patterns and pre-2008 financial crisis have moved from concerning to alarming. Deutsche Bank is now using &lt;strong>Synthetic Risk Transfers (SRTs) to hedge AI infrastructure loans&lt;/strong>—structures that analysts say &amp;ldquo;strongly resemble CDOs and CDSs that amplified the 2008 financial crisis.&amp;rdquo; Tech companies shift AI spending into Special Purpose Vehicles (SPVs) to obscure true costs, echoing Enron&amp;rsquo;s infamous off-balance-sheet accounting.&lt;/p>
&lt;p>The leverage buildup mirrors pre-crisis patterns. Goldman Sachs reports hyperscalers took on &lt;strong>$121 billion in debt&lt;/strong> over the past year—a 300%+ increase from typical levels. Meta raised $29 billion in debt for a single Louisiana data centre. Oracle operates at a &lt;strong>500% debt-to-equity ratio&lt;/strong> to compete in AI data centres. Morgan Stanley estimates Big Tech will spend $3 trillion on AI infrastructure through 2028, with cash flows covering only half.&lt;/p>
&lt;p>The dot-com parallels are equally sobering. In the late 1990s, &lt;strong>80+ million miles of fiber optic cable&lt;/strong> were laid based on inflated demand projections; 85-95% remained unused (&amp;ldquo;dark fiber&amp;rdquo;) years after the bubble burst. Today&amp;rsquo;s equivalent is the massive AI data centre buildout—Meta constructing a facility covering a &amp;ldquo;significant part of Manhattan&amp;rdquo;—predicated on demand growth that may never materialise. The $500 billion Stargate Project promises unprecedented AI infrastructure investment for enterprise AI adoption that&amp;rsquo;s already stalling.&lt;/p>
&lt;p>The NASDAQ took &lt;strong>15 years to recover&lt;/strong> from its 2000 peak. Cisco shares peaked at $80 and still haven&amp;rsquo;t returned to that level twenty-five years later. Those aren&amp;rsquo;t cautionary tales from ancient history—they&amp;rsquo;re previews of what happens when infrastructure investment radically outpaces actual demand.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-the-insiders-are-saying">What the insiders are saying&lt;/h3>
&lt;hr>
&lt;p>Perhaps most telling are the warnings from industry insiders themselves. Sam Altman, OpenAI&amp;rsquo;s CEO, acknowledges that investors are &amp;ldquo;overexcited about AI&amp;rdquo; and predicts &amp;ldquo;people will overinvest and lose money during this phase.&amp;rdquo; Google CEO Sundar Pichai warns of &amp;ldquo;elements of irrationality&amp;rdquo; and says &amp;ldquo;no company is going to be immune&amp;rdquo; if the bubble bursts. Jeff Bezos called the current environment &amp;ldquo;kind of an industrial bubble.&amp;rdquo;&lt;/p>
&lt;p>When the people building and funding AI companies publicly warn about bubble dynamics, when they&amp;rsquo;re literally saying investors will lose money, that&amp;rsquo;s not bearish speculation. That&amp;rsquo;s reality acknowledgment.&lt;/p>
&lt;p>Ray Dalio&amp;rsquo;s proprietary bubble indicator sits at approximately &lt;strong>80% of historical bubble peaks&lt;/strong>, with his AI chatbot estimating a 65-75% probability of meaningful correction in AI equities by end of 2026. Morgan Stanley&amp;rsquo;s Lisa Shalett warned of a &amp;ldquo;Cisco moment&amp;rdquo; within 24 months, describing the market as a &amp;ldquo;one-note narrative&amp;rdquo; entirely dependent on AI capital expenditures.&lt;/p>
&lt;p>The timing predictions are getting specific. Motley Fool analysts suggest the bubble could burst in 2025 due to unsustainable valuations. Ruchir Sharma of Rockefeller International predicts 2026 if interest rates rise. BeyondTrust researchers argue &amp;ldquo;artificial inflation of AI has already peaked in 2024.&amp;rdquo; A November 2025 Nature analysis noted that &amp;ldquo;many financial analysts now agree there is an &amp;lsquo;AI bubble,&amp;rsquo; some speculate it could finally burst in the next few months.&amp;rdquo;&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="preparing-for-what-comes-next">Preparing for what comes next&lt;/h3>
&lt;hr>
&lt;p>Whether the bubble bursts catastrophically or deflates gradually, data professionals can take specific steps to build resilient careers. The skills that remain human-essential include critical thinking and complex decision-making, cross-contextual reasoning (connecting insights across domains), ethical judgment in novel situations, relationship building and stakeholder management, and what researchers call &amp;ldquo;AI discernment&amp;rdquo;—knowing when to trust AI versus human judgment.&lt;/p>
&lt;p>Technical preparation should emphasise both breadth and depth. Core competencies remain essential: probability, statistics, Python, SQL. But add ML fundamentals including embeddings and model monitoring. Develop cloud platform expertise. Build governance and ethics capabilities that will be increasingly valuable as regulation tightens. Document measurable business impact, not just technical achievements—when budgets tighten, demonstrated ROI becomes the difference between retention and reduction.&lt;/p>
&lt;p>For data leaders, the most important preparation may be honest communication. Setting realistic expectations with executive leadership about what AI can deliver—and on what timeline—protects both the organisation and the team. The companies that navigate the potential correction successfully will be those treating AI as a tool for specific problems rather than a magic solution for every challenge.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-happens-when-authenticity-runs-out">What happens when authenticity runs out&lt;/h3>
&lt;hr>
&lt;p>I desperately want to be surrounded by real people with real hobbies and passions. I don&amp;rsquo;t care if a candidate loves remote work, I care about their humanity. I don&amp;rsquo;t care if a candidate tripped over their words a few times, I care about their authenticity.&lt;/p>
&lt;p>This isn&amp;rsquo;t nostalgia for pre-AI times. It&amp;rsquo;s recognition that authenticity, human connection, genuine judgment—these aren&amp;rsquo;t inefficiencies to optimise away. They&amp;rsquo;re the core value proposition. They&amp;rsquo;re what makes work meaningful, what makes organisations effective, what makes products worth buying.&lt;/p>
&lt;p>When companies deploy AI to eliminate these elements—to reduce humans to vectors, to automate judgment, to replace connection with efficiency—they&amp;rsquo;re not just failing to capture value. They&amp;rsquo;re actively destroying it. The companies that will survive aren&amp;rsquo;t those with the most sophisticated AI. They&amp;rsquo;re those that figured out how to use AI to enhance human capability rather than replace it, to preserve authenticity rather than eliminate it.&lt;/p>
&lt;p>The crisis isn&amp;rsquo;t coming. It&amp;rsquo;s already here, building pressure daily in the gap between investment and returns, between promises and delivery, between what AI can actually do and what we&amp;rsquo;ve been told it will become. The question isn&amp;rsquo;t whether there will be a reckoning. The question is whether you&amp;rsquo;ll be positioned to navigate it when it arrives—and whether the companies you work for will still treat you like a person when they do.&lt;/p></content:encoded><category>AI</category><category>Data Quality &amp; Governance</category><category>AI Bubble</category><category>Financial Crisis</category><category>Trust</category><category>Data Teams</category><category>AI Ethics</category><category>Career Development</category></item><item><title>The Four Stages of Data Quality: From Hidden Costs to Measurable Value</title><link>https://ghostinthedata.info/posts/2025/2025-11-24-data-quality-framework/</link><pubDate>Mon, 24 Nov 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-11-24-data-quality-framework/</guid><author>Chris Hillman</author><description>A comprehensive framework for understanding data quality impact, principles, business case development, and implementation strategies with real ROI calculations</description><content:encoded>&lt;p>This is the fundamental problem with data quality. You know it matters. Everyone knows it matters. But until you can quantify the impact, connect it to business outcomes, and build a credible business case, it remains this abstract thing that&amp;rsquo;s important but never urgent enough to properly fund.&lt;/p>
&lt;p>I wrote a &lt;a href="https://ghostinthedata.info/posts/2025/2025-11-22-data-quality/" target="_blank" rel="noopener">practical guide to data quality&lt;/a> last week that walks through hands-on implementation—the SQL queries, the profiling techniques, the actual mechanics of finding and fixing data issues. Think of that as the &amp;ldquo;how to use the tools&amp;rdquo; guide. This article is different. This is the &amp;ldquo;why these tools matter and how to convince your organisation to actually use them&amp;rdquo; guide.&lt;/p>
&lt;p>We&amp;rsquo;re going deep on the four stages every successful data quality initiative must navigate: understanding the hidden impact, establishing principles that matter, building the business case that gets funded, and implementing in a way that sticks. By the end, you&amp;rsquo;ll have frameworks, templates, SQL scripts, and real ROI calculations you can adapt to your own organisation.&lt;/p>
&lt;p>Because honestly? The technical part of data quality isn&amp;rsquo;t the hard part. The hard part is getting everyone to care.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="stage-1-understanding-the-impact-or-why-bad-data-is-expensive-in-ways-youre-not-measuring">Stage 1: Understanding the impact (or why bad data is expensive in ways you&amp;rsquo;re not measuring)&lt;/h3>
&lt;hr>
&lt;p>A retail company spent $2 million planning a premium customer campaign. They&amp;rsquo;d identified 8,000 high-value customers who spent more than $10,000 annually. The plan was solid—15% discounts, priority service, early sale access. Expected ROI: $8 million in incremental revenue. The marketing team was excited. The CFO approved the budget. Everyone was ready to go.&lt;/p>
&lt;p>Then someone actually looked at the data quality. Email addresses? Only 60% complete—meaning they could only reach 4,800 of their 8,000 targets. Phone numbers? 45% complete. Customer segmentation? 70% accurate—meaning 1,600 people were incorrectly classified as high-value when they weren&amp;rsquo;t. Physical addresses? 75% current—meaning 2,000 would bounce back from outdated addresses.&lt;/p>
&lt;p>Do the math. They could reach 60% of their intended audience. They&amp;rsquo;d waste resources targeting 20% who shouldn&amp;rsquo;t be included. The campaign that was supposed to generate $8 million in revenue would reach maybe $3.2 million if they were lucky. The $2 million investment would get torched trying to contact people they couldn&amp;rsquo;t reach, with messages going to people who didn&amp;rsquo;t qualify.&lt;/p>
&lt;p>The marketing team made a gutsy call: delay the campaign six months while they fixed the data. The CFO was furious. The CEO was sceptical. But they did it anyway.&lt;/p>
&lt;p>Six months later, after investing $300,000 in data enrichment, validation, and cleanup, they launched. Email completeness went from 60% to 92%. Phone numbers from 45% to 85%. Segmentation accuracy from 70% to 96%. They could now reach 7,360 customers—92% of their target.&lt;/p>
&lt;p>The campaign generated $10.6 million in revenue. ROI of 430%. If they&amp;rsquo;d launched with bad data? Maybe $3.2 million, and they would&amp;rsquo;ve permanently damaged their brand by sending premium offers to people who barely bought anything while missing their actual high-value customers.&lt;/p>
&lt;p>&lt;strong>This is what bad data costs: not just money, but strategic opportunities&lt;/strong>.&lt;/p>
&lt;p>Let&amp;rsquo;s break down the hidden costs:&lt;/p>
&lt;p>&lt;strong>1. The operational tax&lt;/strong>: Every process touched by bad data runs slower, costs more, requires more manual intervention. Your accounts payable team spending hours reconciling supplier records? That&amp;rsquo;s the data quality tax. Your customer service team asking for information customers already provided? That&amp;rsquo;s the tax. Your analysts spending 60% of their time cleaning data instead of analysing it? Tax, tax, tax.&lt;/p>
&lt;p>&lt;strong>2. The decision paralysis&lt;/strong>: When leadership doesn&amp;rsquo;t trust the data, they either make decisions without it (dangerous) or spend weeks validating every number (expensive). I&amp;rsquo;ve watched executive teams waste hundreds of hours in meetings arguing about whose numbers are right instead of deciding what to do.&lt;/p>
&lt;p>&lt;strong>3. The compliance exposure&lt;/strong>: Regulatory fines aren&amp;rsquo;t theoretical. GDPR violations can hit million&amp;rsquo;s or 4% of global revenue—whichever is higher. Healthcare organisations face HIPAA violations. Financial services face penalties. All because data quality wasn&amp;rsquo;t maintained.&lt;/p>
&lt;p>&lt;strong>4. The customer experience degradation&lt;/strong>: You ever get marketing emails for products you already bought? Or addressed to the wrong name? Or sent to an email you haven&amp;rsquo;t used in five years? That&amp;rsquo;s data quality failure, and it&amp;rsquo;s costing you customers.&lt;/p>
&lt;p>&lt;strong>5. The strategic capability gap&lt;/strong>: Want to use AI? Build predictive models? Enable real-time decision-making? All of that requires high-quality data. Bad data doesn&amp;rsquo;t just slow you down—it makes entire categories of strategic initiatives impossible.&lt;/p>
&lt;p>Here&amp;rsquo;s the framework I use to assess data quality impact. It&amp;rsquo;s proven effective across dozens of organisations:&lt;/p>
&lt;p>&lt;strong>The Data Quality Impact Assessment Matrix&lt;/strong>:&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Impact Area&lt;/th>
 &lt;th>Discovery Questions&lt;/th>
 &lt;th>Measurement Approach&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>&lt;strong>Financial&lt;/strong>&lt;/td>
 &lt;td>Where do we process payments? Where could duplicates occur? What reconciliation issues exist?&lt;/td>
 &lt;td>Audit duplicate payments, measure reconciliation time, calculate lost revenue from data errors&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Operational&lt;/strong>&lt;/td>
 &lt;td>Which processes require manual intervention? Where do we see repeated errors? What takes longer than it should?&lt;/td>
 &lt;td>Time studies, error rates, rework costs, efficiency metrics&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Customer&lt;/strong>&lt;/td>
 &lt;td>What complaints relate to data? Where do communications fail? What&amp;rsquo;s our data-related churn rate?&lt;/td>
 &lt;td>Customer satisfaction scores, complaint analysis, churn attribution&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Compliance&lt;/strong>&lt;/td>
 &lt;td>What regulations require data accuracy? Where are we vulnerable? What&amp;rsquo;s our audit history?&lt;/td>
 &lt;td>Risk assessments, audit findings, near-miss incidents, potential penalties&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>&lt;strong>Strategic&lt;/strong>&lt;/td>
 &lt;td>What initiatives depend on data quality? What capabilities are blocked? What opportunities are we missing?&lt;/td>
 &lt;td>Delayed project costs, opportunity costs, competitive disadvantage metrics&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>The key insight: you can&amp;rsquo;t fix what you can&amp;rsquo;t measure, and you can&amp;rsquo;t measure what you can&amp;rsquo;t see. Most organisations have no systematic way to identify data quality costs. They see symptoms—inefficiency, errors, customer complaints—but they don&amp;rsquo;t connect those symptoms back to root data quality causes.&lt;/p>
&lt;p>Start by picking one area.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="stage-2-the-principles-that-actually-matter-not-the-theoretical-purity-stuff">Stage 2: The principles that actually matter (not the theoretical purity stuff)&lt;/h3>
&lt;hr>
&lt;p>Perfect data doesn&amp;rsquo;t exist. If you&amp;rsquo;re waiting for perfect data before you take action, you&amp;rsquo;ll wait forever. The goal isn&amp;rsquo;t perfection—it&amp;rsquo;s &amp;ldquo;good enough for purpose.&amp;rdquo;&lt;/p>
&lt;p>Here&amp;rsquo;s what I mean. You&amp;rsquo;ve got a customer database with 100,000 records. Email addresses are 92% complete. Is that good or bad?&lt;/p>
&lt;p>It depends. If you&amp;rsquo;re running an email marketing campaign, 92% is pretty good—you can reach most of your audience, and the missing 8% probably includes inactive customers anyway. But if you&amp;rsquo;re using email as your primary customer identifier for account recovery and password resets, 92% means 8,000 customers can&amp;rsquo;t access their accounts. That&amp;rsquo;s a disaster.&lt;/p>
&lt;p>&lt;strong>Context determines quality requirements&lt;/strong>. This is the first principle, and it&amp;rsquo;s the one most people miss.&lt;/p>
&lt;p>The six dimensions of data quality give us a framework for thinking about this:&lt;/p>
&lt;p>&lt;strong>Completeness&lt;/strong>: The proportion of data populated versus what should be there. Sounds simple, but it&amp;rsquo;s deceptive. A product catalog might show 74% completeness overall, but when you break it down, engine parts (which absolutely need batch numbers and manufacturing dates for recalls) might be 40% complete, while non-critical parts (which don&amp;rsquo;t need those fields) are 98% complete. The overall number hides the real problem.&lt;/p>
&lt;p>I&amp;rsquo;ve seen organisations mandate 95% completeness across all fields, leading to users entering &amp;ldquo;N/A&amp;rdquo; or &amp;ldquo;Unknown&amp;rdquo; or &amp;ldquo;TBD&amp;rdquo; just to hit the metric. Congratulations, you&amp;rsquo;ve now got complete data that&amp;rsquo;s completely useless. Better question: which fields are actually required for which purposes?&lt;/p>
&lt;p>&lt;strong>Uniqueness&lt;/strong>: No duplicate records based on how you identify things. But here&amp;rsquo;s where it gets messy—what&amp;rsquo;s a duplicate?&lt;/p>
&lt;p>Say you&amp;rsquo;ve got supplier records. Is &amp;ldquo;Apple Services Ltd&amp;rdquo; the same as &amp;ldquo;Apple Service Limited&amp;rdquo;? What about two companies at the same address in a large office building? What about franchise operations that are legally separate entities but operationally related?&lt;/p>
&lt;p>The right approach isn&amp;rsquo;t picking one rule and enforcing it everywhere. It&amp;rsquo;s defining matching criteria based on business context, documenting allowed exceptions (like IT admin accounts that legitimately need to be separate from regular user accounts), and flagging everything else for review.&lt;/p>
&lt;p>&lt;strong>Timeliness&lt;/strong>: Data represents reality at the required point in time. This one&amp;rsquo;s subtle because it&amp;rsquo;s not just about freshness—it&amp;rsquo;s about currency relative to need.&lt;/p>
&lt;p>Customer credit checks need to be under three months old for sales orders because creditworthiness changes rapidly. But employee department assignments only need updating when people actually move departments, not daily. Cost centre ownership must update immediately when someone leaves the company—having cost centres owned by former employees is both a control issue and a timeliness failure.&lt;/p>
&lt;p>The pharmaceutical industry gets this. Medical practitioner licenses must be verified within the last 12 months. It&amp;rsquo;s not about the data being &amp;ldquo;old&amp;rdquo;—it&amp;rsquo;s about regulatory compliance and patient safety depending on current verification.&lt;/p>
&lt;p>&lt;strong>Validity&lt;/strong>: Data conforms to expected format, type, and range. Email addresses have an @ symbol and a domain. UK postcodes follow the pattern XX1(1) 1XX. US social security numbers are XXX-XX-XXXX.&lt;/p>
&lt;p>But here&amp;rsquo;s the critical distinction people miss: valid doesn&amp;rsquo;t mean accurate. &amp;ldquo;&lt;a href="mailto:chris@hillman.com">chris@hillman.com&lt;/a>&amp;rdquo; is a valid email format, but that doesn&amp;rsquo;t mean Chris Hillman actually owns that email or that it&amp;rsquo;s deliverable. Validity is a technical check—format compliance. Accuracy is a truth check—does it match reality?&lt;/p>
&lt;p>You need both, but they&amp;rsquo;re different gates. Validity catches obvious garbage at entry. Accuracy requires verification against authoritative sources.&lt;/p>
&lt;p>&lt;strong>Accuracy&lt;/strong>: Data correctly describes the real-world object or event. This is the gold standard, but it&amp;rsquo;s also the hardest to achieve because it requires comparison to truth.&lt;/p>
&lt;p>How do you verify an email address is accurate? You can check the domain exists in the domain registry (that&amp;rsquo;s accuracy of the domain portion). But verifying the username? That requires sending a verification email and getting a response. Expensive, slow, but sometimes necessary.&lt;/p>
&lt;p>For supplier data, you might compare against Dun &amp;amp; Bradstreet or other authoritative databases. For physical assets, you might need actual physical verification—someone walks to the location and confirms the equipment exists and is correctly assigned. Yes, this is expensive. Sometimes physical verification is the only way.&lt;/p>
&lt;p>&lt;strong>Consistency&lt;/strong>: The absence of difference when comparing two representations of the same thing. This matters when data is replicated across multiple systems.&lt;/p>
&lt;p>Employee data typically originates in HR systems and replicates to Azure AD, Office 365, ERP systems, CRM systems. Names, email addresses, manager relationships, department codes—all of this should match everywhere. When it doesn&amp;rsquo;t, you get access control failures, incorrect reporting hierarchies, and security vulnerabilities.&lt;/p>
&lt;p>The principle here: define one authoritative source (usually the system where data originates), and measure consistency relative to that source. Don&amp;rsquo;t try to achieve consistency between ten systems simultaneously—that way lies madness.&lt;/p>
&lt;p>Now, here&amp;rsquo;s the meta-principle that governs all of these: &lt;strong>focus on data quality improvements that deliver tangible business value&lt;/strong>. It&amp;rsquo;s the filter that prevents data quality initiatives from becoming academic exercises.&lt;/p>
&lt;p>Every quality improvement should answer three questions:&lt;/p>
&lt;ol>
&lt;li>What business process does this enable or improve?&lt;/li>
&lt;li>What&amp;rsquo;s the measurable impact of the improvement?&lt;/li>
&lt;li>What&amp;rsquo;s the cost to achieve and maintain the improvement?&lt;/li>
&lt;/ol>
&lt;p>If you can&amp;rsquo;t answer these questions, you&amp;rsquo;re probably chasing perfection rather than solving problems.&lt;/p>
&lt;p>I once worked with a team that wanted to achieve 99.9% completeness on every field in their customer master. Noble goal, terrible strategy. Some fields genuinely didn&amp;rsquo;t apply to some customers. Others were nice-to-have but not required for any actual business process. They would&amp;rsquo;ve spent millions chasing that last 0.9% with zero business benefit.&lt;/p>
&lt;p>Instead, we mapped every field to actual usage. Turns out, 12 fields were critical for 95% of business processes. We focused on getting those 12 fields to 98% completeness and accuracy for active customers. Cost: $150,000. Benefit: eliminated 80% of customer service escalations related to incomplete information. Payback: four months.&lt;/p>
&lt;p>That&amp;rsquo;s the principle: pragmatic quality focused on business value. Not theoretical purity.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="stage-3-building-the-business-case-that-actually-gets-funded">Stage 3: Building the business case that actually gets funded&lt;/h3>
&lt;hr>
&lt;p>You know what kills most data quality initiatives? Not the technical complexity. Not the organisational resistance. Not even the lack of executive understanding.&lt;/p>
&lt;p>It&amp;rsquo;s the business case.&lt;/p>
&lt;p>More specifically, it&amp;rsquo;s the absence of a credible business case that connects data quality investment to measurable business outcomes in language executives understand and trust.&lt;/p>
&lt;p>I&amp;rsquo;ve seen dozens of failed business cases. They usually look something like this:&lt;/p>
&lt;p>&lt;em>&amp;ldquo;We need to improve data quality because our data is inaccurate and incomplete. This causes problems for users. We need $500,000 to implement a data quality tool and hire two FTEs. Benefits include better decision-making, improved customer satisfaction, and reduced risk.&amp;rdquo;&lt;/em>&lt;/p>
&lt;p>Everything in that paragraph is true. And it&amp;rsquo;s completely unconvincing.&lt;/p>
&lt;p>Why? Because it&amp;rsquo;s vague, unquantified, and doesn&amp;rsquo;t connect to anything. &amp;ldquo;Better decision-making&amp;rdquo; could mean anything. &amp;ldquo;Improved customer satisfaction&amp;rdquo; by how much? &amp;ldquo;Reduced risk&amp;rdquo; from what level to what level?&lt;/p>
&lt;p>Here&amp;rsquo;s what a winning business case looks like. This is the framework:&lt;/p>
&lt;p>&lt;strong>The Four-Part Business Case Structure:&lt;/strong>&lt;/p>
&lt;p>&lt;strong>Part 1: Quantified Current State Impact&lt;/strong>&lt;/p>
&lt;p>Don&amp;rsquo;t start with what you want to do. Start with what&amp;rsquo;s currently happening and what it costs. This is where that impact assessment from Stage 1 pays off.&lt;/p>
&lt;p>Example (real numbers from an actual engagement):&lt;/p>
&lt;p>&lt;em>Current State Analysis - Customer Master Data&lt;/em>&lt;/p>
&lt;ul>
&lt;li>Total customer records: 145,000&lt;/li>
&lt;li>Duplicate records identified: 12,500 (8.6%)&lt;/li>
&lt;li>Incomplete critical fields: 18% of records missing email, phone, or address&lt;/li>
&lt;li>Inconsistent data across systems: 23% of records show discrepancies between CRM and ERP&lt;/li>
&lt;/ul>
&lt;p>&lt;em>Measured Business Impact - Annual Costs&lt;/em>&lt;/p>
&lt;ul>
&lt;li>Failed customer deliveries: 1,850 incidents × $75 average cost = $138,750&lt;/li>
&lt;li>Duplicate marketing contacts: 12,500 duplicates × $2.50 per contact = $31,250&lt;/li>
&lt;li>Customer service call escalations: 4,200 calls × $12 average handle time cost = $50,400&lt;/li>
&lt;li>Sales opportunity delays: 340 deals delayed × $425 average delay cost = $144,500&lt;/li>
&lt;li>Regulatory risk exposure: 3 GDPR near-miss incidents, potential fines $100K-$500K&lt;/li>
&lt;li>Analyst time spent on data cleanup: 2 FTE × $120K = $240,000&lt;/li>
&lt;li>&lt;strong>Total Quantified Annual Impact: $604,900&lt;/strong>&lt;/li>
&lt;/ul>
&lt;p>Notice what&amp;rsquo;s happening here. Every number is specific. Every cost is calculated. There&amp;rsquo;s no hand-waving about &amp;ldquo;improved quality&amp;rdquo; or &amp;ldquo;better decisions.&amp;rdquo; We&amp;rsquo;re talking about actual incidents, actual costs, actual regulatory exposure.&lt;/p>
&lt;p>&lt;strong>Part 2: Proposed Solution with Tiered Options&lt;/strong>&lt;/p>
&lt;p>Don&amp;rsquo;t present one option. Present three: minimal, recommended, and comprehensive. This gives executives decision-making power and shows you&amp;rsquo;ve thought about trade-offs.&lt;/p>
&lt;p>&lt;em>Option 1 - Tactical Cleanup (Low Investment, Temporary Relief)&lt;/em>&lt;/p>
&lt;ul>
&lt;li>Scope: One-time cleanup of customer master, basic duplicate prevention&lt;/li>
&lt;li>Investment: $85,000&lt;/li>
&lt;li>Timeline: 3 months&lt;/li>
&lt;li>Expected Impact: Reduce duplicates by 70%, improve completeness to 85%&lt;/li>
&lt;li>Annual Benefit: ~$250,000 (41% of total impact)&lt;/li>
&lt;li>Limitations: No ongoing monitoring, issues will return within 12-18 months&lt;/li>
&lt;/ul>
&lt;p>&lt;em>Option 2 - Sustainable Foundation (Recommended)&lt;/em>&lt;/p>
&lt;ul>
&lt;li>Scope: Cleanup + monitoring framework + governance + preventive controls&lt;/li>
&lt;li>Investment: $320,000 (Year 1), $120,000 annual ongoing&lt;/li>
&lt;li>Timeline: 6 months implementation, ongoing operation&lt;/li>
&lt;li>Expected Impact: Reduce duplicates by 95%, improve completeness to 96%, establish consistency monitoring&lt;/li>
&lt;li>Annual Benefit: ~$520,000 (86% of total impact)&lt;/li>
&lt;li>ROI: Break-even in 7 months, $200K annual net benefit&lt;/li>
&lt;li>Limitations: Requires business process changes, 0.5 FTE ongoing stewardship&lt;/li>
&lt;/ul>
&lt;p>&lt;em>Option 3 - Enterprise Data Quality Platform&lt;/em>&lt;/p>
&lt;ul>
&lt;li>Scope: Full DQ suite, automated monitoring, ML-based matching, real-time validation&lt;/li>
&lt;li>Investment: $1.2M (Year 1), $380,000 annual ongoing&lt;/li>
&lt;li>Timeline: 12 months implementation&lt;/li>
&lt;li>Expected Impact: Reduce duplicates by 99%, improve completeness to 99%, automated remediation&lt;/li>
&lt;li>Annual Benefit: ~$575,000 (95% of total impact)&lt;/li>
&lt;li>ROI: Break-even in 25 months, limited incremental benefit over Option 2&lt;/li>
&lt;li>Recommendation: Only if expanding to enterprise-wide data quality program&lt;/li>
&lt;/ul>
&lt;p>The magic here is that Option 2 becomes obviously attractive. It&amp;rsquo;s not the cheapest (that&amp;rsquo;s Option 1), but it delivers 86% of possible benefits at one-quarter the cost of Option 3. You&amp;rsquo;re making the decision easy.&lt;/p>
&lt;p>&lt;strong>Part 3: Implementation Roadmap with Quick Wins&lt;/strong>&lt;/p>
&lt;p>Break the implementation into phases with clear deliverables and benefits realisation points. This is critical—executives need to see value delivery, not just promises of future value.&lt;/p>
&lt;p>&lt;strong>Part 4: Risk Assessment and Mitigation&lt;/strong>&lt;/p>
&lt;p>Be honest about risks. It builds credibility, and it shows you&amp;rsquo;ve thought things through.&lt;/p>
&lt;p>&lt;em>Key Risks and Mitigation Strategies&lt;/em>&lt;/p>
&lt;table>
 &lt;thead>
 &lt;tr>
 &lt;th>Risk&lt;/th>
 &lt;th>Probability&lt;/th>
 &lt;th>Impact&lt;/th>
 &lt;th>Mitigation&lt;/th>
 &lt;/tr>
 &lt;/thead>
 &lt;tbody>
 &lt;tr>
 &lt;td>Business users resist process changes&lt;/td>
 &lt;td>Medium&lt;/td>
 &lt;td>High&lt;/td>
 &lt;td>Early stakeholder engagement, involve users in design, demonstrate personal benefit&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Benefits don&amp;rsquo;t materialize as projected&lt;/td>
 &lt;td>Low&lt;/td>
 &lt;td>High&lt;/td>
 &lt;td>Conservative estimates (targeting 70% of possible benefit), monthly tracking with steering committee&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Technical implementation challenges&lt;/td>
 &lt;td>Medium&lt;/td>
 &lt;td>Medium&lt;/td>
 &lt;td>Phased approach allows course correction, leverage proven tools, experienced implementation partner&lt;/td>
 &lt;/tr>
 &lt;tr>
 &lt;td>Ongoing stewardship not sustained&lt;/td>
 &lt;td>Medium&lt;/td>
 &lt;td>High&lt;/td>
 &lt;td>Embed in existing roles, executive sponsorship, tie to performance metrics&lt;/td>
 &lt;/tr>
 &lt;/tbody>
&lt;/table>
&lt;p>This is the business case structure that gets funded. It&amp;rsquo;s specific, quantified, realistic about trade-offs, delivers early value, and manages risk explicitly.&lt;/p>
&lt;p>But here&amp;rsquo;s the part that often trips people up: how do you actually calculate those benefit numbers?&lt;/p>
&lt;p>&lt;strong>Benefit Calculation Frameworks:&lt;/strong>
&lt;/br>&lt;/p>
&lt;p>Duplicate Record Cost :&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>Number of duplicates x (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Marketing cost per contact x duplicate marketing touches ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Customer service call rate x % calling about duplicates × cost per call ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Duplicate payment rate x average duplicate payment × recovery cost ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Failed delivery rate x cost per failure )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;p>Incomplete Data Cost :&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>Number of incomplete records x (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Process failure rate due to missing data × average failure cost ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Manual lookup time per record x hourly labor cost ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Opportunity delay rate x average opportunity value x discount rate )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;p>Inconsistent Data Cost :&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>Number of inconsistent records x (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Access control failure rate x average incident response cost ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Reporting discrepancy investigation time × hourly cost ) +
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>( Compliance risk premium )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;p>Analyst Productivity Recovery :&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-text" data-lang="text">&lt;span style="display:flex;">&lt;span>Number of analysts x (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>time spent on data cleanup x
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>average fully-loaded cost x
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>cleanup time reduction
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;p>You don&amp;rsquo;t need perfect precision in these calculations. You need reasonable estimates backed by actual measurements. Sample your data. Interview stakeholders. Run time studies. Audit incidents. The numbers don&amp;rsquo;t have to be exact—they need to be defensible.&lt;/p>
&lt;p>At the end of the article I will link to the templates I use to quantify - business case calculation.&lt;/p>
&lt;p>The business case isn&amp;rsquo;t really about the numbers. The numbers need to be solid, yes. But what you&amp;rsquo;re really doing is building confidence that this isn&amp;rsquo;t another IT project that consumes budget and delivers vague benefits. You&amp;rsquo;re showing that you understand the business impact, that you have a pragmatic approach, that you&amp;rsquo;ll deliver value incrementally, and that you&amp;rsquo;ve thought about the risks.&lt;/p>
&lt;p>Do that, and you&amp;rsquo;ll get funded.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="stage-4-implementation-that-sticks-because-most-dont">Stage 4: Implementation that sticks (because most don&amp;rsquo;t)&lt;/h3>
&lt;hr>
&lt;p>You&amp;rsquo;ve got the funding. Executives are on board. You&amp;rsquo;ve got a team and a timeline. Great.&lt;/p>
&lt;p>Here&amp;rsquo;s where most data quality initiatives fail: not in the doing, but in the sustaining.&lt;/p>
&lt;p>I&amp;rsquo;ve seen it play out the same way dozens of times. Big kickoff meeting. Lots of enthusiasm. Consultants or tools or both. Six months of intense work. Data gets cleaned up. Reports look better. Everyone celebrates.&lt;/p>
&lt;p>Then the data quality team moves on to the next thing. The governance committee stops meeting. The monitoring dashboard stops being checked. And within 12 months, data quality has regressed to where it started—or worse, because now there&amp;rsquo;s organisational fatigue around the topic.&lt;/p>
&lt;p>&lt;strong>The fundamental mistake: treating data quality as a project instead of an operating model.&lt;/strong>&lt;/p>
&lt;p>Here&amp;rsquo;s what sustainable implementation actually requires:&lt;/p>
&lt;p>&lt;strong>1. Embedded Ownership&lt;/strong>&lt;/p>
&lt;p>Data quality can&amp;rsquo;t live in IT. It can&amp;rsquo;t live with a central data team. It has to live with the people who create, use, and depend on the data.&lt;/p>
&lt;p>This means identifying data stewards—specific individuals who own data quality for specific domains. Not as a side responsibility (&amp;ldquo;you&amp;rsquo;re also the data steward&amp;rdquo;), but as a core part of their role with clear accountabilities.&lt;/p>
&lt;p>For customer data, your steward might be in sales operations or customer service. For supplier data, in procurement. For product data, in product management. The pattern: wherever the data originates and is most critical to operations, that&amp;rsquo;s where stewardship lives.&lt;/p>
&lt;p>What does a data steward actually do?&lt;/p>
&lt;ul>
&lt;li>Monitors quality metrics for their domain (weekly, not monthly)&lt;/li>
&lt;li>Investigates quality issues and coordinates resolution&lt;/li>
&lt;li>Participates in data quality governance forums&lt;/li>
&lt;li>Reviews and approves changes to data standards&lt;/li>
&lt;li>Trains their team on data quality practices&lt;/li>
&lt;li>Acts as escalation point for quality questions&lt;/li>
&lt;/ul>
&lt;p>This isn&amp;rsquo;t a passive role. It&amp;rsquo;s 10-20% of someone&amp;rsquo;s time, and it needs to be in their job description and performance objectives.&lt;/p>
&lt;p>&lt;strong>2. Automated Monitoring with Human Review&lt;/strong>&lt;/p>
&lt;p>You can&amp;rsquo;t manually check data quality. You&amp;rsquo;ll never keep up. But you also can&amp;rsquo;t fully automate it—human judgment is essential for ambiguous cases and root cause analysis.&lt;/p>
&lt;p>The pattern that works: automated monitoring flags issues, humans investigate and resolve.&lt;/p>
&lt;p>This is where the SQL queries matter. You need standard queries that run daily or weekly to check each quality dimension:&lt;/p>
&lt;p>For completeness: What&amp;rsquo;s the % of required fields populated?
For uniqueness: How many potential duplicates based on matching rules?
For timeliness: How many records exceed freshness thresholds?
For validity: How many records fail format checks?
For accuracy: How many records deviate from authoritative sources?
For consistency: How many records show cross-system discrepancies?&lt;/p>
&lt;p>At the end of the article I will link to the templates I use to for testing these dimensions of quality in ANSI SQL format.&lt;/p>
&lt;p>The monitoring workflow:&lt;/p>
&lt;ol>
&lt;li>Automated queries run on schedule&lt;/li>
&lt;li>Results populate a dashboard (not a 50-page report—a dashboard)&lt;/li>
&lt;li>Issues exceeding thresholds generate alerts to stewards&lt;/li>
&lt;li>Stewards investigate within defined SLA (usually 48 hours)&lt;/li>
&lt;li>Root causes documented&lt;/li>
&lt;li>Fixes prioritised based on business impact&lt;/li>
&lt;li>Remediation tracked to completion&lt;/li>
&lt;li>Patterns analysed monthly for systematic improvements&lt;/li>
&lt;/ol>
&lt;p>&lt;strong>3. Preventive Controls at Source&lt;/strong>&lt;/p>
&lt;p>Cleaning data is expensive. Preventing bad data is cheap.&lt;/p>
&lt;p>The best data quality improvements happen at the point of data creation. This means:&lt;/p>
&lt;p>&lt;strong>Entry validation&lt;/strong>: Real-time checks that catch invalid data before it&amp;rsquo;s saved. Email format validation. Postcode pattern checking. Mandatory field enforcement. Date range validation.&lt;/p>
&lt;p>&lt;strong>Duplicate checking&lt;/strong>: Before creating a new supplier, customer, or product record, check for potential duplicates and force the user to confirm it&amp;rsquo;s genuinely new.&lt;/p>
&lt;p>&lt;strong>Authoritative lookups&lt;/strong>: Instead of free-text entry for standard fields (country, department, product category), use dropdowns populated from authoritative lists.&lt;/p>
&lt;p>&lt;strong>Workflow approvals&lt;/strong>: High-impact data changes (customer credit limits, supplier payment terms) require approval before taking effect.&lt;/p>
&lt;p>&lt;strong>Enrichment on entry&lt;/strong>: When a user enters a company name, automatically look up and populate tax ID, registered address, and other public information.&lt;/p>
&lt;p>The technical implementation varies by system, but the principle is universal: make it easy to enter good data and hard to enter bad data.&lt;/p>
&lt;p>&lt;strong>4. The Feedback Loop That Drives Improvement&lt;/strong>&lt;/p>
&lt;p>Here&amp;rsquo;s what separates sustainable implementations from temporary fixes: systematic learning.&lt;/p>
&lt;p>Every data quality issue is a symptom. Sometimes the symptom points to user error. Sometimes it points to process gaps. Sometimes it points to system limitations. The organisations that succeed treat each issue as a learning opportunity.&lt;/p>
&lt;p>The monthly quality review structure I like:&lt;/p>
&lt;ul>
&lt;li>Stewards present quality metrics for their domains (10 minutes each)&lt;/li>
&lt;li>Top 5 quality issues discussed in depth&lt;/li>
&lt;li>Root cause analysis for each (using 5 Whys or similar)&lt;/li>
&lt;li>Categorisation: user training, process change, or system enhancement&lt;/li>
&lt;li>Action plan developed with owners and deadlines&lt;/li>
&lt;li>Previous month&amp;rsquo;s actions reviewed for effectiveness&lt;/li>
&lt;li>Patterns analysed across domains&lt;/li>
&lt;/ul>
&lt;p>This isn&amp;rsquo;t a bureaucratic committee meeting. It&amp;rsquo;s a working session where problems get solved. The key is having the right people in the room—people with authority to make decisions about training, process changes, and system enhancements.&lt;/p>
&lt;p>&lt;strong>5. Consequences and Incentives&lt;/strong>&lt;/p>
&lt;p>This is the uncomfortable part. Data quality has to matter to individual performance.&lt;/p>
&lt;p>Not in a punitive way—you&amp;rsquo;re not measuring keystrokes or penalizing errors. But data quality practices need to be part of performance discussions, promotion criteria, and team metrics.&lt;/p>
&lt;p>Examples that work:&lt;/p>
&lt;ul>
&lt;li>Sales teams measured on customer data completeness (because it affects marketing effectiveness)&lt;/li>
&lt;li>Procurement teams measured on supplier data accuracy (because it affects payment processing)&lt;/li>
&lt;li>Product teams measured on product data consistency (because it affects inventory management)&lt;/li>
&lt;/ul>
&lt;p>Examples that don&amp;rsquo;t work:&lt;/p>
&lt;ul>
&lt;li>Individual error rates (creates gaming and data hiding)&lt;/li>
&lt;li>Blame for historical bad data (demotivates participation)&lt;/li>
&lt;li>IT metrics that business doesn&amp;rsquo;t understand or care about&lt;/li>
&lt;/ul>
&lt;p>The incentive structure should reward:&lt;/p>
&lt;ul>
&lt;li>Identifying and reporting data quality issues&lt;/li>
&lt;li>Participating in cleanup initiatives&lt;/li>
&lt;li>Following data quality processes consistently&lt;/li>
&lt;li>Sharing best practices across teams&lt;/li>
&lt;li>Driving improvements in their domain&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>6. The Data Quality Flywheel&lt;/strong>&lt;/p>
&lt;p>Once you&amp;rsquo;ve got these elements in place, something interesting happens. Data quality improvements start to compound.&lt;/p>
&lt;p>Better data quality → More trust in data → More data usage → More identified quality issues → More improvements → Better data quality&lt;/p>
&lt;p>The opposite is also true:
Poor data quality → Low trust → Manual workarounds → More data quality issues → Less trust → Poor data quality&lt;/p>
&lt;p>Your job is to get the flywheel spinning in the right direction and keep it there long enough that momentum sustains it.&lt;/p>
&lt;p>How long does this take? 18 months to reach genuine sustainability. The first 6 months are intensive improvement. The next 6 months are stabilization. The final 6 months are optimisation and cultural embedding.&lt;/p>
&lt;p>Most organisations quit at month 9, right when they&amp;rsquo;re on the edge of breakthrough. Don&amp;rsquo;t be most organisations.&lt;/p>
&lt;p>The organisations that succeed treat data quality like they treat financial controls or quality manufacturing processes—as a permanent, essential operating discipline. They don&amp;rsquo;t declare victory and move on. They continuously improve, responding to changing business needs and new data sources.&lt;/p>
&lt;p>That&amp;rsquo;s implementation that sticks.&lt;/p>
&lt;hr>
&lt;p>If you want my version of tools that I use, check out the package here &lt;a href="https://payhip.com/buy?link=Uqlwi" target="_blank" rel="noopener" data-product="Uqlwi" data-payhip-overlay>Buy Data Quality - SQL and Python Scripts&lt;/a>
&lt;/br>
&lt;/br>&lt;/p>
&lt;script src="https://ghostinthedata.info/scripts/payhip-overlay.min.66585eb4da3976da60ea8d3f2da9f48738b3b0a4583c7fd937df92e4e0feaec9.js" integrity="sha256-ZlhetNo5dtpg6o0/Lan0hzizsKRYPH/ZN9&amp;#43;S5OD&amp;#43;rsk=" defer>&lt;/script></content:encoded><category>Data Quality &amp; Governance</category><category>Leadership</category><category>Data Quality</category><category>ROI</category><category>Business Case</category><category>Data Governance</category><category>Strategy</category><category>Frameworks</category></item><item><title>When Your Data Quality Fails at 9 PM on a Friday</title><link>https://ghostinthedata.info/posts/2025/2025-11-22-data-quality/</link><pubDate>Sat, 22 Nov 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-11-22-data-quality/</guid><author>Chris Hillman</author><description>A deep exploration of data quality principles through the lens of a catastrophic Airbnb double-booking bug—and how to prevent your own nightmare</description><content:encoded>&lt;h3 id="when-everything-goes-wrong-at-once">When everything goes wrong at once&lt;/h3>
&lt;hr>
&lt;p>It&amp;rsquo;s 9 PM on a Friday. You&amp;rsquo;re halfway through your second beer, finally relaxing after a brutal week. Your phone buzzes. Then it buzzes again. And again. The support team&amp;rsquo;s in full panic mode, your manager&amp;rsquo;s calling, and somewhere in Melbourne, two very angry guests are standing outside the same Airbnb property—both holding confirmation emails that say the place is theirs for the weekend.&lt;/p>
&lt;p>The keys don&amp;rsquo;t work for one of them. The host isn&amp;rsquo;t answering. Both guests are already tweeting about it. Your company&amp;rsquo;s about to become the main character on social media, and not in a good way.&lt;/p>
&lt;p>This isn&amp;rsquo;t hypothetical. This is what happens when data quality breaks down in production, and it&amp;rsquo;s more common than you&amp;rsquo;d think. I&amp;rsquo;ve been there. Most data professionals have been there at some point—that sinking feeling when you realise a query you wrote, a constraint you didn&amp;rsquo;t add, or a test you didn&amp;rsquo;t think was necessary just created a real-world disaster.&lt;/p>
&lt;p>Today we&amp;rsquo;re going to walk through exactly how a seemingly simple SQL query can create this nightmare. More importantly, we&amp;rsquo;re going to build a framework for preventing it—one that goes beyond &amp;ldquo;just write better code&amp;rdquo; and addresses the systemic issues that let these problems slip through in the first place.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-real-cost-of-getting-it-wrong">The real cost of getting it wrong&lt;/h3>
&lt;hr>
&lt;p>When data quality fails, the consequences don&amp;rsquo;t show up as a ticket in your project management system marked &amp;ldquo;P3 - Low Priority.&amp;rdquo; They show up as:&lt;/p>
&lt;p>&lt;strong>Customer trust evaporating in real-time&lt;/strong>. Those two guests aren&amp;rsquo;t just filing support tickets. They&amp;rsquo;re telling everyone they know. They&amp;rsquo;re warning their friends. They&amp;rsquo;re writing detailed reviews about how your platform failed them when they needed it most. You&amp;rsquo;ve lost them forever, plus everyone they influence.&lt;/p>
&lt;p>&lt;strong>Revenue hemorrhaging faster than you can calculate it&lt;/strong>. Immediate refunds, sure. Compensation packages to prevent lawsuits, definitely. But what about the bookings those guests would&amp;rsquo;ve made next year? The referrals you&amp;rsquo;ll never get? The hosts who lose confidence in your platform and stop accepting new bookings?&lt;/p>
&lt;p>&lt;strong>Operational chaos that cascades&lt;/strong>. Your support team&amp;rsquo;s drowning. They can&amp;rsquo;t keep up with the volume. Other tickets are backing up. Response times are slipping across the board. The hosts are freaking out because they&amp;rsquo;re caught in the middle. Your trust and safety team is scrambling to figure out if this is isolated or systemic.&lt;/p>
&lt;p>&lt;strong>Regulatory nightmares you didn&amp;rsquo;t see coming&lt;/strong>. Depending on your jurisdiction, you might&amp;rsquo;ve just violated consumer protection laws. The legal team&amp;rsquo;s getting involved. Someone&amp;rsquo;s drafting incident reports for regulators. This isn&amp;rsquo;t just an engineering problem anymore.&lt;/p>
&lt;p>&lt;strong>Brand damage that&amp;rsquo;s impossible to quantify&lt;/strong>. Social media amplifies everything. That tweet from Guest #1 is getting retweeted. Tech bloggers are picking it up. Someone on Hacker News is explaining why your engineering team clearly doesn&amp;rsquo;t understand basic database constraints. Your CEO&amp;rsquo;s getting messages from investors asking what&amp;rsquo;s going on.&lt;/p>
&lt;p>And here&amp;rsquo;s the thing—all of this stemmed from a single flawed SQL query.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-bug-that-broke-everything">The bug that broke everything&lt;/h3>
&lt;hr>
&lt;p>Let me show you the actual data that caused this disaster. We&amp;rsquo;ve got a bookings table—nothing fancy, just the basics:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> bookings (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> room_id INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> booking_id INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> guest_name VARCHAR(&lt;span style="color:#ae81ff">100&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_date DATE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> end_date DATE
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> bookings &lt;span style="color:#66d9ef">VALUES&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">101&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Alice&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-01&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-10&amp;#39;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">101&lt;/span>, &lt;span style="color:#ae81ff">2&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Bob&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-05&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-07&amp;#39;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">101&lt;/span>, &lt;span style="color:#ae81ff">3&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Charlie&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-08&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-12&amp;#39;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">101&lt;/span>, &lt;span style="color:#ae81ff">4&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Diana&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-15&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-18&amp;#39;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">102&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Eve&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-03&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-06&amp;#39;&lt;/span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> (&lt;span style="color:#ae81ff">102&lt;/span>, &lt;span style="color:#ae81ff">6&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;Frank&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-08&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;2024-01-11&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Look at Room 101. Alice booked January 1-10. Bob&amp;rsquo;s there January 5-7. Charlie arrives January 8 and stays through the 12th.&lt;/p>
&lt;p>You see the problem, right? Alice and Charlie both think they have the place from January 8-10. That&amp;rsquo;s three nights where two different guests are expecting exclusive access to the same property.&lt;/p>
&lt;p>Now here&amp;rsquo;s where it gets interesting (and by interesting, I mean horrifying). The development team wrote what seemed like a clever query to detect overlaps:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The query that seemed so clever in code review
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> room_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> booking_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> end_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> LEAD(start_date) OVER (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARTITION &lt;span style="color:#66d9ef">BY&lt;/span> room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">as&lt;/span> next_start
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> LEAD(start_date) OVER (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> PARTITION &lt;span style="color:#66d9ef">BY&lt;/span> room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The &lt;code>LEAD()&lt;/code> function looks at the next booking after sorting by start date. It compares each booking to the one immediately following it. Sounds reasonable, right?&lt;/p>
&lt;p>Except it completely misses Alice and Charlie&amp;rsquo;s overlap.&lt;/p>
&lt;p>Why? Because Bob&amp;rsquo;s booking sits between them chronologically. The query compares Alice to Bob (finds overlap), compares Bob to Charlie (finds overlap), but never compares Alice to Charlie directly. It&amp;rsquo;s like having three people in a room where person A and person C are standing on each other&amp;rsquo;s toes, but you only check if adjacent people are touching.&lt;/p>
&lt;p>This is the kind of bug that passes code review. It passes initial testing if you&amp;rsquo;re not thorough. It might even work perfectly fine in production for months until you hit just the right combination of bookings.&lt;/p>
&lt;p>And when it fails? It fails spectacularly, on a Friday night, when two guests are standing outside the same property.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-six-dimensions-that-define-quality">The six dimensions that define quality&lt;/h3>
&lt;hr>
&lt;p>Data quality isn&amp;rsquo;t a binary thing where data is either &amp;ldquo;good&amp;rdquo; or &amp;ldquo;bad.&amp;rdquo; It&amp;rsquo;s multidimensional, and understanding these dimensions changes how you think about building systems.&lt;/p>
&lt;p>&lt;strong>Accuracy&lt;/strong> asks whether your data correctly represents reality. Guest email is &amp;ldquo;&lt;a href="mailto:john.smith@email.com">john.smith@email.com&lt;/a>&amp;rdquo; and that&amp;rsquo;s actually John&amp;rsquo;s email? Accurate. Phone number has a typo? Inaccurate. Here&amp;rsquo;s the tricky part—in our booking scenario, even if the dates are recorded wrong (user books Jan 8-12 but system records Jan 8-10), every downstream calculation becomes inaccurate, even if your overlap detection logic is perfect.&lt;/p>
&lt;p>&lt;strong>Completeness&lt;/strong> means all necessary data is present. Your booking has room_id, booking_id, guest info, dates, contact details? Complete. Missing contact information so you can&amp;rsquo;t notify guests of problems? Incomplete. But here&amp;rsquo;s what most people miss—the overlap detection logic itself was incomplete. It checked some overlaps but not all overlaps. Missing validation logic is as critical as missing data.&lt;/p>
&lt;p>&lt;strong>Consistency&lt;/strong> asks whether data agrees across systems. Guest&amp;rsquo;s booking shows &amp;ldquo;Room 101&amp;rdquo; in both booking and property management systems? Consistent. Booking system shows checkout on Jan 10 but calendar sync shows Jan 11? Inconsistent. And this is where things get messy in real systems—you&amp;rsquo;ve got data flowing between multiple databases, cache layers, message queues, third-party integrations. Maintaining consistency across all of that is harder than it looks.&lt;/p>
&lt;p>&lt;strong>Timeliness&lt;/strong> measures whether data is current and available when needed. Booking confirmation sent within 30 seconds? Timely. Cancellation not reflected in availability for 24 hours? Untimely, and now you&amp;rsquo;re creating new double-bookings. Even perfect overlap detection doesn&amp;rsquo;t help if it runs as a nightly batch job instead of real-time validation.&lt;/p>
&lt;p>&lt;strong>Validity&lt;/strong> checks if data conforms to business rules. Email follows format rules? Valid. Booking end_date after start_date? Valid. Two bookings overlapping for the same room? Invalid—this was our core problem. The business rule was crystal clear: &amp;ldquo;No room can have overlapping bookings.&amp;rdquo; The LEAD() query failed to properly validate this rule.&lt;/p>
&lt;p>&lt;strong>Uniqueness&lt;/strong> ensures entities are recorded once and only once. Each booking has a unique booking_id? Unique. Same guest creates account twice with slightly different emails? Duplicate. And while not our primary issue, if bookings get recorded multiple times (user double-clicks submit), you&amp;rsquo;ve got duplicate records complicating your overlap detection.&lt;/p>
&lt;p>These dimensions interact in ways that&amp;rsquo;ll bite you if you&amp;rsquo;re not careful. A timeliness issue creates stale data, which becomes an accuracy problem, which breaks consistency, which produces invalid states. In our case, validity failed first (overlap logic was wrong), causing accuracy issues (booking data didn&amp;rsquo;t reflect reality), leading to consistency problems (booking table disagreed with availability calendar), creating timeliness challenges (support scrambling to fix in real-time).&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="how-the-disaster-unfolds">How the disaster unfolds&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s walk through what actually happened when our buggy query made it to production.&lt;/p>
&lt;p>&lt;strong>Development phase&lt;/strong>: Someone writes the LEAD() query. It looks elegant. Uses a window function. Modern SQL. The code reviewer sees it&amp;rsquo;s checking for overlaps and approves it. Nobody thinks to test the non-adjacent overlap case because, honestly, who thinks about that?&lt;/p>
&lt;p>&lt;strong>Testing phase&lt;/strong>: QA creates test cases. They test two adjacent bookings. Works perfectly. They test bookings with gaps. Works perfectly. They test bookings in different rooms. Works perfectly. But they don&amp;rsquo;t test three bookings where A overlaps C with B sitting between them. Why would they? The spec didn&amp;rsquo;t call it out.&lt;/p>
&lt;p>&lt;strong>Production deployment&lt;/strong>: Everything&amp;rsquo;s smooth. No errors during rollout. Initial bookings look good. The system&amp;rsquo;s humming along. For weeks, maybe months, you don&amp;rsquo;t see the problem because your booking patterns haven&amp;rsquo;t hit that specific edge case.&lt;/p>
&lt;p>&lt;strong>The incident&lt;/strong>: Then one day, someone books around an existing booking. Alice extends her stay. Or Charlie books early. Or the timing just lines up wrong. Suddenly you&amp;rsquo;ve got overlapping bookings in the database, both confirmed, both valid as far as the system knows.&lt;/p>
&lt;p>&lt;strong>The discovery&lt;/strong>: It&amp;rsquo;s not immediate. Both guests receive confirmation emails. Both guests make travel plans. Both guests show up. That&amp;rsquo;s when someone discovers the problem—when it&amp;rsquo;s too late to fix gracefully.&lt;/p>
&lt;p>&lt;strong>The response&lt;/strong>: Support gets flooded. Engineering gets paged. Someone runs a query to check for other overlaps. Oh no, there are dozens. Some in the future. Some happening right now. The scope of the problem becomes clear, and it&amp;rsquo;s worse than anyone thought.&lt;/p>
&lt;p>&lt;strong>The aftermath&lt;/strong>: You&amp;rsquo;re fixing data in production. You&amp;rsquo;re calling customers. You&amp;rsquo;re issuing refunds. You&amp;rsquo;re updating hosts. You&amp;rsquo;re explaining to executives. You&amp;rsquo;re writing post-mortems. You&amp;rsquo;re implementing new tests. You&amp;rsquo;re adding database constraints. You&amp;rsquo;re probably looking for a new job.&lt;/p>
&lt;p>This sequence plays out somewhere in the data world every single day. The specific domain changes—it might be financial transactions, healthcare records, e-commerce inventory, user permissions. But the pattern stays the same: subtle bug, insufficient testing, production deployment, catastrophic failure, expensive cleanup.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="building-a-framework-that-actually-works">Building a framework that actually works&lt;/h3>
&lt;hr>
&lt;p>Fixing the query is easy. Preventing the next incident requires systemic change. Let me show you a framework that&amp;rsquo;s saved my ass more times than I can count.&lt;/p>
&lt;p>&lt;strong>Principle 1: The database should enforce truth&lt;/strong>&lt;/p>
&lt;p>Your database knows what valid data looks like. Make it impossible to insert invalid data. Not hard, not inconvenient—impossible.&lt;/p>
&lt;p>For our booking scenario:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Basic date validation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> bookings &lt;span style="color:#66d9ef">ADD&lt;/span> &lt;span style="color:#66d9ef">CONSTRAINT&lt;/span> valid_date_range
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CHECK&lt;/span> (end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> start_date);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Now comes the interesting part
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">FUNCTION&lt;/span> prevent_booking_overlap()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">RETURNS&lt;/span> &lt;span style="color:#66d9ef">TRIGGER&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">BEGIN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> room_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> booking_id &lt;span style="color:#f92672">!=&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">THEN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> RAISE &lt;span style="color:#66d9ef">EXCEPTION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Booking overlap detected for room % between % and %&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">NEW&lt;/span>.room_id, &lt;span style="color:#66d9ef">NEW&lt;/span>.start_date, &lt;span style="color:#66d9ef">NEW&lt;/span>.end_date;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">RETURN&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">END&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span> &lt;span style="color:#66d9ef">LANGUAGE&lt;/span> plpgsql;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TRIGGER&lt;/span> validate_booking_overlap
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">BEFORE&lt;/span> &lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">UPDATE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">EACH&lt;/span> &lt;span style="color:#66d9ef">ROW&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">FUNCTION&lt;/span> prevent_booking_overlap();
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This trigger runs before every insert or update. It checks every existing booking for that room. If there&amp;rsquo;s any overlap—adjacent or not, doesn&amp;rsquo;t matter—the insert fails. You can&amp;rsquo;t create invalid data even if you try.&lt;/p>
&lt;p>Some people worry about performance. &amp;ldquo;Running a query on every insert will be slow!&amp;rdquo; Maybe. But you know what&amp;rsquo;s slower? Manually fixing data corruption at 9 PM on Friday. Besides, this query is simple, hits an index, and finishes in milliseconds. Profile it if you&amp;rsquo;re worried, but in my experience, correctness beats speed every time.&lt;/p>
&lt;p>&lt;strong>Principle 2: Tests must cover your assumptions&lt;/strong>&lt;/p>
&lt;p>You can&amp;rsquo;t test everything. But you absolutely must test your critical business rules, and you need to think adversarially about edge cases.&lt;/p>
&lt;p>For overlap detection, the test suite needs to cover:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 1: Adjacent overlaps (the easy one)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Booking A: Jan 1-10, Booking B: Jan 5-7
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: Overlap detected
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 2: Non-adjacent overlaps (the one we missed)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Booking A: Jan 1-10, Booking B: Jan 5-7, Booking C: Jan 8-12
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: A-C overlap detected
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 3: Exact match
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Booking A: Jan 1-10, Booking B: Jan 1-10
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: Overlap detected
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 4: One booking completely contains another
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Booking A: Jan 1-15, Booking B: Jan 5-7
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: Overlap detected
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 5: Bookings touch but don&amp;#39;t overlap
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Booking A: Jan 1-10, Booking B: Jan 10-15
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: No overlap (checkout day is available)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Test Case 6: Different rooms
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Room 101 Booking A: Jan 1-10, Room 102 Booking B: Jan 5-7
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Expected: No overlap
&lt;/span>&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Notice Test Case 5. Should checkout day equal next checkin day trigger an overlap? Depends on your business rules. Maybe you need a few hours between bookings for cleaning. Document these assumptions explicitly.&lt;/p>
&lt;p>Write these tests before fixing the bug. Watch them fail. Then fix the bug and watch them pass. Now you&amp;rsquo;ve got protection against regression.&lt;/p>
&lt;p>&lt;strong>Principle 3: Monitoring catches what testing misses&lt;/strong>&lt;/p>
&lt;p>Tests verify behaviour under known conditions. Monitoring detects problems under real conditions.&lt;/p>
&lt;p>Create a daily data quality report:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">VIEW&lt;/span> daily_data_quality_report &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WITH&lt;/span> overlap_check &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;Booking Overlaps&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> metric,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> issue_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;CRITICAL&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> severity
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> bookings b
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> a.room_id &lt;span style="color:#f92672">=&lt;/span> b.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.booking_id &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> b.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>date_check &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;Invalid Date Ranges&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> metric,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> issue_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;HIGH&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> severity
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> end_date &lt;span style="color:#f92672">&amp;lt;=&lt;/span> start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>completeness_check &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;Missing Guest Names&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> metric,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> issue_count,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#39;MEDIUM&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> severity
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> guest_name &lt;span style="color:#66d9ef">IS&lt;/span> &lt;span style="color:#66d9ef">NULL&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> guest_name &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> overlap_check
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> date_check
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> completeness_check;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Run this every morning. Alert if any check returns &lt;code>issue_count &amp;gt; 0&lt;/code>. You&amp;rsquo;ll catch problems before customers do.&lt;/p>
&lt;p>Better yet, surface these metrics on a dashboard everyone can see. Make data quality visible. When the overlap count goes from zero to three, someone notices immediately.&lt;/p>
&lt;p>&lt;strong>Principle 4: Culture eats process for breakfast&lt;/strong>&lt;/p>
&lt;p>You can have perfect constraints, comprehensive tests, and real-time monitoring. If your team culture doesn&amp;rsquo;t value data quality, you&amp;rsquo;ll still have problems.&lt;/p>
&lt;p>Building the right culture means:&lt;/p>
&lt;p>Making data quality everyone&amp;rsquo;s job, not just the data team&amp;rsquo;s. When a product manager writes requirements, they specify data validation rules. When a designer creates forms, they consider what invalid input looks like. When an engineer writes code, they think about data integrity.&lt;/p>
&lt;p>Treating data quality incidents as learning opportunities, not blame games. When something breaks, the question isn&amp;rsquo;t &amp;ldquo;whose fault is this?&amp;rdquo; It&amp;rsquo;s &amp;ldquo;what system failed and how do we fix it?&amp;rdquo; The engineer who wrote the buggy query isn&amp;rsquo;t the problem. The process that let it reach production is.&lt;/p>
&lt;p>Celebrating prevention as much as firefighting. The engineer who adds a constraint that prevents a future incident deserves recognition. The QA who writes a test that catches a bug before production deserves praise. But this rarely happens—we celebrate the hero who fixed production at 2 AM, not the person who prevented the incident entirely.&lt;/p>
&lt;p>Investing time in foundational work. Yes, building that monitoring dashboard takes time away from new features. Yes, writing comprehensive tests slows down initial development. But it speeds up everything else—debugging, onboarding, refactoring, building on top of existing code.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-correct-query-finally">The correct query (finally)&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s fix the actual overlap detection query. The correct approach uses a self-join:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">DISTINCT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.room_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.booking_id &lt;span style="color:#66d9ef">as&lt;/span> booking_a,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.guest_name &lt;span style="color:#66d9ef">as&lt;/span> guest_a,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.start_date &lt;span style="color:#66d9ef">as&lt;/span> start_a,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.end_date &lt;span style="color:#66d9ef">as&lt;/span> end_a,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.booking_id &lt;span style="color:#66d9ef">as&lt;/span> booking_b,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.guest_name &lt;span style="color:#66d9ef">as&lt;/span> guest_b,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.start_date &lt;span style="color:#66d9ef">as&lt;/span> start_b,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.end_date &lt;span style="color:#66d9ef">as&lt;/span> end_b,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> GREATEST(a.start_date, b.start_date) &lt;span style="color:#66d9ef">as&lt;/span> overlap_start,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> LEAST(a.end_date, b.end_date) &lt;span style="color:#66d9ef">as&lt;/span> overlap_end
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> bookings a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> bookings b
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> a.room_id &lt;span style="color:#f92672">=&lt;/span> b.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.booking_id &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> b.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> a.room_id, a.start_date;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This query compares every booking to every other booking in the same room. No more &amp;ldquo;adjacent only&amp;rdquo; limitation. If any two bookings overlap, we find them.&lt;/p>
&lt;p>The join conditions are crucial:&lt;/p>
&lt;ul>
&lt;li>&lt;code>a.room_id = b.room_id&lt;/code> - only compare bookings in the same room&lt;/li>
&lt;li>&lt;code>a.booking_id &amp;lt; b.booking_id&lt;/code> - avoid comparing a booking to itself and avoid duplicate pairs&lt;/li>
&lt;li>&lt;code>a.start_date &amp;lt; b.end_date&lt;/code> - booking A starts before booking B ends&lt;/li>
&lt;li>&lt;code>a.end_date &amp;gt; b.start_date&lt;/code> - booking A ends after booking B starts&lt;/li>
&lt;/ul>
&lt;p>This is the standard interval overlap formula. It&amp;rsquo;s been used for decades. It works. Use it.&lt;/p>
&lt;p>Run this query against your current data:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Emergency audit script
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> total_overlaps,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#66d9ef">DISTINCT&lt;/span> room_id) &lt;span style="color:#66d9ef">as&lt;/span> affected_rooms
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">DISTINCT&lt;/span> a.room_id, a.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> bookings b
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> a.room_id &lt;span style="color:#f92672">=&lt;/span> b.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.booking_id &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> b.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>) &lt;span style="color:#66d9ef">overlaps&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If this returns anything other than zero, you&amp;rsquo;ve got work to do.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="implementing-the-fix-without-breaking-everything-else">Implementing the fix without breaking everything else&lt;/h3>
&lt;hr>
&lt;p>You can&amp;rsquo;t just deploy new constraints to production and hope for the best. You need a careful rollout strategy that accounts for existing bad data.&lt;/p>
&lt;p>&lt;strong>Phase 1: Discovery (Day 1)&lt;/strong>&lt;/p>
&lt;p>Run the corrected query against production. Export all overlaps. Calculate the scope of damage. How many customers affected? How many rooms? How far into the future?&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Generate a report of all overlaps
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> TEMP &lt;span style="color:#66d9ef">TABLE&lt;/span> overlap_report &lt;span style="color:#66d9ef">AS&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.room_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.booking_id &lt;span style="color:#66d9ef">as&lt;/span> first_booking,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.guest_name &lt;span style="color:#66d9ef">as&lt;/span> first_guest,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.booking_id &lt;span style="color:#66d9ef">as&lt;/span> second_booking,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> b.guest_name &lt;span style="color:#66d9ef">as&lt;/span> second_guest,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> GREATEST(a.start_date, b.start_date) &lt;span style="color:#66d9ef">as&lt;/span> overlap_start,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> LEAST(a.end_date, b.end_date) &lt;span style="color:#66d9ef">as&lt;/span> overlap_end,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> GREATEST(a.start_date, b.start_date) &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Past&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> GREATEST(a.start_date, b.start_date) &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Today&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ELSE&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Future&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> urgency
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> bookings a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> bookings b
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> a.room_id &lt;span style="color:#f92672">=&lt;/span> b.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.booking_id &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> b.start_date;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Prioritize by urgency
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> urgency, &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> &lt;span style="color:#66d9ef">count&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> overlap_report
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> urgency
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">CASE&lt;/span> urgency
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Today&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Future&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHEN&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Past&amp;#39;&lt;/span> &lt;span style="color:#66d9ef">THEN&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Phase 2: Triage (Days 2-3)&lt;/strong>&lt;/p>
&lt;p>Handle immediate issues first. Anyone checking in today or tomorrow needs manual resolution. Contact guests, contact hosts, figure out alternative arrangements. This is messy human work that can&amp;rsquo;t be automated.&lt;/p>
&lt;p>For future bookings, you&amp;rsquo;ve got options:&lt;/p>
&lt;ul>
&lt;li>Cancel the later booking and issue a refund&lt;/li>
&lt;li>Find alternative accommodations&lt;/li>
&lt;li>Negotiate with guests to change dates&lt;/li>
&lt;li>In extreme cases, cancel both and eat the compensation cost&lt;/li>
&lt;/ul>
&lt;p>There&amp;rsquo;s no good answer here. Every option makes someone unhappy. Document your decisions and learn from them.&lt;/p>
&lt;p>&lt;strong>Phase 3: Data Cleanup (Days 3-7)&lt;/strong>&lt;/p>
&lt;p>Once you&amp;rsquo;ve handled the urgent cases, clean up the historical data:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Mark overlapping bookings for review
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> bookings &lt;span style="color:#66d9ef">ADD&lt;/span> &lt;span style="color:#66d9ef">COLUMN&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> data_quality_flag VARCHAR(&lt;span style="color:#ae81ff">50&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">UPDATE&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SET&lt;/span> data_quality_flag &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;overlap_detected&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> booking_id &lt;span style="color:#66d9ef">IN&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">DISTINCT&lt;/span> a.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings a
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> bookings b
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> a.room_id &lt;span style="color:#f92672">=&lt;/span> b.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.booking_id &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> b.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> a.end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> b.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Don&amp;rsquo;t delete data. Flag it. You might need it for analysis, audits, or legal purposes.&lt;/p>
&lt;p>&lt;strong>Phase 4: Prevention (Week 2)&lt;/strong>&lt;/p>
&lt;p>Now you can add the database constraint:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- First, make sure you&amp;#39;ve cleaned up all existing overlaps
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- This should return 0 rows
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">COUNT&lt;/span>(&lt;span style="color:#f92672">*&lt;/span>) &lt;span style="color:#66d9ef">FROM&lt;/span> bookings &lt;span style="color:#66d9ef">WHERE&lt;/span> data_quality_flag &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;overlap_detected&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Deploy the trigger
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">REPLACE&lt;/span> &lt;span style="color:#66d9ef">FUNCTION&lt;/span> prevent_booking_overlap()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">RETURNS&lt;/span> &lt;span style="color:#66d9ef">TRIGGER&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">BEGIN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">IF&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> room_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.room_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> booking_id &lt;span style="color:#f92672">!=&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.booking_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> start_date &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.end_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> end_date &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>.start_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ) &lt;span style="color:#66d9ef">THEN&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> RAISE &lt;span style="color:#66d9ef">EXCEPTION&lt;/span> &lt;span style="color:#e6db74">&amp;#39;Booking overlap detected for room % between % and %&amp;#39;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">NEW&lt;/span>.room_id, &lt;span style="color:#66d9ef">NEW&lt;/span>.start_date, &lt;span style="color:#66d9ef">NEW&lt;/span>.end_date;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">END&lt;/span> &lt;span style="color:#66d9ef">IF&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">RETURN&lt;/span> &lt;span style="color:#66d9ef">NEW&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">END&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#960050;background-color:#1e0010">$$&lt;/span> &lt;span style="color:#66d9ef">LANGUAGE&lt;/span> plpgsql;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TRIGGER&lt;/span> validate_booking_overlap
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">BEFORE&lt;/span> &lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">OR&lt;/span> &lt;span style="color:#66d9ef">UPDATE&lt;/span> &lt;span style="color:#66d9ef">ON&lt;/span> bookings
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">EACH&lt;/span> &lt;span style="color:#66d9ef">ROW&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXECUTE&lt;/span> &lt;span style="color:#66d9ef">FUNCTION&lt;/span> prevent_booking_overlap();
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Test the trigger in staging first. Try to insert overlapping bookings. Verify they&amp;rsquo;re rejected. Try valid bookings. Verify they succeed. Only then deploy to production.&lt;/p>
&lt;p>&lt;strong>Phase 5: Testing (Week 3)&lt;/strong>&lt;/p>
&lt;p>Write automated tests that run against production-like data:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pytest
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> datetime &lt;span style="color:#f92672">import&lt;/span> date, timedelta
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_adjacent_overlap_rejected&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Test that adjacent bookings can overlap&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Insert first booking&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Try to insert overlapping booking&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> pytest&lt;span style="color:#f92672">.&lt;/span>raises(OverlapError):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">7&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_non_adjacent_overlap_rejected&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Test that non-adjacent overlaps are detected&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Insert first booking&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Insert non-overlapping booking&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">5&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">7&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Try to insert booking that overlaps first but not second&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> pytest&lt;span style="color:#f92672">.&lt;/span>raises(OverlapError):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">8&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">12&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">test_different_rooms_allowed&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Test that same dates in different rooms work&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">101&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Should succeed - different room&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> insert_booking(room_id&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">102&lt;/span>, start&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>), end&lt;span style="color:#f92672">=&lt;/span>date(&lt;span style="color:#ae81ff">2024&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">10&lt;/span>))
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Run these tests on every deployment. If they fail, the deployment fails. No exceptions. I would even reference in the code, to the incident, and problem that arose, so if someone things about removing this at a later date they have reference to it.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-this-looks-like-at-scale">What this looks like at scale&lt;/h3>
&lt;hr>
&lt;p>Everything I&amp;rsquo;ve described works at small scale. But what happens when you&amp;rsquo;re dealing with millions of bookings, thousands of rooms, and hundreds of transactions per second?&lt;/p>
&lt;p>&lt;strong>Performance considerations become real&lt;/strong>. That trigger checking for overlaps? It&amp;rsquo;s running a query on every insert. With proper indexing on &lt;code>(room_id, start_date, end_date)&lt;/code>, it&amp;rsquo;ll stay fast. Without indexing, you&amp;rsquo;ve got problems.&lt;/p>
&lt;p>&lt;strong>Monitoring needs to scale too&lt;/strong>. You can&amp;rsquo;t run a full table scan every morning to check for overlaps. You need incremental checks, sampling strategies, or streaming validation.&lt;/p>
&lt;p>&lt;strong>Data cleanup gets complicated&lt;/strong>. You can&amp;rsquo;t manually review millions of records. You need automated classification, bulk correction tools, and clear escalation paths for edge cases.&lt;/p>
&lt;p>&lt;strong>Testing requires different strategies&lt;/strong>. Property-based testing becomes valuable. Generate random booking scenarios and verify overlaps are always caught. Run chaos engineering experiments where you deliberately try to break constraints.&lt;/p>
&lt;p>But here&amp;rsquo;s the thing—the fundamentals don&amp;rsquo;t change. Database constraints still enforce truth. Tests still verify behaviour. Monitoring still catches problems. Culture still determines whether people care.&lt;/p>
&lt;p>The implementation details scale up, but the principles stay the same.&lt;/p>
&lt;h3 id="the-post-mortem-that-actually-matters">The post-mortem that actually matters&lt;/h3>
&lt;hr>
&lt;p>After you&amp;rsquo;ve fixed the bug, cleaned the data, and added constraints, you need a post-mortem. Not the kind where everyone pretends to be honest while actually covering their ass. The kind where you genuinely figure out what went wrong and how to prevent it.&lt;/p>
&lt;p>Here&amp;rsquo;s what a good post-mortem covers:&lt;/p>
&lt;p>&lt;strong>What happened, precisely&lt;/strong>. Timeline of events. Who discovered the issue, when, how. What was the immediate impact. What systems were affected. Use facts, not blame.&lt;/p>
&lt;p>&lt;strong>Root cause analysis&lt;/strong>. Not &amp;ldquo;the engineer wrote bad code&amp;rdquo; (that&amp;rsquo;s always true). Dig deeper. Why did the code review miss it? Why didn&amp;rsquo;t tests catch it? Why wasn&amp;rsquo;t there a database constraint? Why didn&amp;rsquo;t monitoring alert us? Why weren&amp;rsquo;t business rules documented?&lt;/p>
&lt;p>&lt;strong>Contributing factors&lt;/strong>. Was the team under time pressure? Were requirements unclear? Did we lack domain expertise? Were we using unfamiliar technology? Did we skip steps in the normal process?&lt;/p>
&lt;p>&lt;strong>What worked&lt;/strong>. Don&amp;rsquo;t only focus on failures. What did work? How did we detect the problem? What allowed us to respond quickly? Who stepped up? What processes helped?&lt;/p>
&lt;p>&lt;strong>Action items with owners and deadlines&lt;/strong>. Vague &amp;ldquo;we should do better&amp;rdquo; doesn&amp;rsquo;t help. Specific &amp;ldquo;Jane will implement overlap detection tests by next Friday&amp;rdquo; does. Assign owners. Set deadlines. Follow up.&lt;/p>
&lt;p>&lt;strong>Distribution and discussion&lt;/strong>. Share the post-mortem widely. Engineering, product, support, leadership. Create space for people to ask questions and add context. Update it based on feedback.&lt;/p>
&lt;p>The goal isn&amp;rsquo;t to make everyone feel bad. It&amp;rsquo;s to extract maximum learning from a painful experience so you don&amp;rsquo;t repeat it.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="building-data-quality-into-your-dna">Building data quality into your DNA&lt;/h3>
&lt;hr>
&lt;p>Everything we&amp;rsquo;ve discussed—constraints, tests, monitoring, culture—needs to become reflexive, not something you remember to do when you have time.&lt;/p>
&lt;p>This means changing how you approach new features. Before you write a single line of code, you ask:&lt;/p>
&lt;ul>
&lt;li>What are the business rules this needs to enforce?&lt;/li>
&lt;li>What invalid states need to be impossible?&lt;/li>
&lt;li>What edge cases could break this?&lt;/li>
&lt;li>How will we verify it works correctly?&lt;/li>
&lt;li>How will we know if it breaks in production?&lt;/li>
&lt;/ul>
&lt;p>This means changing how you do code reviews. You&amp;rsquo;re not just checking for syntax errors or performance issues. You&amp;rsquo;re asking:&lt;/p>
&lt;ul>
&lt;li>What data quality problems could this create?&lt;/li>
&lt;li>What constraints are missing?&lt;/li>
&lt;li>What tests would catch regressions?&lt;/li>
&lt;li>What monitoring would detect problems?&lt;/li>
&lt;/ul>
&lt;p>This means changing how you measure success. You don&amp;rsquo;t just track feature velocity or query performance. You track:&lt;/p>
&lt;ul>
&lt;li>Data quality metrics over time&lt;/li>
&lt;li>Time to detect data quality issues&lt;/li>
&lt;li>Time to resolve data quality issues&lt;/li>
&lt;li>Percentage of issues caught before production&lt;/li>
&lt;li>Customer impact of data quality incidents&lt;/li>
&lt;/ul>
&lt;p>This means changing your hiring and onboarding. New engineers learn about your data quality framework on day one. They see examples of past incidents and what was learned. They understand that data quality isn&amp;rsquo;t someone else&amp;rsquo;s job.&lt;/p>
&lt;p>This means changing your documentation. Your data dictionary doesn&amp;rsquo;t just list column names and types. It documents business rules, constraints, validation logic, known edge cases, and monitoring approaches.&lt;/p>
&lt;p>This means changing your architecture. You design systems with data quality in mind from the start. You choose tools that support validation. You build pipelines that can detect and quarantine bad data. You create feedback loops so quality issues surface quickly.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="when-youre-the-person-who-has-to-fix-this">When you&amp;rsquo;re the person who has to fix this&lt;/h3>
&lt;hr>
&lt;p>Maybe you&amp;rsquo;re reading this because you&amp;rsquo;re dealing with a data quality crisis right now. You&amp;rsquo;ve discovered overlapping bookings, or duplicate transactions, or inconsistent user states, or some other nightmare. Everyone&amp;rsquo;s freaking out. Leadership wants answers. Customers are angry. And somehow, you&amp;rsquo;re the person who has to fix it.&lt;/p>
&lt;p>First, breathe. This isn&amp;rsquo;t the end of the world, even though it feels like it. I&amp;rsquo;ve been there. Most experienced data professionals have been there. You&amp;rsquo;ll get through it.&lt;/p>
&lt;p>Second, assess the scope quickly. How bad is it? How many records affected? How far back does it go? How many customers impacted? You need these numbers to make decisions.&lt;/p>
&lt;p>Third, stabilize before you optimise. Stop the bleeding. Prevent new bad data from entering the system. That might mean temporarily disabling features, adding emergency validation, or manually reviewing submissions. Do what&amp;rsquo;s necessary to prevent the problem from growing.&lt;/p>
&lt;p>Fourth, communicate clearly and often. Tell leadership what you know, what you don&amp;rsquo;t know, and what you&amp;rsquo;re doing about it. Tell affected customers what happened and how you&amp;rsquo;re resolving it. Tell your team what needs to happen and when.&lt;/p>
&lt;p>Fifth, fix the immediate problem before solving the systemic issue. Get the data cleaned up, get customers taken care of, get operations stable. Then you can think about long-term improvements.&lt;/p>
&lt;p>Sixth, document everything. What you found, what you did, what worked, what didn&amp;rsquo;t. This documentation becomes the foundation for your post-mortem and future prevention.&lt;/p>
&lt;p>Finally, remember that mistakes don&amp;rsquo;t define you. How you respond does. Handle this professionally, learn from it, and implement systems to prevent recurrence. That&amp;rsquo;s what separates senior engineers from junior engineers—not avoiding mistakes, but responding to them effectively.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="from-nightmare-to-framework">From nightmare to framework&lt;/h3>
&lt;hr>
&lt;p>The Airbnb double-booking scenario isn&amp;rsquo;t just a cautionary tale. It&amp;rsquo;s a template for thinking about data quality across every domain.&lt;/p>
&lt;p>Maybe you&amp;rsquo;re working on financial transactions where duplicate charges cause customer disputes. Maybe you&amp;rsquo;re dealing with medical records where data inconsistencies impact patient care. Maybe you&amp;rsquo;re managing inventory where overselling creates fulfilment nightmares. Maybe you&amp;rsquo;re building user permissions where invalid states create security vulnerabilities.&lt;/p>
&lt;p>The specifics change, but the framework stays the same:&lt;/p>
&lt;p>Define your business rules clearly. What states should be impossible? What invariants must hold true? Document these explicitly.&lt;/p>
&lt;p>Enforce rules at the database level. Make invalid states impossible to create. Use constraints, triggers, and checks liberally.&lt;/p>
&lt;p>Test your assumptions thoroughly. Cover edge cases. Think adversarially. Verify that your logic actually prevents what you think it prevents.&lt;/p>
&lt;p>Monitor continuously. Don&amp;rsquo;t wait for customers to discover problems. Detect issues before they impact operations.&lt;/p>
&lt;p>Build a culture where data quality matters. Make it everyone&amp;rsquo;s job. Celebrate prevention. Learn from incidents. Invest in foundational work.&lt;/p>
&lt;p>This approach doesn&amp;rsquo;t eliminate all data quality issues. Nothing does. But it dramatically reduces their frequency and impact. And when issues do occur, you&amp;rsquo;ve got the systems to detect and resolve them quickly.&lt;/p>
&lt;p>The investment pays for itself the first time you don&amp;rsquo;t have a data quality crisis at 9 PM on Friday.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="moving-forward-with-confidence">Moving forward with confidence&lt;/h3>
&lt;hr>
&lt;p>Data quality isn&amp;rsquo;t a project you complete. It&amp;rsquo;s a continuous practice that becomes part of how you build systems.&lt;/p>
&lt;p>Start small. Pick your most critical business rule. Implement database constraints. Write tests. Add monitoring. Learn what works in your environment.&lt;/p>
&lt;p>Expand gradually. Apply the same approach to other critical rules. Build momentum. Share successes. Show ROI.&lt;/p>
&lt;p>Make it systematic. Create templates for new features. Establish patterns for common scenarios. Document your approaches.&lt;/p>
&lt;p>Keep improving. When incidents occur, extract lessons. When new patterns emerge, update your framework. When technology changes, adapt your tools.&lt;/p>
&lt;p>And remember—the goal isn&amp;rsquo;t perfection. It&amp;rsquo;s building systems that fail gracefully, detect problems quickly, and learn continuously.&lt;/p>
&lt;p>Your customers won&amp;rsquo;t thank you for preventing crises they never knew about. Your boss might not notice the incidents that didn&amp;rsquo;t happen. But you&amp;rsquo;ll know. You&amp;rsquo;ll sleep better. You&amp;rsquo;ll respond faster when problems occur. You&amp;rsquo;ll build with confidence instead of fear.&lt;/p>
&lt;p>That&amp;rsquo;s worth the investment.&lt;/p>
&lt;p>Now go add some database constraints. Future you will thank you for it—probably at 9 PM on a Friday when you&amp;rsquo;re enjoying that second beer instead of fielding angry support calls.&lt;/p>
&lt;/br>
&lt;/br>
&lt;p>Reference to code:
&lt;a href="https://github.com/krshillman/ghost_in_the_data_data_quality/blob/main/examples/2025-11-22-data-quality.sql" target="_blank" rel="noopener">https://github.com/krshillman/ghost_in_the_data_data_quality/blob/main/examples/2025-11-22-data-quality.sql&lt;/a>&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Quality</category><category>SQL</category><category>Database Design</category><category>Data Validation</category><category>Testing</category><category>Data Engineering</category><category>Production Issues</category></item><item><title>Balancing Data Accessibility and Privacy in Financial Services</title><link>https://ghostinthedata.info/posts/2025/2025-11-21-data-accessibility-and-privacy/</link><pubDate>Fri, 21 Nov 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-11-21-data-accessibility-and-privacy/</guid><author>Chris Hillman</author><description>Explore the complex interplay between data accessibility and privacy in banking, with practical strategies for implementing effective anonymisation policies, retention protocols, and data minimisation techniques that maintain regulatory compliance without sacrificing analytical value.</description><content:encoded>&lt;h2 id="the-data-tightrope-where-accessibility-meets-privacy">The Data Tightrope: Where Accessibility Meets Privacy&lt;/h2>
&lt;p>Let&amp;rsquo;s face it—in today&amp;rsquo;s data landscape, data is simultaneously your most valuable asset and your biggest potential liability.&lt;/p>
&lt;p>Finding that sweet spot where data remains accessible enough to drive business decisions while being locked down enough to satisfy privacy regulations. It&amp;rsquo;s not just about ticking compliance boxes—it&amp;rsquo;s about maintaining customer trust while still extracting every bit of analytical value from your data assets.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-four-tier-data-storage-reality">The Four-Tier Data Storage Reality&lt;/h2>
&lt;hr>
&lt;p>Most financial institutions operate with multiple data storage options, each with its own tradeoffs. Here&amp;rsquo;s how I&amp;rsquo;ve seen it play out in practice:&lt;/p>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>Glacier/Cold Storage&lt;/strong>: Dirt cheap but painfully slow—we&amp;rsquo;re talking potentially days to retrieve data. Perfect for those regulatory archival requirements where immediacy isn&amp;rsquo;t essential.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Iceberg Relational Databases&lt;/strong>: Your traditional SQL databases offering a balance of speed and cost. This is where most of your operational data lives.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>In-memory Caching Solutions&lt;/strong> (Redis, Memcached): Lightning-fast but expensive. Reserve these for those mission-critical nanosecond accesses.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Hybrid Solutions&lt;/strong> (like Druid): Offering column-store capabilities with better performance than traditional databases but lower cost than pure in-memory solutions.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;p>Each tier has its place in your architecture, but—and this is crucial—each needs its own privacy controls. I&amp;rsquo;ve seen too many organisations focus all their privacy efforts on relational databases while neglecting controls on their cold storage or caching layers. Remember Facebook&amp;rsquo;s Cambridge Analytica debacle? That wasn&amp;rsquo;t a failure of their primary database privacy controls; it was a failure in their broader data ecosystem.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="anonymisation-vs-pseudonymization-more-than-just-semantics">Anonymisation vs. Pseudonymization: More Than Just Semantics&lt;/h2>
&lt;hr>
&lt;p>Here&amp;rsquo;s where things get interesting—and where I&amp;rsquo;ve seen many companies stumble. The distinction between anonymisation and pseudonymization isn&amp;rsquo;t just academic; it fundamentally affects how you can use your data and what regulatory requirements apply.&lt;/p>
&lt;h3 id="anonymisation-the-one-way-trip">Anonymisation: The One-Way Trip&lt;/h3>
&lt;p>Truly anonymised data is aggregated to the point where it cannot be traced back to an individual. Take our retail banking metrics dashboard showing &amp;ldquo;170 million active accounts in Australia&amp;rdquo;—valuable for trend analysis but impossible to connect to any specific customer.&lt;/p>
&lt;p>But anonymisation has limitations. I learned this the hard way when we created what we thought was an anonymised dashboard. By slicing the data across too many dimensions (location, account type, account age, balance tier), we inadvertently created a situation where some data points could be traced back to specific high-net-worth individuals. The fix? Adding minimum thresholds for each dimensional cut to prevent de-anonymisation.&lt;/p>
&lt;h3 id="pseudonymization-maintaining-the-bridge">Pseudonymization: Maintaining the Bridge&lt;/h3>
&lt;p>Pseudonymization differs fundamentally in that it maintains a separate mapping table linking the scrubbed identifiers to the original data. Replace customer IDs with randomly generated tokens while maintaining a secured mapping table accessible only to authorised personnel.&lt;/p>
&lt;p>This approach gives you more analytical flexibility but requires rigorous governance of that mapping table. The moment that mapping table becomes compromised, all pseudonymized data effectively becomes identified again.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="practical-implementation">Practical Implementation&lt;/h2>
&lt;hr>
&lt;p>Let me walk you through a example. We created an anonymised version of sport player data as a proxy for customer financial data. Here&amp;rsquo;s how we structured it:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Player Name&lt;/strong> (representing our customer identifier)&lt;/li>
&lt;li>&lt;strong>Anonymised ID&lt;/strong> (a bigint with no connection to the original ID)&lt;/li>
&lt;li>&lt;strong>Last Active Season&lt;/strong> (similar to &amp;ldquo;last transaction date&amp;rdquo;)&lt;/li>
&lt;li>&lt;strong>Current Season&lt;/strong> (representing the present time benchmark)&lt;/li>
&lt;/ul>
&lt;p>The anonymisation policy stated that players inactive for five years or more would be completely anonymised. This mirrors our policy where customer data achieves a different privacy status after extended inactivity.&lt;/p>
&lt;p>Here&amp;rsquo;s what the implementation looked like:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Create the anonymized table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">CREATE&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> anonymized_players (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> anonymized_player_ID INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> start_position VARCHAR,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> points DOUBLE,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> season INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> game_ID INT,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">PRIMARY&lt;/span> &lt;span style="color:#66d9ef">KEY&lt;/span> (anonymized_player_ID)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Partition for performance
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">ALTER&lt;/span> &lt;span style="color:#66d9ef">TABLE&lt;/span> anonymized_players
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PARTITION &lt;span style="color:#66d9ef">BY&lt;/span> RANGE (season);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Identify players for anonymization
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">WITH&lt;/span> combined &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">-- Join original and anonymized tables
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> p.player_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> a.anonymized_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> p.last_active_season,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">EXTRACT&lt;/span>(&lt;span style="color:#66d9ef">YEAR&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> current_season
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> players p
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">LEFT&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> anonymized_players a &lt;span style="color:#66d9ef">ON&lt;/span> p.player_id &lt;span style="color:#f92672">=&lt;/span> a.original_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Apply the anonymization policy
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">INSERT&lt;/span> &lt;span style="color:#66d9ef">INTO&lt;/span> anonymized_players (player_name, anonymized_id)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> player_name, anonymized_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> combined
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> last_active_season &lt;span style="color:#f92672">&amp;lt;&lt;/span> current_season &lt;span style="color:#f92672">-&lt;/span> &lt;span style="color:#ae81ff">5&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The real magic happens with those time-based policies. We have to carefully track player (customer) returns and maintain consistent anonymised IDs. Think about Michael Jordan&amp;rsquo;s comeback—if he retired in 1998 and returned in 2001, he&amp;rsquo;d maintain the same anonymised ID because he didn&amp;rsquo;t exceed our five-year threshold. If he&amp;rsquo;d stayed retired for six years, he&amp;rsquo;d receive a completely new anonymised ID upon return.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-sliding-window-challenge">The Sliding Window Challenge&lt;/h2>
&lt;hr>
&lt;p>This sliding window concept creates fascinating technical challenges. We need to:&lt;/p>
&lt;ol>
&lt;li>Maintain rolling retention policies that automatically anonymise data after the specified period&lt;/li>
&lt;li>Handle returns gracefully without compromising privacy&lt;/li>
&lt;li>Ensure consistency across all data tiers&lt;/li>
&lt;/ol>
&lt;p>One approach we&amp;rsquo;ve found effective is using Iceberg tables with time travel capabilities. This allows us to maintain a view of our table history while only serving the current state to clients.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The simplified approach to identifying last partition update
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> partition_key,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">MAX&lt;/span>(committed_at) &lt;span style="color:#66d9ef">as&lt;/span> last_update
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> iceberg_partitions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> partition_key &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;season=2003&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">GROUP&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> partition_key;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h2 id="retention-policies-logical-vs-creation-date">Retention Policies: Logical vs. Creation Date&lt;/h2>
&lt;hr>
&lt;p>The retention policy headache doesn&amp;rsquo;t end with anonymisation. You also need to decide whether to base your retention on:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Logical Date&lt;/strong>: Tied to when the event actually occurred&lt;/li>
&lt;li>&lt;strong>Creation Date&lt;/strong>: When the data was loaded into your warehouse&lt;/li>
&lt;/ol>
&lt;p>This seemingly minor distinction creates massive downstream implications. Let me illustrate:&lt;/p>
&lt;p>Imagine you have a 50-year-old partition (data from 1975). With a logical date retention policy, that partition might be deleted tomorrow because it exceeds your retention window. With a creation date policy, if you just loaded that data last week, it would be retained regardless of how old the actual data is.&lt;/p>
&lt;p>My recommendation? Use logical date retention for operational data and creation date for historical archives. This gives you the best of both worlds—efficient purging of operational data while maintaining valuable historical information.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="implementing-data-minimisation-in-practice">Implementing Data Minimisation in Practice&lt;/h2>
&lt;hr>
&lt;p>Data minimisation isn&amp;rsquo;t just theoretical—here&amp;rsquo;s how we&amp;rsquo;ve implemented it:&lt;/p>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>Identify Low-Value Data&lt;/strong>: We analyse query patterns to identify rarely accessed data that can be archived or removed.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Regular Audits&lt;/strong>: Schedule quarterly reviews of storage and retention policies.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Efficient Data Modelling&lt;/strong>: Design schemas to minimise redundancy while maximising query performance.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Compression Strategies&lt;/strong>: Apply appropriate compression techniques based on access patterns.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;p>Some organisations, like Facebook, have sophisticated tools like &amp;ldquo;Cockpit&amp;rdquo; to analyse data usage patterns. While most of us don&amp;rsquo;t have such advanced tooling, we can implement similar practices at a smaller scale:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Simple query to identify rarely used tables
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">table_name&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> last_query_date,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> DATEDIFF(&lt;span style="color:#66d9ef">day&lt;/span>, last_query_date, &lt;span style="color:#66d9ef">CURRENT_DATE&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> days_since_last_access
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> table_access_logs
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> days_since_last_access &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The beauty of this approach is that it helps us maintain compliance with data minimisation principles in privacy regulations while also improving our operational efficiency.&lt;/p>
&lt;h2 id="the-special-case-of-active-investigations">The Special Case of Active Investigations&lt;/h2>
&lt;hr>
&lt;p>Data privacy isn&amp;rsquo;t always straightforward. What happens when a customer requests deletion, but their data is needed for an active fraud investigation? These edge cases require careful consideration and clear policies.&lt;/p>
&lt;p>We&amp;rsquo;ve implemented a special retention flag in our system that can override standard deletion processes when legal requirements necessitate it. This ensures we maintain compliance with both privacy regulations and legal obligations.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Simplified example of deletion logic with legal hold exceptions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">DELETE&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> customer_data
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#f92672">?&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> &lt;span style="color:#66d9ef">NOT&lt;/span> &lt;span style="color:#66d9ef">EXISTS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> legal_holds
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> customer_id &lt;span style="color:#f92672">=&lt;/span> customer_data.customer_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> is_active &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> );
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h2 id="wrapping-up-the-human-element">Wrapping Up: The Human Element&lt;/h2>
&lt;hr>
&lt;p>At the end of the day, data management isn&amp;rsquo;t just a technical challenge—it&amp;rsquo;s a human one. The electricity consumed by our data operations, the privacy of our customers, and the insights we generate all have real-world impacts on people&amp;rsquo;s lives.&lt;/p>
&lt;p>As data professionals in the financial sector, we have a responsibility to make conscious choices about how we manage data. By prioritising data efficiency, respecting privacy, and fostering a culture of awareness within our organisations, we can contribute to a more sustainable and ethical data ecosystem.&lt;/p>
&lt;p>I&amp;rsquo;d love to hear about your experiences balancing data accessibility and privacy in your organisation. What challenges have you faced, and what solutions have you implemented? Let&amp;rsquo;s continue this conversation in the comments below.&lt;/p>
&lt;p>Remember, every bit of data we manage represents a real person with real privacy concerns. Let&amp;rsquo;s never lose sight of that as we navigate the complex world of data engineering in financial services.&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>Data Privacy</category><category>Anonymisation</category><category>Retention Policies</category><category>Banking</category><category>Data Minimisation</category><category>GDPR</category><category>Data Governance</category></item><item><title>Continuous Integration for Data Teams: Beyond the Buzzwords</title><link>https://ghostinthedata.info/posts/2025/2025-06-28-continuous-integration-data-teams/</link><pubDate>Sat, 28 Jun 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-06-28-continuous-integration-data-teams/</guid><author>Chris Hillman</author><description>A practical guide to implementing CI/CD for analytics engineering teams. From dbt Cloud's Slim CI to Datafold's data validation - what actually works in production and why your data quality depends on it.</description><content:encoded>&lt;h2 id="the-day-everything-broke-and-how-ci-could-have-saved-us">The Day Everything Broke (And How CI Could Have Saved Us)&lt;/h2>
&lt;p>Picture this: It&amp;rsquo;s 9 AM on a Monday, and your Slack is exploding. The executive dashboard is showing impossible numbers. Customer support is fielding complaints about incorrect billing amounts. The marketing team is questioning why their conversion metrics suddenly dropped to zero.&lt;/p>
&lt;p>You trace it back to a seemingly innocent change you merged Friday afternoon—a simple column rename that seemed harmless enough. But that &amp;ldquo;harmless&amp;rdquo; change cascaded through your entire data pipeline, breaking downstream models, dashboards, and automated reports.&lt;/p>
&lt;p>Sound familiar? If you&amp;rsquo;ve worked in data for more than a few months, you&amp;rsquo;ve probably lived through some version of this nightmare. The good news is that it&amp;rsquo;s entirely preventable with proper continuous integration practices.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="what-ci-actually-means-for-data-teams">What CI Actually Means for Data Teams&lt;/h2>
&lt;hr>
&lt;p>Let&amp;rsquo;s start with the basics. Continuous Integration (CI) in the data world isn&amp;rsquo;t just about running tests—it&amp;rsquo;s about creating a safety net that catches problems before they reach production.&lt;/p>
&lt;p>Traditional software development has had this figured out for years. You write code, you write tests, you run those tests automatically before merging changes. If something breaks, the merge is blocked until you fix it.&lt;/p>
&lt;p>Data teams have been slower to adopt these practices, partly because our &amp;ldquo;tests&amp;rdquo; are different. We&amp;rsquo;re not just checking if code compiles—we&amp;rsquo;re validating that data transformations produce the expected results, that schema changes don&amp;rsquo;t break downstream dependencies, and that performance doesn&amp;rsquo;t degrade.&lt;/p>
&lt;p>The core principle remains the same: validate changes in an isolated environment before they affect production systems. But the implementation looks different when you&amp;rsquo;re dealing with data pipelines instead of web applications.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-github-foundation">The GitHub Foundation&lt;/h2>
&lt;hr>
&lt;p>Before we talk about specific tools, let&amp;rsquo;s address the elephant in the room: version control. If you&amp;rsquo;re not using Git for your data transformation code, stop reading this article and go set that up first. Seriously.&lt;/p>
&lt;p>GitHub (or GitLab, or Bitbucket) isn&amp;rsquo;t just a nice-to-have for data teams—it&amp;rsquo;s the foundation that makes everything else possible. Here&amp;rsquo;s why:&lt;/p>
&lt;p>&lt;strong>Branching Strategy&lt;/strong>: Different team members can work on different features simultaneously without stepping on each other&amp;rsquo;s toes. You create a branch, make your changes, test them, and then merge back to main.&lt;/p>
&lt;p>&lt;strong>Pull Request Workflow&lt;/strong>: This is where the magic happens. Before any code reaches production, it goes through a review process. Other team members can see exactly what changed, ask questions, and suggest improvements.&lt;/p>
&lt;p>&lt;strong>Webhook Integration&lt;/strong>: This is what enables automated CI. When you open a pull request, GitHub can trigger automated processes to validate your changes.&lt;/p>
&lt;p>&lt;strong>History and Rollbacks&lt;/strong>: When something does go wrong (and it will), you can quickly identify what changed and roll back to a known good state.&lt;/p>
&lt;p>I&amp;rsquo;ve seen teams try to implement CI without proper version control, and it&amp;rsquo;s like trying to build a house without a foundation. The tools we&amp;rsquo;ll discuss next all assume you have a solid Git workflow in place.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="dbt-cloud-the-integrated-approach">dbt Cloud: The Integrated Approach&lt;/h2>
&lt;hr>
&lt;p>If you&amp;rsquo;re using dbt (and if you&amp;rsquo;re doing analytics engineering, you probably should be), dbt Cloud offers the most seamless CI experience I&amp;rsquo;ve encountered.&lt;/p>
&lt;p>Here&amp;rsquo;s how it works: when you open a pull request in GitHub, dbt Cloud automatically detects the change and spins up a Slim CI job. This job builds only the models that have changed or depend on changed models—not your entire project.&lt;/p>
&lt;p>The magic happens in the temporary schema creation. dbt Cloud creates a schema with the naming pattern &lt;code>dbt_cloud_pr_&amp;lt;job_id&amp;gt;_&amp;lt;pr_id&amp;gt;&lt;/code>. This gives you a completely isolated environment to test your changes without affecting production data.&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Your production models live here
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>analytics.dim_customers
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Your PR changes get built here
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>dbt_cloud_pr_123_456.dim_customers
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This isolation is crucial. You can run queries against your PR schema to validate that your changes produce the expected results. You can even point BI tools at the PR schema to see how dashboard changes will look.&lt;/p>
&lt;p>The workflow is beautifully simple:&lt;/p>
&lt;ol>
&lt;li>Make changes in a feature branch&lt;/li>
&lt;li>Open a pull request&lt;/li>
&lt;li>dbt Cloud automatically builds and tests your changes&lt;/li>
&lt;li>Review the results directly in GitHub&lt;/li>
&lt;li>Merge when everything looks good&lt;/li>
&lt;li>dbt Cloud automatically cleans up the temporary schema&lt;/li>
&lt;/ol>
&lt;p>What I particularly appreciate about dbt Cloud&amp;rsquo;s approach is the concurrency. Multiple team members can have active pull requests simultaneously, each with their own isolated environment. No more waiting for someone else&amp;rsquo;s testing to finish.&lt;/p>
&lt;p>The status checks integrate directly into GitHub&amp;rsquo;s interface. You can see at a glance whether your CI run passed or failed, and drill down into the details if something went wrong. No context switching between tools.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="datafold-beyond-code-validation">Datafold: Beyond Code Validation&lt;/h2>
&lt;hr>
&lt;p>dbt Cloud&amp;rsquo;s CI is excellent for validating that your code runs successfully, but what about the data itself? This is where Datafold shines.&lt;/p>
&lt;p>While dbt Cloud tells you &amp;ldquo;your models built successfully,&amp;rdquo; Datafold tells you &amp;ldquo;here&amp;rsquo;s exactly how your data changed.&amp;rdquo; It compares the data in your PR schema against production and highlights differences in:&lt;/p>
&lt;ul>
&lt;li>Row counts&lt;/li>
&lt;li>Column distributions&lt;/li>
&lt;li>Data types&lt;/li>
&lt;li>Null percentages&lt;/li>
&lt;li>Unique value counts&lt;/li>
&lt;li>Statistical summaries&lt;/li>
&lt;/ul>
&lt;p>But Datafold goes beyond simple data profiling. It performs actual data diffs, showing you exactly which rows changed and how. This is incredibly valuable when you&amp;rsquo;re making complex transformations and want to understand the downstream impact.&lt;/p>
&lt;p>Here&amp;rsquo;s a real example: you&amp;rsquo;re updating a customer segmentation model to fix a bug in the logic. dbt Cloud confirms that the model builds successfully, but Datafold shows you that 15% of customers moved to different segments. That&amp;rsquo;s exactly the kind of insight you need to communicate the impact to stakeholders.&lt;/p>
&lt;p>The external dependency checking is another killer feature. Datafold can analyse how your changes will affect downstream tools like Mode, Looker, or Tableau. It parses the SQL in your dashboards and reports to identify which ones might be impacted by your changes.&lt;/p>
&lt;p>Imagine knowing before you merge that your change will break three executive dashboards and two automated reports. That&amp;rsquo;s the kind of information that prevents those Monday morning fire drills.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-data-quality-connection">The Data Quality Connection&lt;/h2>
&lt;hr>
&lt;p>Here&amp;rsquo;s what I&amp;rsquo;ve learned after years of implementing CI for data teams: it&amp;rsquo;s not really about the tools. It&amp;rsquo;s about creating a culture where data quality is everyone&amp;rsquo;s responsibility, not just something you check at the end.&lt;/p>
&lt;p>Traditional data quality approaches are reactive. You build your pipeline, deploy it to production, and then add monitoring to catch problems after they happen. CI flips this model—you catch problems before they reach production.&lt;/p>
&lt;p>This shift in mindset is profound. Instead of data quality being a bottleneck that slows down development, it becomes a natural part of the development process. You make a change, validate it automatically, and merge with confidence.&lt;/p>
&lt;p>The business impact is immediate. Stakeholders get the changes they need when they need them, without sacrificing reliability. No more choosing between speed and quality—you can have both.&lt;/p>
&lt;p>I&amp;rsquo;ve seen teams reduce their data incidents by 80% just by implementing basic CI practices. The time saved on firefighting more than pays for the initial setup effort.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="practical-implementation-strategy">Practical Implementation Strategy&lt;/h2>
&lt;hr>
&lt;p>If you&amp;rsquo;re convinced that CI is worth implementing (and you should be), here&amp;rsquo;s how to approach it practically:&lt;/p>
&lt;p>&lt;strong>Start Small&lt;/strong>: Don&amp;rsquo;t try to implement everything at once. Begin with basic dbt Cloud CI for your most critical models. Get comfortable with the workflow before adding more sophisticated validation.&lt;/p>
&lt;p>&lt;strong>Focus on High-Impact Models&lt;/strong>: Identify the models that, if they break, cause the most pain. These are your tier-1 models that feed executive dashboards, customer-facing features, or automated processes. Implement CI for these first.&lt;/p>
&lt;p>&lt;strong>Establish Clear Standards&lt;/strong>: Define what &amp;ldquo;passing CI&amp;rdquo; means for your team. Is it enough that models build successfully? Do you require certain test coverage? What about performance thresholds?&lt;/p>
&lt;p>&lt;strong>Train Your Team&lt;/strong>: CI is only effective if everyone uses it consistently. Make sure your team understands not just how to use the tools, but why they matter.&lt;/p>
&lt;p>&lt;strong>Iterate and Improve&lt;/strong>: Start with basic validation and gradually add more sophisticated checks. Monitor what types of issues slip through and adjust your CI process accordingly.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="common-pitfalls-and-how-to-avoid-them">Common Pitfalls and How to Avoid Them&lt;/h2>
&lt;hr>
&lt;p>I&amp;rsquo;ve seen teams struggle with CI implementation, usually for predictable reasons:&lt;/p>
&lt;p>&lt;strong>Over-Engineering&lt;/strong>: Don&amp;rsquo;t build a complex CI system from day one. Start with the basics and add complexity only when you need it.&lt;/p>
&lt;p>&lt;strong>Ignoring Performance&lt;/strong>: CI jobs that take 30 minutes to run won&amp;rsquo;t get used. Optimise for speed, even if it means running fewer tests initially.&lt;/p>
&lt;p>&lt;strong>Weak Standards&lt;/strong>: If your CI checks are too lenient, they won&amp;rsquo;t catch real problems. If they&amp;rsquo;re too strict, developers will find ways to bypass them. Find the right balance.&lt;/p>
&lt;p>&lt;strong>Poor Communication&lt;/strong>: Make sure your team understands what CI is checking and why. If people don&amp;rsquo;t trust the process, they won&amp;rsquo;t follow it.&lt;/p>
&lt;p>&lt;strong>Neglecting Maintenance&lt;/strong>: CI systems need ongoing attention. Tests become outdated, performance degrades, and new edge cases emerge. Plan for regular maintenance.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-roi-of-data-ci">The ROI of Data CI&lt;/h2>
&lt;hr>
&lt;p>Let&amp;rsquo;s talk numbers. Implementing CI for data teams requires investment—time to set up the tools, train the team, and adjust workflows. But the return on investment is typically dramatic.&lt;/p>
&lt;p>Consider the cost of a single data incident: developer time to investigate and fix the issue, stakeholder time dealing with incorrect information, potential business impact from wrong decisions, and the opportunity cost of not working on new features.&lt;/p>
&lt;p>I&amp;rsquo;ve seen single incidents cost organisations tens of thousands of dollars in lost productivity and business impact. A robust CI system that prevents even one major incident per quarter easily pays for itself.&lt;/p>
&lt;p>But the benefits go beyond incident prevention. CI enables faster development cycles because developers have confidence in their changes. It reduces the cognitive load of remembering to run tests manually. It creates better documentation through the PR review process.&lt;/p>
&lt;p>Most importantly, it shifts the team&amp;rsquo;s focus from reactive firefighting to proactive development. Instead of spending time fixing problems, you spend time building new capabilities.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="looking-ahead-the-future-of-data-ci">Looking Ahead: The Future of Data CI&lt;/h2>
&lt;hr>
&lt;p>The CI landscape for data teams is evolving rapidly. We&amp;rsquo;re seeing more sophisticated data validation tools, better integration between different parts of the stack, and AI-powered anomaly detection.&lt;/p>
&lt;p>But the fundamentals remain the same: validate changes before they reach production, automate as much as possible, and create feedback loops that help teams learn and improve.&lt;/p>
&lt;p>The teams that embrace these practices now will have a significant advantage as the data landscape becomes more complex. They&amp;rsquo;ll be able to move faster, with higher quality, and with greater confidence.&lt;/p>
&lt;p>If you&amp;rsquo;re not already implementing CI for your data team, start today. Begin with basic dbt Cloud integration, establish a solid Git workflow, and gradually add more sophisticated validation. Your future self (and your stakeholders) will thank you.&lt;/p>
&lt;p>The goal isn&amp;rsquo;t perfection—it&amp;rsquo;s progress. Every problem you catch in CI is a problem that doesn&amp;rsquo;t wake you up at 2 AM. Every automated check is time saved for more valuable work. Every successful deployment is proof that you can move fast without breaking things.&lt;/p>
&lt;p>That&amp;rsquo;s the promise of continuous integration for data teams: the ability to deliver value quickly and reliably, without sacrificing quality or sanity. In a world where data drives increasingly critical business decisions, that&amp;rsquo;s not just nice to have—it&amp;rsquo;s essential.&lt;/p></content:encoded><category>Data Engineering</category><category>Data Quality &amp; Governance</category><category>CI/CD</category><category>Data Quality</category><category>dbt</category><category>DevOps</category><category>Data Engineering</category><category>GitHub</category><category>Datafold</category><category>Data Validation</category></item></channel></rss>