<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>AI on Ghost in the data</title><link>https://ghostinthedata.info/categories/ai/</link><description>Ghost in the data</description><generator>Hugo -- gohugo.io</generator><language>en</language><copyright>Ghost in the data</copyright><lastBuildDate>Sun, 15 Nov 2026 09:00:00 +1100</lastBuildDate><atom:link href="https://ghostinthedata.info/categories/ai/index.xml" rel="self" type="application/rss+xml"/><item><title>Very Boring but Very Responsible</title><link>https://ghostinthedata.info/posts/2026/2026-11-15-supervisory-engineering-data-managers/</link><pubDate>Sun, 15 Nov 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-11-15-supervisory-engineering-data-managers/</guid><author>Chris Hillman</author><description>Annie Vella says the job is now directing, evaluating and correcting agents. What that means when the output is a pipeline and the person accountable is you.</description><content:encoded>&lt;p>In 1983 Lisanne Bainbridge wrote a short paper called &lt;em>Ironies of Automation&lt;/em>. The irony she cared most about was that the more capable the automation, the less practice the human operator gets, and the more important it becomes that the human can take over when the automation fails. A job that&amp;rsquo;s mostly monitoring, she wrote, is &amp;ldquo;very boring but very responsible&amp;rdquo;.&lt;/p>
&lt;p>She was writing about chemical plants and cockpits. Annie Vella has spent this year arguing that she was also writing about us.&lt;/p>
&lt;p>Vella&amp;rsquo;s post on &lt;a href="https://annievella.com/posts/supervisory-engineering-work/" target="_blank" rel="noopener">supervisory engineering work&lt;/a> is worth reading in full. It&amp;rsquo;s the clearest framing I&amp;rsquo;ve read of what has changed in software engineering since agents started writing code. Her research found what she calls a creation-to-verification shift, with engineers spending less time producing code and more time testing and reviewing it. She boils the new job down to three verbs: directing the agent, evaluating what it produced, and correcting it when it&amp;rsquo;s wrong.&lt;/p>
&lt;p>I&amp;rsquo;ve been reading that post with a specific question in mind: what changes when the thing being supervised isn&amp;rsquo;t an application but a pipeline, and when the supervisor is also the manager who signs off on it?&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="the-shift-happened-faster-than-the-job-descriptions">The shift happened faster than the job descriptions&lt;/h3>
&lt;/br>
&lt;p>An engineer picks up a ticket: add a new source to the ingestion layer, model it into a dimension, wire up the tests. Eighteen months ago that was two or three days of writing SQL and dbt YAML and arguing with the orchestrator. Now a reasonable chunk of it is a prompt and a pattern pulled from the &lt;a href="https://ghostinthedata.info/posts/2026/2026-06-27-pattern-bank/">pattern bank&lt;/a>, and an agent turns that into a pull request that is, on first read, pretty good.&lt;/p>
&lt;p>I recently sat through what happens when nobody gets past the first read. Someone was given a task, and the instructions looked like they&amp;rsquo;d been pasted into an AI prompt word for word. The output was never tested against the ask, and it wasn&amp;rsquo;t what we expected.&lt;/p>
&lt;p>That led to several (yes, several) follow-up meetings to agree on what we actually wanted, how it worked, and how we&amp;rsquo;d know it was working. Most of it came down to testing the output. The frustrating part is that we did that testing in review meetings, because none of it was baked into the work itself.&lt;/p>
&lt;p>I&amp;rsquo;m seeing this more and more with AI. Anyone can write a prompt. The job is checking that the output actually behaves correctly, and if you&amp;rsquo;re not doing that, you&amp;rsquo;re pushing the burden onto everyone else to do it for you.&lt;/p>
&lt;p>Whether people have noticed or not, the engineer&amp;rsquo;s day has moved from the left of the feedback loop to the right of it. Less time with a blank editor, more time with a diff. And the diff is longer than anything they&amp;rsquo;d have written themselves in the same time, which means the review is larger than any review they&amp;rsquo;d have done before. Call it the review burden.&lt;/p>
&lt;p>Vella cites a 1995 study by Endsley and Kiris where workload didn&amp;rsquo;t appear to drop even at full automation, possibly because monitoring load went up (&amp;ldquo;sound familiar?&amp;rdquo; she asks). And she&amp;rsquo;s blunt that &amp;ldquo;what an AI agent says it did and what it actually did may not be the same thing.&amp;rdquo; You can&amp;rsquo;t skip the check.&lt;/p>
&lt;p>I &lt;a href="https://ghostinthedata.info/posts/2026/2026-07-11-code-reviews-just-do-it/">wrote in July&lt;/a> that data engineers should just do code reviews, because the evidence says reading code finds roughly twice as many defects per hour as testing does. I stand by it. But that post assumed the author was a person who could be asked why they&amp;rsquo;d done something. The author is now, often, a process that will confidently answer any question you ask it, including the ones where it&amp;rsquo;s wrong.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="pipelines-fail-differently-and-that-changes-the-supervision">Pipelines fail differently, and that changes the supervision&lt;/h3>
&lt;/br>
&lt;p>Vella&amp;rsquo;s framework comes from human factors research: Sheridan&amp;rsquo;s supervisory control, Endsley&amp;rsquo;s situation awareness, the aviation studies that moved the blame from &amp;ldquo;pilot error&amp;rdquo; to &amp;ldquo;design error&amp;rdquo;. It transfers to data work well, but there&amp;rsquo;s one property of pipelines that makes the supervision harder than it is for application code.&lt;/p>
&lt;p>A pipeline&amp;rsquo;s output is data, and data doesn&amp;rsquo;t crash.&lt;/p>
&lt;p>If an agent writes a web handler with a bug, something usually goes red, like a failing test or a 500 in production. If an agent writes a dbt model with a subtly wrong join, typically nothing goes red. The model builds. The tests pass, because the tests were written by the same agent against the same misunderstanding. The row count looks plausible. The number then lands in a dashboard, and the first signal that anything is wrong is an analyst three weeks later asking why a metric doubled.&lt;/p>
&lt;p>That&amp;rsquo;s the same failure as the drift I described in &lt;a href="https://ghostinthedata.info/posts/2026/2026-05-09-broken-window-theory/">The Broken Window&lt;/a>. Plausible output is the enemy. And an agent is a machine for producing plausible output.&lt;/p>
&lt;p>So when Vella writes about situation awareness, about the operator perceiving what changed but understanding its meaning less well, I think about a reviewer looking at forty lines of generated SQL that reads cleanly and still can&amp;rsquo;t tell you whether the fan-out on that left join is intended. Reading is more passive than writing. You can approve a diff you don&amp;rsquo;t actually understand, and the pipeline will still run.&lt;/p>
&lt;p>Here&amp;rsquo;s how those three verbs play out on a data team.&lt;/p>
&lt;p>&lt;strong>Directing.&lt;/strong> This is where most of the payoff is, and it&amp;rsquo;s where teams tend to under-invest. Directing an agent well on data work means giving it the grain, the keys, the known traps of the source, and the house pattern for the thing it&amp;rsquo;s building. If that context lives in people&amp;rsquo;s heads, the agent guesses, and the guess is usually generic.&lt;/p>
&lt;p>The reason I built &lt;a href="https://ghostinthedata.info/posts/2026/2026-06-14-ai-ghost-skills-ai-agents/">Ghost Skills&lt;/a> was that I was tired of re-explaining the same six things about SCD2 to every agent session. The pattern bank is the same idea for humans. Directing is mostly &lt;a href="https://ghostinthedata.info/posts/2026/2026-01-31-context-engineering/">context engineering&lt;/a>, and context engineering is mostly writing down what your team already knows. A lot of the MCP integration work happening now can help here too (MCP is the protocol that lets an agent pull in your docs and tools directly), but only if the documentation behind it is actually maintained.&lt;/p>
&lt;p>If you&amp;rsquo;re taking a data team down the AI path, get these principles written down and hardened first. They&amp;rsquo;re your primer.&lt;/p>
&lt;p>&lt;strong>Evaluating.&lt;/strong> For pipelines, evaluation can&amp;rsquo;t stop at reading the code. It has to include running the reconciliation: does the target tie out to source, do the row counts and the sums and the distinct keys match, is the overlap test on the dimension clean. I&amp;rsquo;ve started treating a reconciliation query as the mandatory second half of any agent-produced PR, and I&amp;rsquo;d rather see a short reconciliation than a long explanation. The agent can write the reconciliation too, which is fine, as long as a human reads the result and not just the code.&lt;/p>
&lt;p>Linus Torvalds showed the limits of agent-written tests recently. He used Claude Code on a &lt;a href="https://neurohive.io/en/news/linus-torvalds-claude-code/" target="_blank" rel="noopener">page-cache problem&lt;/a> that kernel developers had been stuck on for about two and a half years. He directed the work and ran the benchmarks on his own machine, but the agent wrote the code &lt;em>and&lt;/em> the tests. Four hours after the patches went out, another kernel developer found a race condition those tests had missed. Torvalds hadn&amp;rsquo;t caught it either.&lt;/p>
&lt;p>&lt;strong>Correcting.&lt;/strong> Correcting an agent is easy; you tell it what&amp;rsquo;s wrong and it fixes it. Correcting the &lt;em>pattern&lt;/em> that produced the error is the manager&amp;rsquo;s job, and it means feeding the correction back into the skill, the template, the &lt;a href="https://ghostinthedata.info/posts/2026/2026-04-11-questions-that-predict-pipeline-disasters/">pre-mortem checklist&lt;/a>, so the next session doesn&amp;rsquo;t make the same mistake. If you only correct the output, you&amp;rsquo;re on a treadmill. It&amp;rsquo;s the difference between fixing a pipeline and fixing the factory that makes pipelines.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="the-managers-version-of-bainbridges-irony">The manager&amp;rsquo;s version of Bainbridge&amp;rsquo;s irony&lt;/h3>
&lt;/br>
&lt;p>Vella&amp;rsquo;s piece is addressed to engineers. There&amp;rsquo;s a second audience she doesn&amp;rsquo;t quite address, which is the people accountable for the engineers&amp;rsquo; work, and that&amp;rsquo;s where Bainbridge&amp;rsquo;s irony hits hardest.&lt;/p>
&lt;p>As a manager, I was already a supervisor. I didn&amp;rsquo;t write most of the code my team shipped before agents turned up either. What I had was a chain of accountability I understood: an engineer wrote it, a second engineer reviewed it, both could explain it, and if something went wrong I knew who&amp;rsquo;d learned what.&lt;/p>
&lt;p>That chain now has a link in it that can&amp;rsquo;t be held accountable, doesn&amp;rsquo;t remember last week, and will produce a thorough-sounding justification for anything it did. The engineer who directed it is accountable. But as Vella asks, &amp;ldquo;what does it really mean to hold engineers accountable for code they&amp;rsquo;re less and less likely to have written, or even read?&amp;rdquo;&lt;/p>
&lt;p>Here&amp;rsquo;s what&amp;rsquo;s helped on my team.&lt;/p>
&lt;p>&lt;strong>The reviewer must be able to explain the change without the agent.&lt;/strong> Not re-derive it, just explain it: what it does, what it assumes, what would break it. If they can&amp;rsquo;t, the PR isn&amp;rsquo;t ready, regardless of what the tests say. This is the situation-awareness test, and it&amp;rsquo;s the one I&amp;rsquo;d keep if I could only keep one.&lt;/p>
&lt;p>A method that works for me is to draw it. Diagram the process step by step and explode out what each step is doing. Seeing the change from a different angle makes problems in the logic or the sequence much easier to spot.&lt;/p>
&lt;p>&lt;strong>Decide the automation level per kind of work, deliberately.&lt;/strong> Vella&amp;rsquo;s first recommendation, and the one I&amp;rsquo;d underline. Exploratory analysis and boilerplate ingestion: let the agent run, review the output. Dimension logic, anything touching money or compliance, migrations: agent drafts, human owns every line. We&amp;rsquo;ve written that mapping down, and it changes which PRs get a thirty-second glance and which get a sit-down.&lt;/p>
&lt;p>&lt;strong>Keep people practising.&lt;/strong> Bainbridge&amp;rsquo;s point was that the best automation needs the most-trained operators, because interventions are rare and high-stakes. For us that means the occasional piece of work done without an agent, on purpose, especially for juniors. Not as a hazing ritual. Because when the agent produces something wrong in a way nobody&amp;rsquo;s seen before, the only person who&amp;rsquo;ll catch it is one who has built the thing by hand at least once.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="the-tradeoff">The tradeoff&lt;/h3>
&lt;/br>
&lt;p>I&amp;rsquo;m not nostalgic about this. I&amp;rsquo;m seeing teams move faster than they ever have, and the engineers I&amp;rsquo;d most want to keep have taken to directing agents with the same instincts they&amp;rsquo;d use directing a junior. More than ever, the skills that matter are leadership and testing.&lt;/p>
&lt;p>But the human factors people saw this coming more than forty years ago. Supervision looks like less work and is actually different work: more responsibility, less feedback, and a skill that atrophies quietly if nobody tends it. The managers who come out of this well are going to be the ones who treat the review burden as the job rather than the overhead, and who keep deliberately rebuilding the manual skill in a team that no longer needs it on an ordinary Tuesday.&lt;/p>
&lt;p>Vella says this work needs new definitions of what good looks like. Here&amp;rsquo;s a modest start for data teams. A good supervisor is someone who can tell you what a pipeline does, why it&amp;rsquo;s right, and how they&amp;rsquo;d know if it stopped being right. That was always the standard. It&amp;rsquo;s just that for a while, writing the code did most of the proving for us.&lt;/p>
&lt;p>&lt;/br>&lt;/br>&lt;/p></content:encoded><category>Leadership</category><category>AI</category><category>AI Agents</category><category>Engineering Leadership</category><category>Code Review</category><category>Team Culture</category><category>Claude Code</category><category>dbt</category><category>Supervisory Control</category></item><item><title>World Building: Your World Still Needs a Why</title><link>https://ghostinthedata.info/posts/2026/2026-10-30-handlers-vault-part-1/</link><pubDate>Fri, 30 Oct 2026 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-10-30-handlers-vault-part-1/</guid><author>Chris Hillman</author><description>Part 1 of The Handler's Vault: building an RPG world in Obsidian as small, linked notes with motives attached, and why that is a data modelling problem in a cloak.</description><content:encoded>&lt;p>On the 17th of October, people sat down in the Great Hall of Stirling Castle to play &lt;a href="https://www.geeknative.com/242404/chaosium-brings-call-of-cthulhu-to-stirling-castle-for-a-live-historical-game/" target="_blank" rel="noopener">Call of Cthulhu&lt;/a>.&lt;/p>
&lt;p>I was in Scotland at the time. I didn&amp;rsquo;t make it. That still annoys me.&lt;/p>
&lt;p>What caught my attention wasn&amp;rsquo;t the castle (well fine, it kind of was the castle). It was what sat underneath the show. The game came out of &lt;a href="https://blogs.ed.ac.uk/hardrada/2026/" target="_blank" rel="noopener">A Viking in the Sun&lt;/a>, a University of Edinburgh research project on Harald Hardrada, and its sister show in Istanbul earlier this year was built from real archaeological objects, historical sources and Icelandic sagas. Historians did months of careful work so that four players could walk into a room and go wherever they liked.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="the-prep-paradox">The prep paradox&lt;/h3>
&lt;/br>
&lt;p>I&amp;rsquo;m building a Delta Green campaign for next year. Delta Green is the Call of Cthulhu offshoot where the players are federal agents trying to contain things that shouldn&amp;rsquo;t exist, usually at a terrible cost to themselves. Think True Detective or The X-Files. The person running the game is called the Handler, which is a much better job title than Dungeon Master - which can turn heads if said in the wrong room - &amp;ldquo;Im a Dungeon Master on weekends&amp;rdquo;.&lt;/p>
&lt;p>Every game runner learns the same lesson early: players will ignore your plot. They&amp;rsquo;ll interrogate the wrong witness, burn down the wrong barn, and get emotionally attached to the bloke who was meant to be a one-line clue. That&amp;rsquo;s fine. It&amp;rsquo;s what makes collaborative storytelling work, and the chaos is where the memories come from. So why prep at all?&lt;/p>
&lt;p>Because in my experience, the sessions that feel &lt;em>real&lt;/em> are the ones where the world pushes back with reasons. When the players do something unexpected and an NPC responds the way that person would actually respond, the table feels it. Nobody claps. They just lean in.&lt;/p>
&lt;p>Delta Green also needs the cosmic horror to land, and that&amp;rsquo;s slow work. Seeds planted early are what let the background creep in later without you forcing it.&lt;/p>
&lt;p>You can&amp;rsquo;t improvise a reason you never wrote down. You can only improvise a reaction.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="this-is-a-data-problem-wearing-a-cloak">This is a data problem wearing a cloak&lt;/h3>
&lt;/br>
&lt;p>Most campaign notes look like most data platforms. There&amp;rsquo;s a 40-page setting document nobody re-reads, a notebook with half the NPCs in it, a Google Doc with the other half, and a lot of important facts living in one person&amp;rsquo;s head. Everything exists. Nothing joins.&lt;/p>
&lt;p>Now there&amp;rsquo;s a new consumer of that mess. If you want an LLM to help you build the world, it needs context, and it can only use what it can find and understand. There&amp;rsquo;s been a steady stream of writing this year about &lt;a href="https://martinfowler.com/articles/making-data-ready-for-agentic-ai.html" target="_blank" rel="noopener">making data ready for agents&lt;/a>. A campaign setting is the friendliest sandbox I can think of for trying that out, because the stakes are a ruined Thursday night rather than a ruined quarterly report.&lt;/p>
&lt;p>Here&amp;rsquo;s how I&amp;rsquo;m structuring it.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="one-note-one-thing">One note, one thing&lt;/h3>
&lt;/br>
&lt;p>Every NPC, organisation, place, object and event gets its own note. One thing per note. If you&amp;rsquo;ve done any dimensional modelling, you already know this move: pick the grain first, and everything after gets easier.&lt;/p>
&lt;p>A big setting document is a table with no keys. You can read it, but you can&amp;rsquo;t point at a row in it.&lt;/p>
&lt;p>Here&amp;rsquo;s an NPC note (She&amp;rsquo;s invented for this post, so no spoilers for anyone in my group).&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>type: npc
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>name: Dr Ruth Calder
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>role: Archivist, county historical society
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>location: &amp;#34;[[Ashby Falls]]&amp;#34;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>faction: &amp;#34;[[Calder Family Trust]]&amp;#34;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>status: canon
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>wants: Keep the 1952 flood records sealed
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>fears: Her brother&amp;#39;s name turning up in them
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>knows:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> &amp;#34;[[The Drowned Chapel]]&amp;#34;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> &amp;#34;[[Agent Morrow]]&amp;#34;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>secret: She burned three ledgers in 1998 and kept the fourth
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Polite, precise, and helpful right up to the moment anyone
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>asks about the chapel. Offers tea. Watches the door.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Look at the fields. &lt;code>name&lt;/code>, &lt;code>role&lt;/code> and &lt;code>location&lt;/code> are what most people write down. &lt;code>wants&lt;/code>, &lt;code>fears&lt;/code>, &lt;code>knows&lt;/code> and &lt;code>secret&lt;/code> are what make her playable.&lt;/p>
&lt;p>Strip those four out and you&amp;rsquo;ve got a dimension with a surrogate key and a job title. Leave them in and you know what she does when an agent turns up asking about 1952, even though you never planned for that scene. She stalls. She calls her brother. She doesn&amp;rsquo;t mention the fourth ledger.&lt;/p>
&lt;p>That&amp;rsquo;s the &amp;ldquo;why&amp;rdquo;, and it&amp;rsquo;s the part nobody writes down.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="links-are-your-relationships">Links are your relationships&lt;/h3>
&lt;/br>
&lt;p>Obsidian&amp;rsquo;s &lt;code>[[wiki links]]&lt;/code> turn the notes into a graph. Ruth links to the chapel, the chapel links to the flood, the flood links to three other people who&amp;rsquo;d rather it stayed buried. Backlinks show you every note that points at a given place or person, for free.&lt;/p>
&lt;p>Add the Dataview community plugin and the frontmatter becomes queryable:&lt;/p>
&lt;pre tabindex="0">&lt;code class="language-dataview" data-lang="dataview">TABLE wants, fears
FROM &amp;#34;npcs&amp;#34;
WHERE contains(knows, [[The Drowned Chapel]])
&lt;/code>&lt;/pre>&lt;p>You can run this query you run at the table, mid-session, when the players knock on a door you didn&amp;rsquo;t plan for. Who in this town knows about the chapel, and what are they afraid of? Ten seconds, and you&amp;rsquo;re improvising from something solid.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="sessions-are-an-event-log">Sessions are an event log&lt;/h3>
&lt;/br>
&lt;p>Session notes are dated and append-only. I don&amp;rsquo;t go back and edit what happened in session three, any more than you&amp;rsquo;d update last month&amp;rsquo;s transactions.&lt;/p>
&lt;p>The current state of the world gets derived from the log. Ruth&amp;rsquo;s &lt;code>status&lt;/code> changes because something happened in a session, and the session note is the record of why. If you&amp;rsquo;ve spent any time with slowly changing dimensions, you&amp;rsquo;ll recognise the shape. The players&amp;rsquo; choices are the facts. Your prep is the dimension they land on.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="canon-draft-and-rumour">Canon, draft and rumour&lt;/h3>
&lt;/br>
&lt;p>Every note carries a status:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>canon&lt;/strong>: true in the world, and it&amp;rsquo;ll stay true&lt;/li>
&lt;li>&lt;strong>draft&lt;/strong>: an idea I haven&amp;rsquo;t committed to&lt;/li>
&lt;li>&lt;strong>rumour&lt;/strong>: something people in the world believe, which may be wrong on purpose&lt;/li>
&lt;/ul>
&lt;p>Ideas start as drafts. They get checked against canon for contradictions, then promoted. &lt;a href="https://ghostinthedata.info/posts/2025/2025-05-18-wap-data-pipelines/">Write, audit, publish&lt;/a>, applied to fiction.&lt;/p>
&lt;p>That gate matters a lot more in Part 2, when Claude starts proposing ideas. Nothing it writes goes straight to canon. Ever.&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="why-bother-with-a-vector-index">Why bother with a vector index?&lt;/h3>
&lt;/br>
&lt;p>Text search finds words. Embeddings find meaning.&lt;/p>
&lt;p>Ask &amp;ldquo;who would be nervous if the flood records came out?&amp;rdquo; and a keyword search returns nothing, because none of the notes contain the word &lt;em>nervous&lt;/em>. An embedding search finds Ruth, because her fears line is about exactly that.&lt;/p>
&lt;p>This is also why the one-note-one-thing rule pays off twice. A note about a single person is the right size for retrieval. A 40-page document chopped into arbitrary chunks gives the model half of Ruth and half of someone else, and then it confidently tells you Ruth runs the pub.&lt;/p>
&lt;p>(Yes, you could just use Ctrl-F. Right up until you can&amp;rsquo;t remember whether you called it the Drowned Chapel or the Flooded Church.)&lt;/p>
&lt;/br>
&lt;hr>
&lt;h3 id="the-tradeoff">The tradeoff&lt;/h3>
&lt;/br>
&lt;p>None of this is free.&lt;/p>
&lt;p>&lt;strong>What you get:&lt;/strong> a world that answers back. When the players go off the map, you&amp;rsquo;re improvising from motives you wrote down, not making them up under pressure. The vault is also ready for an AI collaborator, because it&amp;rsquo;s already in small, linked, labelled pieces.&lt;/p>
&lt;p>&lt;strong>What it costs:&lt;/strong> prep time, and the real risk of building an iceberg. Most of what you write the players will never see. Some Handlers will find that satisfying. Others will find it maddening.&lt;/p>
&lt;p>&lt;strong>What to watch for:&lt;/strong> the moment you catch yourself defending a note against a player&amp;rsquo;s choice. When the note and the players disagree, the note loses. The vault exists to serve their story, not to script it.&lt;/p>
&lt;p>The players at Stirling Castle got to wander wherever they liked because historians had done the slow work first. That&amp;rsquo;s the job of the vault: be the slow work, so the table can be fast.&lt;/p>
&lt;p>Im still finding my feed, becoming a Handler. Having a co-Handler helper gives me the confidence, it reads the vault, checks motives, generates rumours from secrets, and helps the world react when the players do something nobody expected. Everything it writes goes into drafts, and only I decide what comes out.&lt;/p>
&lt;p>&lt;/br>&lt;/br>&lt;/p></content:encoded><category>Data Modelling</category><category>AI</category><category>Obsidian</category><category>Knowledge Management</category><category>Vector Search</category><category>Claude</category><category>Tabletop RPG</category><category>Data Modelling</category><category>Semantic Layer</category></item><item><title>Ghost Skills: Teaching AI Agents to Think Like Data Engineers</title><link>https://ghostinthedata.info/posts/2026/2026-06-14-ai-ghost-skills-ai-agents/</link><pubDate>Sun, 14 Jun 2026 09:00:00 +1000</pubDate><guid>https://ghostinthedata.info/posts/2026/2026-06-14-ai-ghost-skills-ai-agents/</guid><author>Chris Hillman</author><description>A new open-source skills repo that gives AI coding agents the methodology they keep forgetting — grain, keys, SCD2, incident comms, and more. Tool-agnostic, composable, and opinionated.</description><content:encoded>&lt;p>Another week, another skills repo on the GitHub trending page. I know. There are roughly seventeen of them now, all promising to turn your AI coding agent from a confident intern into a slightly-less-confident intern. Most of them are great. Most of them are also built by solo devs, for solo devs, on solo-dev codebases that fit comfortably in a context window.&lt;/p>
&lt;p>Which is fine, if that&amp;rsquo;s your world.&lt;/p>
&lt;p>Less fine if your world involves a Snowflake warehouse with four tables that &lt;em>could&lt;/em> be the source of truth for &amp;ldquo;customer&amp;rdquo;, an SCD2 someone half-built in 2021 and quietly walked away from, and a dbt project where &lt;code>stg_users_final_v3_actually_use_this&lt;/code> is, somehow, the one you&amp;rsquo;re meant to use. (Don&amp;rsquo;t laugh. You&amp;rsquo;ve seen worse.)&lt;/p>
&lt;p>I&amp;rsquo;ve been using AI agents day-to-day for a while now — mostly as a peer reviewer, a second pair of eyes, a thinking partner that doesn&amp;rsquo;t sigh when I ask it to look at the same query three times. They&amp;rsquo;re genuinely useful.&lt;/p>
&lt;p>So I built something &lt;a href="https://github.com/ghostinthedata-info/skills" target="_blank" rel="noopener">Ghost Skills&lt;/a> — a collection of data-engineering methodology skills for AI coding agents. Before I get into what&amp;rsquo;s in it, let me explain why I bothered.&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="the-problem-isnt-the-model-its-that-data-work-has-rules-nobody-told-it-about">The problem isn&amp;rsquo;t the model. It&amp;rsquo;s that data work has rules nobody told it about.&lt;/h3>
&lt;br>
&lt;p>Your AI agent is, by default, an extremely bright graduate on their first day. It knows Python. It knows SQL. It can write a beautifully-commented &lt;code>for&lt;/code> loop and explain window functions to you in five different ways. It will happily generate a fact table, a dbt model, an Airflow DAG, all of it looking very professional.&lt;/p>
&lt;p>What it doesn&amp;rsquo;t know is everything that actually matters.&lt;/p>
&lt;p>It doesn&amp;rsquo;t know that the &lt;code>customer_id&lt;/code> in that source isn&amp;rsquo;t really unique because of the 2022 CRM migration. It doesn&amp;rsquo;t know that Roger pushed a release on Friday afternoon (because of course he did) and the platform&amp;rsquo;s been wheezing through a seven-year backfill ever since. It doesn&amp;rsquo;t know that this dimension changes slowly and the audit team will lose their minds if you flatten the history. It doesn&amp;rsquo;t know that the business definition of &amp;ldquo;active customer&amp;rdquo; changed in March, but only in the marketing data mart, and only sometimes.&lt;/p>
&lt;p>These aren&amp;rsquo;t model intelligence problems. The model is plenty smart. They&amp;rsquo;re &lt;em>context&lt;/em> problems — and the context is exactly the stuff that takes a data engineer two years to learn and roughly thirty minutes to forget when they leave the company.&lt;/p>
&lt;p>What agents are missing isn&amp;rsquo;t capability. It&amp;rsquo;s methodology. It&amp;rsquo;s the durable craft of data engineering — the part that doesn&amp;rsquo;t change when you swap Snowflake for BigQuery, or dbt for SQLMesh, or your warehouse for a &amp;ldquo;lakehouse&amp;rdquo; that&amp;rsquo;s somehow priced like a warehouse anyway.&lt;/p>
&lt;p>That&amp;rsquo;s the gap this is attempting to fill.&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="skills-briefly-for-anyone-who-hasnt-drunk-this-particular-kool-aid-yet">Skills, briefly, for anyone who hasn&amp;rsquo;t drunk this particular Kool-Aid yet&lt;/h3>
&lt;br>
&lt;p>If you&amp;rsquo;ve not been living inside Claude Code or Cursor for the last six months: a &lt;em>skill&lt;/em> is a folder of instructions you give an AI coding agent. &amp;ldquo;When you build a fact table, declare the grain first.&amp;rdquo; &amp;ldquo;When you write tests, follow this severity model.&amp;rdquo; &amp;ldquo;When something breaks, here&amp;rsquo;s the comms workflow.&amp;rdquo; You version it, you commit it, you keep it in the repo. The agent reads it before doing the thing, and re-reads it next session because agents have the long-term memory of a goldfish.&lt;/p>
&lt;p>It&amp;rsquo;s a deceptively powerful pattern. Instead of pasting the same prompt every session (and forgetting half of it, and writing it slightly differently this time), the standard lives in the repo. New team member joins? They get the skills. Agent updates next month? It still reads the skills.&lt;/p>
&lt;p>The catch — and we&amp;rsquo;ll come back to this — is that skills tell an agent &lt;em>how&lt;/em> you do something. They don&amp;rsquo;t tell it &lt;em>what&lt;/em> you&amp;rsquo;ve built, &lt;em>why&lt;/em> you built it that way, or &lt;em>which&lt;/em> of your seven utility tables is the one that&amp;rsquo;s still maintained.&lt;/p>
&lt;p>For solo devs, that&amp;rsquo;s fine. There is no &amp;ldquo;seven utility tables&amp;rdquo;. For data teams at any kind of scale, it&amp;rsquo;s a real limit. More on that at the end.&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="introducing-ghost-skills">Introducing Ghost Skills&lt;/h3>
&lt;br>
&lt;p>Right. The repo: &lt;a href="https://github.com/ghostinthedata-info/skills" target="_blank" rel="noopener">github.com/ghostinthedata-info/skills&lt;/a>. Built mostly for myself, polished up for sharing.&lt;/p>
&lt;p>30-second setup:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>npx skills@latest add ghostinthedata-info/skills
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Pick the skills you want, pick your agent (Claude Code, Codex, Cursor — whatever you&amp;rsquo;re using), and run &lt;code>/setup-ghost-skills&lt;/code>. It&amp;rsquo;ll ask you three questions: warehouse dialect, transform tooling, and how your domain docs are laid out. Then it writes itself a configuration block into your &lt;code>CLAUDE.md&lt;/code> or &lt;code>AGENTS.md&lt;/code> and gets out of your way.&lt;/p>
&lt;p>The agent reads it next session. And the one after that. And the one after that. The same standards, every time, without you re-explaining them.&lt;/p>
&lt;p>That&amp;rsquo;s it. That&amp;rsquo;s the whole thing.&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="whats-in-the-catalogue">What&amp;rsquo;s in the catalogue&lt;/h3>
&lt;br>
&lt;p>The skills group into four areas, mapped to roughly how data work actually happens.&lt;/p>
&lt;p>&lt;strong>Discovery&lt;/strong> is everything you should do before you build anything, and frequently don&amp;rsquo;t. &lt;code>profile-data&lt;/code> runs the baseline checks on a new dataset — row counts, cardinality, null analysis, key uniqueness — so the agent isn&amp;rsquo;t generating models against assumptions that fall apart in week two. &lt;code>gather-requirements&lt;/code> pins down grain, sources, consumers, freshness SLAs, and acceptance criteria one question at a time, instead of letting the agent guess. &lt;code>refine-context&lt;/code> stress-tests a plan against your documented domain model and writes the decisions back to &lt;code>CONTEXT.md&lt;/code> and ADRs as they crystallise.&lt;/p>
&lt;p>&lt;strong>Modelling&lt;/strong> is the methodology your agent should already be applying and usually isn&amp;rsquo;t. &lt;code>dimensional-modeling&lt;/code> walks the four-step process. &lt;code>fact-table-design&lt;/code> forces grain declaration and measure classification before a single line of SQL gets generated. &lt;code>keys&lt;/code> covers business, natural, surrogate, composite, and durable keys — and the anti-patterns that bite you eighteen months later when the source system gets re-platformed. &lt;code>slowly-changing-dimensions&lt;/code> covers SCD types 0 through 7, including my Healing Tables approach to deterministic, path-independent SCD2.&lt;/p>
&lt;p>&lt;strong>Quality&lt;/strong> is where defensive engineering lives. &lt;code>test-data&lt;/code> produces a test plan you can actually defend in a code review — uniqueness, referential integrity, nulls, accepted values, freshness, volume variance — and maps each test to a severity level. &lt;code>performance-tuning&lt;/code> follows the measure-first philosophy: find the critical path, then fix partition pruning, incremental processing, and the phantom dependencies that are silently doubling your costs. &lt;code>spark-performance&lt;/code> handles the distributed case — partition counts, shuffle minimisation, skew handling, broadcasting small tables.&lt;/p>
&lt;p>&lt;strong>Operations&lt;/strong> is what should happen when things go pop. &lt;code>incident-comms&lt;/code> gives the agent the severity classification, notification workflow, update cadence, and post-incident review template — the &amp;ldquo;Don&amp;rsquo;t Go Dark&amp;rdquo; pattern in skill form. &lt;code>pipeline-design&lt;/code> encodes idempotency, reproducibility, and defensive engineering as defaults instead of afterthoughts. &lt;code>data-as-a-product&lt;/code> brings data mesh thinking into the agent — domain ownership, discoverability, SLAs, federated governance. &lt;code>data-security-classification&lt;/code> covers the four A&amp;rsquo;s, PII/PHI handling, and least privilege.&lt;/p>
&lt;p>There&amp;rsquo;s a setup skill (&lt;code>setup-ghost-skills&lt;/code>) that runs once per repo and handles the configuration. Everything else, you opt into.&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="why-tool-agnostic-even-though-tool-specific-would-sell-better">Why tool-agnostic, even though tool-specific would sell better&lt;/h3>
&lt;br>
&lt;p>The temptation when building something like this is to specialise. Snowflake-only. dbt-only. Make every skill assume your exact stack and produce immediately runnable code.&lt;/p>
&lt;p>I deliberately didn&amp;rsquo;t do that, and the tradeoff is worth naming honestly. Generic skills are less immediately useful than highly specific ones. A skill that knows your exact dbt project structure, your column naming conventions, your team&amp;rsquo;s snake_case-vs-camelCase argument that&amp;rsquo;s been going on since 2022 — that&amp;rsquo;s more powerful, day one.&lt;/p>
&lt;p>But it&amp;rsquo;s also yours. It&amp;rsquo;s not shareable. And the bits that are genuinely universal — &lt;em>declare the grain before you build the fact&lt;/em>, &lt;em>a business key isn&amp;rsquo;t a real business key until you&amp;rsquo;ve checked it&amp;rsquo;s unique&lt;/em>, &lt;em>test before publish&lt;/em> — those don&amp;rsquo;t change between platforms. They didn&amp;rsquo;t change when we moved from Teradata to Snowflake. They won&amp;rsquo;t change when we move from Snowflake to whatever everyone&amp;rsquo;s furiously rebranding next year.&lt;/p>
&lt;p>Ghost Skills is the base layer. Your repo&amp;rsquo;s &lt;code>CONTEXT.md&lt;/code> is your specialisation layer. They&amp;rsquo;re meant to work together, not replace each other. Fork it. Add a &lt;code>snowflake-cost-optimisation&lt;/code> skill on top. Open a PR if you want to share it back. (Or don&amp;rsquo;t. Do what you like.)&lt;/p>
&lt;br>
&lt;hr>
&lt;h3 id="skills-help-they-dont-solve-everything-lets-not-pretend">Skills help. They don&amp;rsquo;t solve everything. Let&amp;rsquo;s not pretend.&lt;/h3>
&lt;br>
&lt;p>The honest thing to say at this point is that skills are not a silver bullet, and anyone telling you otherwise is selling you a course.&lt;/p>
&lt;p>Skills tell an agent &lt;em>how&lt;/em> to do things. They don&amp;rsquo;t tell it &lt;em>what&amp;rsquo;s there&lt;/em> or &lt;em>why&lt;/em>. An agent that knows the Kimball four-step process will still make the wrong grain declaration if it doesn&amp;rsquo;t understand the business process it&amp;rsquo;s modelling. A skill can encode what SCD2 is; it can&amp;rsquo;t replace a conversation with the stakeholder who actually needs the history. A skill can encode the Write-Audit-Publish pattern; it can&amp;rsquo;t tell the agent which of your fourteen &amp;ldquo;users&amp;rdquo; tables is the one to run it against.&lt;/p>
&lt;p>The other thing skills can&amp;rsquo;t do is fix a stale skill. The conventions you encode today are the conventions you knew today. If your team&amp;rsquo;s standards evolve — and they should — your skills need to evolve with them. A stale skill is worse than no skill, because it gives the agent a confident wrong answer instead of asking a question.&lt;/p>
&lt;p>So treat them as a starting point. Use them. Fork them. Argue with them in a code review. Update them when your standards shift. The agent will keep reading whatever&amp;rsquo;s in the repo, faithfully, every session — which means the skills are only ever as good as the last person who maintained them.&lt;/p>
&lt;p>The repo is at &lt;a href="https://github.com/ghostinthedata-info/skills" target="_blank" rel="noopener">github.com/ghostinthedata-info/skills&lt;/a>. Fork it for your cloud-specific layer. Open issues if something&amp;rsquo;s wrong. Open a PR if you&amp;rsquo;ve got methodology worth encoding and you don&amp;rsquo;t want to keep it to yourself.&lt;/p>
&lt;p>And if you build something better on top of it, please tell me.&lt;/p>
&lt;p>&lt;br>&lt;br>&lt;/p></content:encoded><category>Data Engineering</category><category>AI</category><category>AI Agents</category><category>Claude Code</category><category>dbt</category><category>Data Modelling</category><category>Dimensional Modelling</category><category>Data Quality</category><category>Open Source</category><category>Snowflake</category></item><item><title>When AI Stops Treating Us Like People: The Twin Crises Nobody Wants to Discuss</title><link>https://ghostinthedata.info/posts/2025/2025-12-12-ai-bubble/</link><pubDate>Fri, 12 Dec 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-12-12-ai-bubble/</guid><author>Chris Hillman</author><description>AI companies have burned $560 billion chasing $35 billion in revenue while public trust collapses. The intersection of financial instability and human cost could trigger a correction that reshapes the industry.</description><content:encoded>&lt;h3 id="the-moment-it-stops-being-theoretical">The moment it stops being theoretical&lt;/h3>
&lt;hr>
&lt;p>A friend was telling me about their recent job interviews. Not the questions they struggled with or the technical challenge they bombed—that&amp;rsquo;s normal interview anxiety. No, they were frustrated because they&amp;rsquo;d spent forty-five minutes talking to an AI chatbot that analysed their &amp;ldquo;enthusiasm for remote work&amp;rdquo; and tracked their eye movements while asking whether they&amp;rsquo;d &amp;ldquo;misrepresent themselves 3.8x more than average candidates.&amp;rdquo;&lt;/p>
&lt;p>The speed with which I would exit an AI interview would rival even the most seasoned teenager&amp;rsquo;s alt-tab skills.&lt;/p>
&lt;p>But here&amp;rsquo;s the thing—this isn&amp;rsquo;t just another complaint about awkward technology or corporate efficiency theatre. This moment, repeated thousands of times daily across hiring processes, represents something more fundamental breaking down. AI was supposed to eliminate the boring parts of work, the repetitive drudgery that didn&amp;rsquo;t require human judgment. Instead, it&amp;rsquo;s eliminating the human parts of work—the connection, the authenticity, the actual assessment of whether someone can do the job.&lt;/p>
&lt;p>And that pattern—AI deployed to dehumanize rather than enhance, to reduce costs rather than create value—isn&amp;rsquo;t limited to hiring. It&amp;rsquo;s the story of a $560 billion investment that&amp;rsquo;s generated just $35 billion in revenue. It&amp;rsquo;s the story of public trust collapsing from cautious optimism to outright scepticism. It&amp;rsquo;s the story of two interconnected crises that could reshape the entire technology industry if either one fails.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-in-the-dystopian-hellscape-is-going-on">What in the dystopian hellscape is going on&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s stay with that hiring example for a moment, because it crystallizes the problem. Companies now use AI to screen resumes, conduct first-round interviews, analyse voice patterns for &amp;ldquo;culture fit,&amp;rdquo; track eye movements for &amp;ldquo;engagement signals,&amp;rdquo; and evaluate &amp;ldquo;authenticity&amp;rdquo; through speech pattern analysis. The justification? Candidates are supposedly committing fraud &amp;ldquo;3.8x higher rates&amp;rdquo;—though higher than what, when, or where remains conveniently unspecified.&lt;/p>
&lt;p>So AI is good when companies use it to dehumanize candidates, but it&amp;rsquo;s bad when candidates use it to stay competitive in that same dehumanizing process. Got it.&lt;/p>
&lt;p>The fundamental issue, is that &amp;ldquo;AI was supposed to do the parts of the jobs humans found boring or repetitive.&amp;rdquo; Taking meeting notes. Generating documentation. Processing expense reports. The stuff that makes you think &amp;ldquo;surely a computer could do this.&amp;rdquo; Instead, we&amp;rsquo;re using it for the parts that desperately need human judgment—assessing whether someone will thrive in a role, whether they&amp;rsquo;ll mesh with a team, whether their particular combination of skills and experience makes them the right fit for this specific challenge.&lt;/p>
&lt;p>If I&amp;rsquo;m hiring a human for a role, I don&amp;rsquo;t care about grading their every word or analysing their eye movement patterns. Doing so is fundamentally counterproductive since it encourages candidates to be as boring and uninteresting as possible while maintaining perfect eye contact and regurgitating perfect phrases, lest they don&amp;rsquo;t get the job.&lt;/p>
&lt;p>Honestly? I don&amp;rsquo;t think I want to hang out with the people who AI likes anyway.&lt;/p>
&lt;p>But this pattern—deploying AI in ways that optimise for the wrong things—appears everywhere once you start looking. McDonald&amp;rsquo;s AI drive-thru adding 260 Chicken McNuggets to orders. Klarna claiming AI &amp;ldquo;did the work of 700 agents&amp;rdquo; before quietly reversing course when quality tanked. Google&amp;rsquo;s AI Overview suggesting you add glue to pizza. These aren&amp;rsquo;t bugs. They&amp;rsquo;re symptoms of a systematic misapplication of technology by companies that haven&amp;rsquo;t figured out what problems AI actually solves.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-numbers-that-should-terrify-you">The numbers that should terrify you&lt;/h3>
&lt;hr>
&lt;p>OpenAI, the company most synonymous with the AI revolution, lost &lt;strong>$5 billion in 2024 on revenue of just $3.7 billion&lt;/strong>. Let that sink in. They&amp;rsquo;re spending $2.25 for every dollar they earn. The projected loss for 2025? $14.4 billion. Between 2023 and 2028, the company expects cumulative losses of &lt;strong>$44 billion&lt;/strong> before potentially—possibly, theoretically—reaching cash-flow positive status in 2029 or 2030.&lt;/p>
&lt;p>Despite these staggering losses, OpenAI&amp;rsquo;s valuation rocketed from $157 billion in October 2024 to somewhere between $300-500 billion today. That&amp;rsquo;s a tripling in twelve months with no profitability in sight. These aren&amp;rsquo;t valuations based on business fundamentals. They&amp;rsquo;re bets that future revenue will somehow, miraculously, justify present spending.&lt;/p>
&lt;p>Anthropic tells the same story with different numbers. Revenue jumped from $1 billion annually in 2024 to a projected $9 billion by end of 2025—impressive growth, right? Except they&amp;rsquo;re burning through &lt;strong>$2.7-3 billion annually&lt;/strong> and don&amp;rsquo;t expect to break even until 2028. The valuation inflated tenfold in eighteen months, from $18.4 billion to $183 billion. Again: not a sustainable business, but a bet on a future that may never materialise.&lt;/p>
&lt;p>The cost structure explains why profitability remains this elusive unicorn. Every ChatGPT query costs real money to compute. Every Claude conversation. Every AI-generated image. &lt;strong>Inference—running trained models—accounts for 80-90% of total AI lifetime expenses&lt;/strong>. Unlike traditional software where marginal costs approach zero as you scale, AI scales in reverse. More users mean more costs, linearly and predictably.&lt;/p>
&lt;p>Cursor, the AI coding assistant, reportedly sends 100% of its revenue to Anthropic just for model access. Perplexity AI spent &lt;strong>164% of revenue&lt;/strong> on compute providers in 2024. When your biggest customers operate at negative margins, when the companies building businesses on your platform lose money on every transaction, the entire ecosystem becomes brittle.&lt;/p>
&lt;p>The hyperscalers—Microsoft, Google, Amazon, Meta—are betting their futures on AI infrastructure with combined 2025 capital expenditures projected at &lt;strong>$325-380 billion&lt;/strong>, up 46% from 2024. These spending levels now exceed 1% of U.S. GDP quarterly. Microsoft committed $88.7 billion for fiscal 2025. Amazon projected $105-125 billion. Yet Meta&amp;rsquo;s CFO acknowledged that &amp;ldquo;genAI work is not going to be a meaningful driver of revenue this year or next year.&amp;rdquo;&lt;/p>
&lt;p>Read that again. Companies are spending hundreds of billions building infrastructure for revenue that their own financial officers admit won&amp;rsquo;t materialise for years. The gap between investment and returns has never been wider in technology history.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="when-the-guy-who-called-2008-places-a-billion-dollar-bet">When the guy who called 2008 places a billion-dollar bet&lt;/h3>
&lt;hr>
&lt;p>Michael Burry isn&amp;rsquo;t some permabear shouting about every market downturn. He&amp;rsquo;s the investor who correctly identified the 2008 subprime mortgage crisis before it happened, who placed massive bets against the housing market when everyone called him crazy, who was proven devastatingly right when the entire financial system nearly collapsed.&lt;/p>
&lt;p>In early November 2025, Burry disclosed regulatory filings showing over &lt;strong>$1 billion in put options against AI leaders&lt;/strong>: $912 million against Palantir (66% of his portfolio) and $187 million against Nvidia. His thesis is explicit and detailed, laid out in his Substack newsletter &amp;ldquo;Cassandra Unchained.&amp;rdquo;&lt;/p>
&lt;p>&amp;ldquo;I am not claiming Nvidia is Enron. It is clearly Cisco,&amp;rdquo; Burry wrote. He&amp;rsquo;s identifying Nvidia as today&amp;rsquo;s equivalent of Cisco during the dot-com bubble—the essential hardware supplier powering a massive capital investment cycle that will eventually see overbuilt supply meet far less demand than expected.&lt;/p>
&lt;p>But his most damning observation concerns accounting practices. Burry projects that tech companies are &lt;strong>understating depreciation by $176 billion between 2026-2028&lt;/strong> by extending the useful life of AI computing equipment from realistic 2-3 year product cycles to artificially long periods. This accounting treatment inflates reported earnings—Oracle by 26.9%, Meta by 20.8%—creating the illusion of profitability where losses actually exist.&lt;/p>
&lt;p>His criticism of &amp;ldquo;circular financing&amp;rdquo; cuts deeper. OpenAI owns 10% of AMD while Nvidia invested $100 billion in OpenAI. Microsoft is both a major OpenAI shareholder and one of Nvidia&amp;rsquo;s largest customers (representing roughly 20% of Nvidia revenue). CoreWeave, funded by Nvidia, serves OpenAI&amp;rsquo;s compute needs. &amp;ldquo;True end demand is ridiculously small,&amp;rdquo; Burry observed. &amp;ldquo;Almost all customers are funded by their dealers.&amp;rdquo;&lt;/p>
&lt;p>When an investor who successfully predicted the last major financial crisis places billion-dollar bets against your industry and explicitly compares your accounting practices to pre-2008 mortgage fraud, that&amp;rsquo;s not speculation. That&amp;rsquo;s pattern recognition.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-trust-collapse-nobody-wants-to-acknowledge">The trust collapse nobody wants to acknowledge&lt;/h3>
&lt;hr>
&lt;p>Here&amp;rsquo;s the part where we connect the financial instability back to that AI interview experience. Because these crises aren&amp;rsquo;t separate—they&amp;rsquo;re feeding each other in ways that threaten the entire AI business model.&lt;/p>
&lt;p>According to Pew Research, &lt;strong>only 17% of Americans believe AI will have a positive impact&lt;/strong> on the U.S. over the next 20 years. A full 51% are more concerned than excited about AI&amp;rsquo;s increased use in daily life. These figures represent dramatic worsening since 2021, when optimism still outweighed concern.&lt;/p>
&lt;p>The gap between expert opinion and public sentiment has become a chasm. While 56% of AI experts see positive impact ahead, only 17% of the general public agrees. While 73% of experts expect positive employment effects, only 23% of the public shares that view. Either the experts are completely disconnected from reality, or the public has accurately identified risks the industry refuses to acknowledge.&lt;/p>
&lt;p>The trust erosion isn&amp;rsquo;t abstract. It&amp;rsquo;s rooted in lived experience. That AI interview that made you feel reduced to &amp;ldquo;an n-dimensional matrix.&amp;rdquo; The AI-generated customer service response that completely missed your problem. The hiring algorithm that rejected your application before a human ever saw it. The chatbot that confidently gave you wrong information. The image generator that couldn&amp;rsquo;t distinguish historical accuracy from Reddit jokes.&lt;/p>
&lt;p>That &amp;ldquo;authenticity is a limited resource that is diminishing every day.&amp;rdquo; Every time someone encounters AI deployed to reduce costs rather than add value, deployed to eliminate human judgment rather than enhance it, that person becomes incrementally more sceptical. Not of technology in general—of this specific application pattern where efficiency trumps everything else.&lt;/p>
&lt;p>Consumer behaviour reflects this scepticism in ways that directly threaten revenue. A Washington State University study found that products described as &amp;ldquo;AI-powered&amp;rdquo; are consistently less popular with consumers. Gartner surveys show &lt;strong>64% of customers prefer companies NOT use AI for customer service&lt;/strong>, with 53% willing to switch to competitors that don&amp;rsquo;t. Only &lt;strong>3% of smartphone owners&lt;/strong> are willing to pay extra for AI features.&lt;/p>
&lt;p>Three percent.&lt;/p>
&lt;p>When you&amp;rsquo;re building a business model that depends on massive consumer adoption and widespread willingness to pay premium prices, 3% willingness to pay is a death sentence. It means the revenue models underpinning all those billion-dollar valuations are built on fundamentally wrong assumptions about demand.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-enterprise-adoption-crisis">The enterprise adoption crisis&lt;/h3>
&lt;hr>
&lt;p>Maybe consumer scepticism doesn&amp;rsquo;t matter, you might argue. Maybe the real money is in enterprise. Except enterprise adoption is stalling for the same reasons, just with bigger price tags attached to the disappointment.&lt;/p>
&lt;p>The MIT NANDA study found that &lt;strong>95% of enterprise AI pilots yield zero measurable business return&lt;/strong> despite $30-40 billion in GenAI investment. BCG reports 74% of companies haven&amp;rsquo;t seen tangible value from AI initiatives. McKinsey&amp;rsquo;s surveys show 80%+ of organisations haven&amp;rsquo;t realised significant bottom-line AI impact. Gartner predicts &lt;strong>30% of GenAI projects will be abandoned by end of 2025&lt;/strong> due to poor data quality, escalating costs, or unclear value.&lt;/p>
&lt;p>Monte Carlo&amp;rsquo;s 2024 survey of data professionals reveals the pressure cooker: &lt;strong>100% feel pressure&lt;/strong> from leadership to implement GenAI strategy, yet &lt;strong>90% believe their leaders do NOT have realistic expectations&lt;/strong> for what&amp;rsquo;s technically feasible or capable of driving business value. Only 12% of companies have dedicated AI teams—meaning &lt;strong>84% rely on existing data teams&lt;/strong> to figure it out.&lt;/p>
&lt;p>This is the reality on the ground for data professionals. Leadership demands AI transformation. Budgets flow to AI initiatives. But the actual work of implementation falls to teams who know the technology isn&amp;rsquo;t mature enough, the data isn&amp;rsquo;t clean enough, the use cases aren&amp;rsquo;t well-defined enough. The pressure to deploy anyway—to show &amp;ldquo;innovation,&amp;rdquo; to justify spending, to keep up with competitors—leads to implementations that fail quietly, generate minimal value, and burn credibility.&lt;/p>
&lt;p>The case studies pile up. McDonald&amp;rsquo;s ended its three-year AI drive-thru partnership with IBM in June 2024 after viral videos showed the system adding 260 Chicken McNuggets to orders and repeatedly misinterpreting requests. Klarna initially claimed AI was &amp;ldquo;doing the work of 700 agents&amp;rdquo; after cutting 22% of its workforce, then reversed course with a &amp;ldquo;major hiring initiative&amp;rdquo; in May 2025. The CEO admitted AI was actually &amp;ldquo;doing work of 700 really bad agents&amp;rdquo;—quality decline cost customers, and the rollback cost approximately $15 million with no net gain.&lt;/p>
&lt;p>IBM&amp;rsquo;s Watson Health, once touted as transformative for cancer diagnosis, never reached production use, ran over budget, and was eventually sold for approximately $1 billion after multi-billion dollar investment. The Humane Ai Pin and Rabbit R1 represent hardware category failures, with slashed prices following weak sales and critical reviews noting the products solve &amp;ldquo;problems that don&amp;rsquo;t exist.&amp;rdquo;&lt;/p>
&lt;p>Each failure compounds the trust deficit. Each over-promised and under-delivered implementation makes the next pitch harder. Each news story about AI gone wrong creates more sceptics who start questioning whether this technology delivers value or just disrupts things that were working fine.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="the-2008-playbook-now-in-tech">The 2008 playbook, now in tech&lt;/h3>
&lt;hr>
&lt;p>The parallels between current AI investment patterns and pre-2008 financial crisis have moved from concerning to alarming. Deutsche Bank is now using &lt;strong>Synthetic Risk Transfers (SRTs) to hedge AI infrastructure loans&lt;/strong>—structures that analysts say &amp;ldquo;strongly resemble CDOs and CDSs that amplified the 2008 financial crisis.&amp;rdquo; Tech companies shift AI spending into Special Purpose Vehicles (SPVs) to obscure true costs, echoing Enron&amp;rsquo;s infamous off-balance-sheet accounting.&lt;/p>
&lt;p>The leverage buildup mirrors pre-crisis patterns. Goldman Sachs reports hyperscalers took on &lt;strong>$121 billion in debt&lt;/strong> over the past year—a 300%+ increase from typical levels. Meta raised $29 billion in debt for a single Louisiana data centre. Oracle operates at a &lt;strong>500% debt-to-equity ratio&lt;/strong> to compete in AI data centres. Morgan Stanley estimates Big Tech will spend $3 trillion on AI infrastructure through 2028, with cash flows covering only half.&lt;/p>
&lt;p>The dot-com parallels are equally sobering. In the late 1990s, &lt;strong>80+ million miles of fiber optic cable&lt;/strong> were laid based on inflated demand projections; 85-95% remained unused (&amp;ldquo;dark fiber&amp;rdquo;) years after the bubble burst. Today&amp;rsquo;s equivalent is the massive AI data centre buildout—Meta constructing a facility covering a &amp;ldquo;significant part of Manhattan&amp;rdquo;—predicated on demand growth that may never materialise. The $500 billion Stargate Project promises unprecedented AI infrastructure investment for enterprise AI adoption that&amp;rsquo;s already stalling.&lt;/p>
&lt;p>The NASDAQ took &lt;strong>15 years to recover&lt;/strong> from its 2000 peak. Cisco shares peaked at $80 and still haven&amp;rsquo;t returned to that level twenty-five years later. Those aren&amp;rsquo;t cautionary tales from ancient history—they&amp;rsquo;re previews of what happens when infrastructure investment radically outpaces actual demand.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-the-insiders-are-saying">What the insiders are saying&lt;/h3>
&lt;hr>
&lt;p>Perhaps most telling are the warnings from industry insiders themselves. Sam Altman, OpenAI&amp;rsquo;s CEO, acknowledges that investors are &amp;ldquo;overexcited about AI&amp;rdquo; and predicts &amp;ldquo;people will overinvest and lose money during this phase.&amp;rdquo; Google CEO Sundar Pichai warns of &amp;ldquo;elements of irrationality&amp;rdquo; and says &amp;ldquo;no company is going to be immune&amp;rdquo; if the bubble bursts. Jeff Bezos called the current environment &amp;ldquo;kind of an industrial bubble.&amp;rdquo;&lt;/p>
&lt;p>When the people building and funding AI companies publicly warn about bubble dynamics, when they&amp;rsquo;re literally saying investors will lose money, that&amp;rsquo;s not bearish speculation. That&amp;rsquo;s reality acknowledgment.&lt;/p>
&lt;p>Ray Dalio&amp;rsquo;s proprietary bubble indicator sits at approximately &lt;strong>80% of historical bubble peaks&lt;/strong>, with his AI chatbot estimating a 65-75% probability of meaningful correction in AI equities by end of 2026. Morgan Stanley&amp;rsquo;s Lisa Shalett warned of a &amp;ldquo;Cisco moment&amp;rdquo; within 24 months, describing the market as a &amp;ldquo;one-note narrative&amp;rdquo; entirely dependent on AI capital expenditures.&lt;/p>
&lt;p>The timing predictions are getting specific. Motley Fool analysts suggest the bubble could burst in 2025 due to unsustainable valuations. Ruchir Sharma of Rockefeller International predicts 2026 if interest rates rise. BeyondTrust researchers argue &amp;ldquo;artificial inflation of AI has already peaked in 2024.&amp;rdquo; A November 2025 Nature analysis noted that &amp;ldquo;many financial analysts now agree there is an &amp;lsquo;AI bubble,&amp;rsquo; some speculate it could finally burst in the next few months.&amp;rdquo;&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="preparing-for-what-comes-next">Preparing for what comes next&lt;/h3>
&lt;hr>
&lt;p>Whether the bubble bursts catastrophically or deflates gradually, data professionals can take specific steps to build resilient careers. The skills that remain human-essential include critical thinking and complex decision-making, cross-contextual reasoning (connecting insights across domains), ethical judgment in novel situations, relationship building and stakeholder management, and what researchers call &amp;ldquo;AI discernment&amp;rdquo;—knowing when to trust AI versus human judgment.&lt;/p>
&lt;p>Technical preparation should emphasise both breadth and depth. Core competencies remain essential: probability, statistics, Python, SQL. But add ML fundamentals including embeddings and model monitoring. Develop cloud platform expertise. Build governance and ethics capabilities that will be increasingly valuable as regulation tightens. Document measurable business impact, not just technical achievements—when budgets tighten, demonstrated ROI becomes the difference between retention and reduction.&lt;/p>
&lt;p>For data leaders, the most important preparation may be honest communication. Setting realistic expectations with executive leadership about what AI can deliver—and on what timeline—protects both the organisation and the team. The companies that navigate the potential correction successfully will be those treating AI as a tool for specific problems rather than a magic solution for every challenge.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="what-happens-when-authenticity-runs-out">What happens when authenticity runs out&lt;/h3>
&lt;hr>
&lt;p>I desperately want to be surrounded by real people with real hobbies and passions. I don&amp;rsquo;t care if a candidate loves remote work, I care about their humanity. I don&amp;rsquo;t care if a candidate tripped over their words a few times, I care about their authenticity.&lt;/p>
&lt;p>This isn&amp;rsquo;t nostalgia for pre-AI times. It&amp;rsquo;s recognition that authenticity, human connection, genuine judgment—these aren&amp;rsquo;t inefficiencies to optimise away. They&amp;rsquo;re the core value proposition. They&amp;rsquo;re what makes work meaningful, what makes organisations effective, what makes products worth buying.&lt;/p>
&lt;p>When companies deploy AI to eliminate these elements—to reduce humans to vectors, to automate judgment, to replace connection with efficiency—they&amp;rsquo;re not just failing to capture value. They&amp;rsquo;re actively destroying it. The companies that will survive aren&amp;rsquo;t those with the most sophisticated AI. They&amp;rsquo;re those that figured out how to use AI to enhance human capability rather than replace it, to preserve authenticity rather than eliminate it.&lt;/p>
&lt;p>The crisis isn&amp;rsquo;t coming. It&amp;rsquo;s already here, building pressure daily in the gap between investment and returns, between promises and delivery, between what AI can actually do and what we&amp;rsquo;ve been told it will become. The question isn&amp;rsquo;t whether there will be a reckoning. The question is whether you&amp;rsquo;ll be positioned to navigate it when it arrives—and whether the companies you work for will still treat you like a person when they do.&lt;/p></content:encoded><category>AI</category><category>Data Quality &amp; Governance</category><category>AI Bubble</category><category>Financial Crisis</category><category>Trust</category><category>Data Teams</category><category>AI Ethics</category><category>Career Development</category></item><item><title>Building AI Agents with Claude Code</title><link>https://ghostinthedata.info/posts/2025/2025-09-13-building-ai-agents-with-claude-code/</link><pubDate>Sat, 13 Sep 2025 10:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-09-13-building-ai-agents-with-claude-code/</guid><author>Chris Hillman</author><description>Learn to create, test, and deploy custom SQL validation agents using Claude Code, with hands-on examples for Starburst/Trino and Delta Lake syntax checking.</description><content:encoded>&lt;h3 id="introduction">Introduction&lt;/h3>
&lt;hr>
&lt;p>Imagine you&amp;rsquo;re reviewing a pull request with dozens of SQL files, each containing complex queries for your data pipeline. You spot inconsistent formatting, or syntax which doesn&amp;rsquo;t work with your infrastructure. Sound familiar?&lt;/p>
&lt;p>It&amp;rsquo;s common for data professionals to struggle with maintaining consistent SQL standards across their projects, especially when working with specialised platforms and it can be time consuming to review these elements within a peer review. It would be better use of time to focus on the hard thinking elements, like logic etc. However these small syntax or style issues, can be distracting. Well at least they are for me.&lt;/p>
&lt;p>That&amp;rsquo;s where Claude Code comes in. Claude code enables custom agents - we can then create agents that automatically validate your SQL code, enforce style guidelines, and catch platform-specific issues before they reach production. There are tools out there like SQLRuff and others that will likely do a better job that a ai agent. However the purpose of this is to have a basic understanding, but these agents could be applied to &amp;ldquo;Testing Agent&amp;rdquo; or &amp;ldquo;Data Modelling Agent&amp;rdquo; and far more.&lt;/p>
&lt;p>Here&amp;rsquo;s our objectives:&lt;/p>
&lt;ul>
&lt;li>Install and configure Claude Code for your development environment&lt;/li>
&lt;li>Create a custom SQL style guide agent that enforces dbt Labs standards&lt;/li>
&lt;li>Build a Starburst/Delta Lake syntax validator&lt;/li>
&lt;li>Set up automated validation workflows in VSCode&lt;/li>
&lt;li>Test your agents with real-world SQL examples&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h3 id="prerequisites">Prerequisites&lt;/h3>
&lt;hr>
&lt;p>Before starting this tutorial, ensure you have the following tools installed and configured:&lt;/p>
&lt;h4 id="required-tools">Required Tools&lt;/h4>
&lt;p>&lt;strong>Claude Code&lt;/strong>: Latest version (1.0.112+)&lt;/p>
&lt;ul>
&lt;li>This tutorial requires Claude Code with the new hooks format support&lt;/li>
&lt;li>Verify installation: &lt;code>claude --version&lt;/code>&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>SQLRuff&lt;/strong>: For SQL linting and formatting&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Install SQLRuff&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>pip install sqruff
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Verify installation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sqruff --version
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Git&lt;/strong>: For version control examples&lt;/p>
&lt;ul>
&lt;li>Most systems have this pre-installed&lt;/li>
&lt;li>Verify: &lt;code>git --version&lt;/code>&lt;/li>
&lt;/ul>
&lt;h4 id="optional-tools">Optional Tools&lt;/h4>
&lt;p>&lt;strong>Node.js&lt;/strong>: Only needed if using npm installation method&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Verify if installed&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>node --version
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>npm --version
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>VSCode&lt;/strong>: Recommended editor for best integration&lt;/p>
&lt;ul>
&lt;li>Download from: &lt;a href="https://code.visualstudio.com/" target="_blank" rel="noopener">https://code.visualstudio.com/&lt;/a>&lt;/li>
&lt;li>Ensure &lt;code>code&lt;/code> command is available in PATH&lt;/li>
&lt;/ul>
&lt;h4 id="system-requirements">System Requirements&lt;/h4>
&lt;ul>
&lt;li>&lt;strong>Operating System&lt;/strong>: Windows 10+, macOS 10.15+, or Linux&lt;/li>
&lt;li>&lt;strong>Memory&lt;/strong>: 4GB RAM minimum (8GB recommended)&lt;/li>
&lt;li>&lt;strong>Storage&lt;/strong>: 500MB free space for tools and project files&lt;/li>
&lt;li>&lt;strong>Network&lt;/strong>: Internet connection for initial setup and API calls&lt;/li>
&lt;/ul>
&lt;h4 id="installation-verification">Installation Verification&lt;/h4>
&lt;p>Run these commands to verify your setup:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Verify all required tools&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude --version &lt;span style="color:#75715e"># Should show 1.0.112 or higher&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sqruff --version &lt;span style="color:#75715e"># Should show SQLRuff version&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>git --version &lt;span style="color:#75715e"># Should show Git version&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Optional verifications&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>node --version &lt;span style="color:#75715e"># If using npm method&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>code --version &lt;span style="color:#75715e"># If using VSCode&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If any commands fail, install the missing tools before proceeding with the tutorial.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="how-to-install-claude-code-setup-guide-for-data-engineers">How to Install Claude Code: Setup Guide for Data Engineers&lt;/h3>
&lt;hr>
&lt;p>Before we dive into building SQL agents, let&amp;rsquo;s get Claude Code properly installed and configured. The setup process has evolved significantly in early 2025, with multiple installation options depending on your environment.&lt;/p>
&lt;h4 id="claude-code-installation-methods">Claude Code Installation Methods&lt;/h4>
&lt;p>The recommended approach is using the native binary installation, which provides the most stable experience:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># macOS/Linux/WSL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>curl -fsSL https://claude.ai/install.sh | bash
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Windows PowerShell&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>irm https://claude.ai/install.ps1 | iex
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Verify installation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude --version
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude doctor
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>If you prefer npm-based installation or need it for CI/CD environments:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Global installation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>npm install -g @anthropic-ai/claude-code
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Navigate to your project&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>cd your-sql-project
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="claude-code-authentication-setup">Claude Code Authentication Setup&lt;/h4>
&lt;p>You have two main options for authentication. For production environments, I recommend using the Anthropic API key approach:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Set environment variable&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>export ANTHROPIC_API_KEY&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;your-api-key-here&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Add to shell profile for persistence&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#39;export ANTHROPIC_API_KEY=&amp;#34;your-api-key-here&amp;#34;&amp;#39;&lt;/span> &amp;gt;&amp;gt; ~/.bashrc
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>source ~/.bashrc
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Test connection&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude -p &lt;span style="color:#e6db74">&amp;#34;Test API connection&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Alternatively, if you have a Claude Pro ($20/month) or Max ($200/month) subscription, you can use OAuth authentication during the initial setup process.&lt;/p>
&lt;h4 id="sql-agent-project-structure-setup">SQL Agent Project Structure Setup&lt;/h4>
&lt;p>Once Claude Code is installed, you&amp;rsquo;ll want to organise your project with a structure that supports SQL validation. Here&amp;rsquo;s the recommended layout:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 528 297"
 >
 &lt;g transform='translate(8,16)'>
&lt;path d='M 88,48 L 96,32' fill='none' stroke='currentColor'>&lt;/path>
&lt;circle cx='96' cy='32' r='6' stroke='currentColor' fill='#fff'>&lt;/circle>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='0' y='20' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='36' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='52' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='68' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='84' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='100' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='116' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='132' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='148' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='164' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='180' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='196' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='212' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='228' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='0' y='244' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='260' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='0' y='276' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='8' y='20' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='52' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='180' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='228' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='8' y='276' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='16' y='20' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='36' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='52' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='180' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='228' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='16' y='276' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='32' y='52' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='32' y='68' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='84' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='100' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='116' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='132' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='148' fill='currentColor' style='font-size:1em'>│&lt;/text>
&lt;text text-anchor='middle' x='32' y='164' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='196' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='212' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='32' y='244' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='32' y='260' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='32' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='40' y='20' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='40' y='52' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='40' y='68' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='84' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='132' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='180' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='40' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='212' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='228' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='40' y='244' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='40' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='48' y='20' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='48' y='52' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='48' y='68' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='84' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='132' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='164' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='180' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='48' y='196' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='212' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='228' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='48' y='244' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='260' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='48' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='56' y='20' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='56' y='36' fill='currentColor' style='font-size:1em'>U&lt;/text>
&lt;text text-anchor='middle' x='56' y='52' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='56' y='180' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='56' y='228' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='56' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='20' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>D&lt;/text>
&lt;text text-anchor='middle' x='64' y='52' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='64' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='64' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='64' y='100' fill='currentColor' style='font-size:1em'>├&lt;/text>
&lt;text text-anchor='middle' x='64' y='116' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='64' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='64' y='148' fill='currentColor' style='font-size:1em'>└&lt;/text>
&lt;text text-anchor='middle' x='64' y='164' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='64' y='180' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='64' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='64' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='64' y='244' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='64' y='260' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='64' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='72' y='20' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='72' y='52' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='72' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='72' y='84' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='72' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='72' y='116' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='72' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='72' y='148' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='72' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='72' y='180' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='72' y='196' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='72' y='212' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='72' y='244' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='72' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='72' y='276' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='80' y='4' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='80' y='20' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='80' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='80' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='80' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='80' y='100' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='80' y='116' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='80' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='80' y='148' fill='currentColor' style='font-size:1em'>─&lt;/text>
&lt;text text-anchor='middle' x='80' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='80' y='180' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='80' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='80' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='80' y='244' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='80' y='260' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='88' y='20' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='88' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='88' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='88' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='88' y='164' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='88' y='180' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='88' y='196' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='88' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='88' y='244' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='88' y='260' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='96' y='4' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='96' y='20' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='96' y='68' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='96' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='96' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='96' y='148' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='164' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='96' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='96' y='244' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='96' y='260' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='104' y='36' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='104' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='104' y='100' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='104' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='104' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='148' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='104' y='164' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='104' y='196' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='104' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='244' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='104' y='260' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='112' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='68' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='112' y='84' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='112' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='112' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='112' y='132' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='112' y='148' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='112' y='164' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='112' y='196' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='112' y='212' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='112' y='244' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='112' y='260' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='120' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='120' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='100' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='120' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='120' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='148' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='120' y='164' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='120' y='196' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='120' y='244' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='120' y='260' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='128' y='36' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='128' y='100' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='116' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='128' y='132' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='128' y='148' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='128' y='164' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='128' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='212' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='128' y='260' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='136' y='4' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='136' y='36' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='136' y='68' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='136' y='100' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='136' y='116' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='136' y='148' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='136' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='136' y='212' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='136' y='260' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='36' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='144' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='100' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='144' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='144' y='148' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='144' y='212' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='144' y='260' fill='currentColor' style='font-size:1em'>/&lt;/text>
&lt;text text-anchor='middle' x='152' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='152' y='100' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='152' y='116' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='152' y='148' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='152' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='160' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='160' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='160' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='160' y='148' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='160' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='168' y='100' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='168' y='116' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='168' y='148' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='176' y='100' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='176' y='116' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='176' y='148' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='184' y='100' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='184' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='184' y='148' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='192' y='100' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='192' y='116' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='192' y='148' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='200' y='116' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='200' y='148' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='208' y='100' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='208' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='208' y='148' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='216' y='100' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='216' y='116' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='224' y='100' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='224' y='116' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='232' y='68' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='84' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='100' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='232' y='116' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='232' y='132' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='164' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='196' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='212' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='228' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='232' y='276' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='240' y='20' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='240' y='36' fill='currentColor' style='font-size:1em'>#&lt;/text>
&lt;text text-anchor='middle' x='240' y='116' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='248' y='68' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='248' y='84' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='248' y='116' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='248' y='132' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='248' y='164' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='248' y='196' fill='currentColor' style='font-size:1em'>V&lt;/text>
&lt;text text-anchor='middle' x='248' y='212' fill='currentColor' style='font-size:1em'>E&lt;/text>
&lt;text text-anchor='middle' x='248' y='228' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='248' y='276' fill='currentColor' style='font-size:1em'>A&lt;/text>
&lt;text text-anchor='middle' x='256' y='20' fill='currentColor' style='font-size:1em'>M&lt;/text>
&lt;text text-anchor='middle' x='256' y='36' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='256' y='68' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='256' y='84' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='256' y='116' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='256' y='132' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='256' y='164' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='256' y='196' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='256' y='212' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='256' y='228' fill='currentColor' style='font-size:1em'>Q&lt;/text>
&lt;text text-anchor='middle' x='256' y='276' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='264' y='20' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='264' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='264' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='264' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='264' y='116' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='264' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='264' y='164' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='264' y='196' fill='currentColor' style='font-size:1em'>C&lt;/text>
&lt;text text-anchor='middle' x='264' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='264' y='228' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='264' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='272' y='20' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='272' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='272' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='272' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='272' y='132' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='272' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='272' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='272' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='272' y='276' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='280' y='20' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='280' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='280' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='280' y='84' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='280' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='280' y='164' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='280' y='196' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='280' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='280' y='228' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='280' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='288' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='288' y='68' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='288' y='84' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='288' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='288' y='164' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='288' y='196' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='288' y='212' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='288' y='228' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='296' y='20' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='296' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='296' y='164' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='296' y='228' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='296' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='304' y='20' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='304' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='304' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='304' y='84' fill='currentColor' style='font-size:1em'>S&lt;/text>
&lt;text text-anchor='middle' x='304' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='304' y='164' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='304' y='196' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='304' y='212' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='304' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='304' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='20' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='312' y='36' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='312' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='312' y='84' fill='currentColor' style='font-size:1em'>Q&lt;/text>
&lt;text text-anchor='middle' x='312' y='132' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='312' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='312' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='312' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='312' y='228' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='312' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='320' y='20' fill='currentColor' style='font-size:1em'>j&lt;/text>
&lt;text text-anchor='middle' x='320' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='84' fill='currentColor' style='font-size:1em'>L&lt;/text>
&lt;text text-anchor='middle' x='320' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='320' y='164' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='320' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='320' y='276' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='328' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='36' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='328' y='68' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='328' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='328' y='196' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='328' y='212' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='328' y='228' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='336' y='20' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='336' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='336' y='84' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='336' y='132' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='336' y='164' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='336' y='196' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='336' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='336' y='228' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='336' y='276' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='344' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='344' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='344' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='344' y='84' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='344' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='344' y='196' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='344' y='212' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='344' y='276' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='352' y='36' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='352' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='352' y='84' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='352' y='132' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='352' y='164' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='352' y='196' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='352' y='212' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='352' y='228' fill='currentColor' style='font-size:1em'>v&lt;/text>
&lt;text text-anchor='middle' x='352' y='276' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='360' y='20' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='360' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='360' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='360' y='84' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='360' y='132' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='360' y='164' fill='currentColor' style='font-size:1em'>k&lt;/text>
&lt;text text-anchor='middle' x='360' y='196' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='360' y='212' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='360' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='360' y='276' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='368' y='20' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='368' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='368' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='368' y='84' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='368' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='368' y='164' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='368' y='196' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='368' y='212' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='368' y='228' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='368' y='276' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='376' y='20' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='376' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='376' y='68' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='376' y='84' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='376' y='132' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='376' y='196' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='376' y='212' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='376' y='228' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='384' y='20' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='384' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='132' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='384' y='196' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='384' y='212' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='384' y='228' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='392' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='392' y='36' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='392' y='68' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='392' y='132' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='392' y='212' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='392' y='228' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='400' y='20' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='400' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='400' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='400' y='132' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='400' y='212' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='400' y='228' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='408' y='20' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='408' y='36' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='408' y='132' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='408' y='228' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='416' y='20' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='424' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='424' y='36' fill='currentColor' style='font-size:1em'>(&lt;/text>
&lt;text text-anchor='middle' x='432' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='432' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='440' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='448' y='36' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='456' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='464' y='36' fill='currentColor' style='font-size:1em'>g&lt;/text>
&lt;text text-anchor='middle' x='472' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='480' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='488' y='36' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='496' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='504' y='36' fill='currentColor' style='font-size:1em'>d&lt;/text>
&lt;text text-anchor='middle' x='512' y='36' fill='currentColor' style='font-size:1em'>)&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;p>The &lt;code>CLAUDE.md&lt;/code> file serves as your project&amp;rsquo;s main configuration. Here&amp;rsquo;s a template specifically designed for SQL validation projects:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span># SQL Validation Agent Project
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Project Overview
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>This project uses Claude Code to validate SQL code according to:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> dbt Labs SQL Style Guide
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Starburst/Trino syntax requirements 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Delta Lake connector best practices
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Key Commands
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Validate SQL: &lt;span style="color:#e6db74">`/sql-validate filename.sql`&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Style check: Use sql-style-guide agent
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Syntax check: Use starburst-validator agent
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Coding Standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Use SQLRuff for automated linting
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Follow dbt naming conventions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> 4-space indentation with trailing commas
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Lowercase for field names, keywords, and functions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Lines should not exceed 80 characters
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Database Context
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Primary engine: Starburst/Trino
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Data lake: Delta Lake format
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Metastore: AWS Glue / Hive Metastore
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;h4 id="essential-claude-code-configuration">Essential Claude Code Configuration&lt;/h4>
&lt;p>Create a &lt;code>.claude/settings.json&lt;/code> file to configure permissions and automation hooks:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;permissions&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;allow&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;Read&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Edit&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Bash&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;deny&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;Read(./.env)&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Read(./.env.*)&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Read(./secrets/**)&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;hooks&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;PostToolUse&amp;#34;&lt;/span>: [{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;matcher&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;tools&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;EditTool&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;hooks&amp;#34;&lt;/span>: [{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;command&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;command&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;sqruff lint \&amp;#34;$CLAUDE_FILE_PATHS\&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This configuration automatically runs SQLRuff linting whenever you edit SQL files, providing immediate feedback on style and syntax issues.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="creating-your-first-sql-agent-style-guide-validator">Creating Your First SQL Agent: Style Guide Validator&lt;/h3>
&lt;hr>
&lt;p>Now comes the exciting part - creating your first custom SQL agent. We&amp;rsquo;ll start with a style guide validator that enforces dbt Labs standards and general SQL best practices.&lt;/p>
&lt;h4 id="sql-agent-structure-and-configuration">SQL Agent Structure and Configuration&lt;/h4>
&lt;p>Create a new file at &lt;code>.claude/agents/sql-style-guide.md&lt;/code> with the following content:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>name: sql-style-guide
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>description: Use this agent for SQL code style validation, formatting checks, and dbt best practice enforcement
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>tools: Read, Write, Bash
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>color: blue
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>You are a SQL style guide specialist focusing on dbt Labs standards and general SQL best practices. When analyzing SQL code, you:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Core Responsibilities
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Validate dbt naming conventions (staging: &lt;span style="color:#e6db74">`stg_[source]__[entity]s.sql`&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Enforce 4-space indentation with trailing commas
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Check lowercase usage for field names, keywords, and functions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Validate 80-character line limits
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Ensure proper CTE organization and naming
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Verify explicit JOIN types and column aliasing
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Validation Checklist
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Field names use snake_case
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Reserved keywords in UPPERCASE
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Trailing commas in SELECT statements
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Explicit &lt;span style="color:#e6db74">`AS`&lt;/span> keywords for aliases
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Proper CTE ordering (imports first, then transformations)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> No SELECT * usage (except in final CTE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> GROUP BY uses numbers, not column names
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">- [ ]&lt;/span> Prefer UNION ALL over UNION
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Output Format
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>Provide findings as:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">1.&lt;/span> &lt;span style="font-weight:bold">**Style Issues Found**&lt;/span>: Bulleted list with line numbers
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">2.&lt;/span> &lt;span style="font-weight:bold">**Suggested Fixes**&lt;/span>: Code snippets showing corrections
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">3.&lt;/span> &lt;span style="font-weight:bold">**dbt Best Practices**&lt;/span>: Additional recommendations
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">4.&lt;/span> &lt;span style="font-weight:bold">**SQLRuff Command**&lt;/span>: Automated fix command if applicable
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Tool Usage
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Use Read to examine SQL files
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Use Bash to run &lt;span style="color:#e6db74">`sqruff lint filename.sql`&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Use Write to create corrected versions when requested
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="understanding-claude-code-agent-components">Understanding Claude Code Agent Components&lt;/h4>
&lt;p>Let&amp;rsquo;s break down what makes this SQL agent effective:&lt;/p>
&lt;p>&lt;strong>Frontmatter Configuration&lt;/strong>: The YAML frontmatter defines the agent&amp;rsquo;s identity and capabilities. The &lt;code>tools&lt;/code> field specifies which Claude Code tools the agent can access - in this case, Read (for examining files), Write (for creating corrections), and Bash (for running SQLRuff).&lt;/p>
&lt;p>&lt;strong>Role Definition&lt;/strong>: The agent has a clear, specific role as a &amp;ldquo;SQL style guide specialist.&amp;rdquo; This focused responsibility ensures it doesn&amp;rsquo;t try to handle tasks outside its expertise.&lt;/p>
&lt;p>&lt;strong>Validation Checklist&lt;/strong>: The structured checklist provides consistent, repeatable validation criteria. This ensures every SQL file gets the same thorough review.&lt;/p>
&lt;p>&lt;strong>Output Format&lt;/strong>: By defining a specific output structure, we ensure consistent, actionable feedback that developers can easily follow.&lt;/p>
&lt;h4 id="testing-your-sql-style-guide-agent">Testing Your SQL Style Guide Agent&lt;/h4>
&lt;p>Let&amp;rsquo;s create some test files to validate our agent works correctly. First, create a test file with intentional style issues:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- tests/bad_style.sql
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">select&lt;/span> u.id,u.name,p.company &lt;span style="color:#66d9ef">from&lt;/span> users u &lt;span style="color:#66d9ef">join&lt;/span> profiles p &lt;span style="color:#66d9ef">on&lt;/span> u.id&lt;span style="color:#f92672">=&lt;/span>p.user_id &lt;span style="color:#66d9ef">where&lt;/span> u.active&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now test your agent:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the sql-style-guide agent to analyze tests/bad_style.sql&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The agent should identify multiple issues:&lt;/p>
&lt;ul>
&lt;li>Missing spaces around operators&lt;/li>
&lt;li>Implicit JOIN instead of explicit INNER JOIN&lt;/li>
&lt;li>No trailing commas&lt;/li>
&lt;li>Line length exceeding 80 characters&lt;/li>
&lt;li>Missing proper indentation&lt;/li>
&lt;/ul>
&lt;/br>
&lt;h4 id="creating-good-sql-examples">Creating Good SQL Examples&lt;/h4>
&lt;p>It&amp;rsquo;s equally important to test with properly formatted SQL:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- tests/good_style.sql
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">WITH&lt;/span> users_with_profiles &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> users.id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> users.name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> users.email,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> profiles.company,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> profiles.created_at
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;users&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> &lt;span style="color:#960050;background-color:#1e0010">{{&lt;/span> &lt;span style="color:#66d9ef">ref&lt;/span>(&lt;span style="color:#e6db74">&amp;#39;profiles&amp;#39;&lt;/span>) &lt;span style="color:#960050;background-color:#1e0010">}}&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">ON&lt;/span> users.id &lt;span style="color:#f92672">=&lt;/span> profiles.user_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">WHERE&lt;/span> users.active &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> profiles.verified &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">TRUE&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>),
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">final&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> (
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> users_with_profiles
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#66d9ef">final&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>When you run the agent on this file, it should pass validation with minimal or no issues, confirming your agent correctly identifies good practices.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="advanced-sql-validation-starburst-and-delta-lake-agent">Advanced SQL Validation: Starburst and Delta Lake Agent&lt;/h3>
&lt;hr>
&lt;p>While style validation is important, syntax and platform-specific optimisation are equally critical. Let&amp;rsquo;s build a specialised agent that understands Starburst/Trino syntax and Delta Lake connector patterns.&lt;/p>
&lt;h4 id="the-starburst-delta-lake-validator-agent">The Starburst Delta Lake Validator Agent&lt;/h4>
&lt;p>Create &lt;code>.claude/agents/starburst-validator.md&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>name: starburst-validator 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>description: Validate Starburst/Trino syntax and Delta Lake connector usage patterns
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>tools: Read, Write, Bash
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>color: green
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>---
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>You are a Starburst/Trino syntax specialist with expertise in Delta Lake connector usage. Focus on:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Core Validation Areas
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Trino-specific SQL syntax and functions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Delta Lake time travel queries syntax
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Proper catalog/schema/table references
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Starburst Warp Speed indexing opportunities
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Memory usage optimization patterns
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Cross-connector query optimization
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Delta Lake Specific Checks
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Time travel syntax: &lt;span style="color:#e6db74">`FOR VERSION AS OF`&lt;/span> vs &lt;span style="color:#e6db74">`FOR TIMESTAMP AS OF`&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> System table usage: &lt;span style="color:#e6db74">`&amp;#34;table$history&amp;#34;`&lt;/span>, &lt;span style="color:#e6db74">`&amp;#34;table$partitions&amp;#34;`&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> OPTIMIZE and VACUUM operations
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Change data feed (CDF) syntax when applicable
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Column mapping and deletion vectors compatibility
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Starburst Features Validation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Connector-specific syntax (s3, glue, delta-lake)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Authentication method compatibility
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Query federation patterns across data sources
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Performance optimization suggestions (pushdown predicates)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Common Anti-patterns to Flag
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> SELECT * on large Delta tables
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Missing partition filters on time-partitioned data
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Inappropriate DISTINCT usage
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Cross-connector joins without proper optimization
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Missing time travel version management
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Output Format
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>Return structured analysis:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">1.&lt;/span> &lt;span style="font-weight:bold">**Syntax Validation**&lt;/span>: Trino compatibility check
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">2.&lt;/span> &lt;span style="font-weight:bold">**Delta Lake Usage**&lt;/span>: Connector-specific recommendations 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">3.&lt;/span> &lt;span style="font-weight:bold">**Performance Optimization**&lt;/span>: Query improvement suggestions
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">4.&lt;/span> &lt;span style="font-weight:bold">**Security Review**&lt;/span>: Access pattern analysis
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">5.&lt;/span> &lt;span style="font-weight:bold">**Next Steps**&lt;/span>: Actionable improvements with examples
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Tool Commands
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Use Read for SQL file analysis
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Use Bash for &lt;span style="color:#e6db74">`trino --execute &amp;#34;EXPLAIN (TYPE VALIDATE) query&amp;#34;`&lt;/span> when available
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Check system tables with queries like &lt;span style="color:#e6db74">`SELECT * FROM &amp;#34;table$properties&amp;#34;`&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="delta-lake-time-travel-query-patterns">Delta Lake Time Travel Query Patterns&lt;/h4>
&lt;p>This agent specialises in recognising and validating Delta Lake-specific syntax. Here are some examples it should handle:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Good: Proper time travel syntax
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> example.testdb.customer_orders 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">VERSION&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">OF&lt;/span> &lt;span style="color:#ae81ff">3&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Good: Timestamp-based time travel
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> example.testdb.customer_orders 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">OF&lt;/span> &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2022-03-23 09:59:29.803 America/Los_Angeles&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Good: System table monitoring
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#e6db74">&amp;#34;customer_orders$history&amp;#34;&lt;/span> &lt;span style="color:#66d9ef">ORDER&lt;/span> &lt;span style="color:#66d9ef">BY&lt;/span> &lt;span style="color:#66d9ef">version&lt;/span> &lt;span style="color:#66d9ef">DESC&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#f92672">*&lt;/span> &lt;span style="color:#66d9ef">FROM&lt;/span> &lt;span style="color:#e6db74">&amp;#34;customer_orders$partitions&amp;#34;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The agent should flag problematic patterns like:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Bad: No partition filtering on large table
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> customer_id, order_total
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> customer_orders 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> order_date &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2023-01-01&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Better: Partition filtering
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> customer_id, order_total
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> customer_orders 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> date_partition &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2023-01-01&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> customer_id &lt;span style="color:#66d9ef">IN&lt;/span> (&lt;span style="color:#ae81ff">1&lt;/span>, &lt;span style="color:#ae81ff">2&lt;/span>, &lt;span style="color:#ae81ff">3&lt;/span>);
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;h4 id="starburst-performance-optimisation-detection">Starburst Performance Optimisation Detection&lt;/h4>
&lt;p>One of the most valuable features of this agent is its ability to suggest performance improvements:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- The agent should suggest predicate pushdown optimization
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.name, o.total
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> customers &lt;span style="color:#66d9ef">c&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">INNER&lt;/span> &lt;span style="color:#66d9ef">JOIN&lt;/span> orders o &lt;span style="color:#66d9ef">ON&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.id &lt;span style="color:#f92672">=&lt;/span> o.customer_id
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> &lt;span style="color:#66d9ef">c&lt;/span>.region &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;US&amp;#39;&lt;/span> &lt;span style="color:#75715e">-- Filter pushed down
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">AND&lt;/span> o.status &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;completed&amp;#39;&lt;/span>;
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="building-automated-sql-validation-workflows">Building Automated SQL Validation Workflows&lt;/h3>
&lt;hr>
&lt;p>Individual agents are powerful, but the real magic happens when you orchestrate them into automated workflows. Let&amp;rsquo;s build a comprehensive validation system that runs automatically as you work.&lt;/p>
&lt;h4 id="vscode-integration-for-sql-validation">VSCode Integration for SQL Validation&lt;/h4>
&lt;p>First, let&amp;rsquo;s set up VSCode tasks that make it easy to run validation with keyboard shortcuts. Create or update &lt;code>.vscode/tasks.json&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-json" data-lang="json">&lt;span style="display:flex;">&lt;span>{
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;version&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;2.0.0&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;tasks&amp;#34;&lt;/span>: [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;label&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;Validate All SQL Files&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;shell&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;command&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;claude&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;args&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;-p&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Use sql-style-guide and starburst-validator agents to validate all .sql files in workspace. Generate summary report.&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;group&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;kind&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;build&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;isDefault&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;presentation&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;echo&amp;#34;&lt;/span>: &lt;span style="color:#66d9ef">true&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;reveal&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;always&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;panel&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;new&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> },
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;label&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;SQL Style Check&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;type&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;shell&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;command&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;claude&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;args&amp;#34;&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;-p&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Use sql-style-guide agent to check style of ${file}&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">&amp;#34;group&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;test&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Now you can press &lt;code>Ctrl+Shift+P&lt;/code> (or &lt;code>Cmd+Shift+P&lt;/code> on Mac), type &amp;ldquo;Tasks: Run Task&amp;rdquo;, and select &amp;ldquo;Validate All SQL Files&amp;rdquo; to run comprehensive validation across your entire project.&lt;/p>
&lt;h4 id="automated-file-watching-with-claude-code">Automated File Watching with Claude Code&lt;/h4>
&lt;p>For real-time validation, create &lt;code>.claude/hooks.mjs&lt;/code> to automatically check files as you save them:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-javascript" data-lang="javascript">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">export&lt;/span> &lt;span style="color:#66d9ef">async&lt;/span> &lt;span style="color:#66d9ef">function&lt;/span> &lt;span style="color:#a6e22e">postEdit&lt;/span>({ &lt;span style="color:#a6e22e">filePath&lt;/span>, &lt;span style="color:#a6e22e">success&lt;/span> }) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> (&lt;span style="color:#f92672">!&lt;/span>&lt;span style="color:#a6e22e">success&lt;/span>) &lt;span style="color:#66d9ef">return&lt;/span>;
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">// Auto-validate SQL files on save
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> (&lt;span style="color:#a6e22e">filePath&lt;/span>.&lt;span style="color:#a6e22e">match&lt;/span>(&lt;span style="color:#e6db74">/\.sql$/&lt;/span>)) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#a6e22e">console&lt;/span>.&lt;span style="color:#a6e22e">log&lt;/span>(&lt;span style="color:#e6db74">`[Auto-validation] Checking &lt;/span>&lt;span style="color:#e6db74">${&lt;/span>&lt;span style="color:#a6e22e">filePath&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">`&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">// Run style validation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#a6e22e">execSync&lt;/span>(&lt;span style="color:#e6db74">`claude -p &amp;#34;Use sql-style-guide agent to quickly validate &lt;/span>&lt;span style="color:#e6db74">${&lt;/span>&lt;span style="color:#a6e22e">filePath&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;`&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">// Run syntax validation for Starburst files
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> (&lt;span style="color:#66d9ef">await&lt;/span> &lt;span style="color:#a6e22e">fileContains&lt;/span>(&lt;span style="color:#a6e22e">filePath&lt;/span>, &lt;span style="color:#e6db74">/starburst|trino|delta/i&lt;/span>)) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#a6e22e">execSync&lt;/span>(&lt;span style="color:#e6db74">`claude -p &amp;#34;Use starburst-validator agent to check &lt;/span>&lt;span style="color:#e6db74">${&lt;/span>&lt;span style="color:#a6e22e">filePath&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;`&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e">// Auto-format with SQLRuff
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> (&lt;span style="color:#a6e22e">filePath&lt;/span>.&lt;span style="color:#a6e22e">match&lt;/span>(&lt;span style="color:#e6db74">/\.sql$/&lt;/span>)) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#a6e22e">execSync&lt;/span>(&lt;span style="color:#e6db74">`sqruff fix &amp;#34;&lt;/span>&lt;span style="color:#e6db74">${&lt;/span>&lt;span style="color:#a6e22e">filePath&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;`&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> } &lt;span style="color:#66d9ef">catch&lt;/span> (&lt;span style="color:#a6e22e">error&lt;/span>) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#a6e22e">console&lt;/span>.&lt;span style="color:#a6e22e">log&lt;/span>(&lt;span style="color:#e6db74">`[Warning] SQLRuff formatting failed: &lt;/span>&lt;span style="color:#e6db74">${&lt;/span>&lt;span style="color:#a6e22e">error&lt;/span>.&lt;span style="color:#a6e22e">message&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">`&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">async&lt;/span> &lt;span style="color:#66d9ef">function&lt;/span> &lt;span style="color:#a6e22e">fileContains&lt;/span>(&lt;span style="color:#a6e22e">filePath&lt;/span>, &lt;span style="color:#a6e22e">pattern&lt;/span>) {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">const&lt;/span> &lt;span style="color:#a6e22e">content&lt;/span> &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">await&lt;/span> &lt;span style="color:#a6e22e">fs&lt;/span>.&lt;span style="color:#a6e22e">readFile&lt;/span>(&lt;span style="color:#a6e22e">filePath&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;utf8&amp;#39;&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#a6e22e">pattern&lt;/span>.&lt;span style="color:#a6e22e">test&lt;/span>(&lt;span style="color:#a6e22e">content&lt;/span>);
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>}
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This hook automatically runs validation whenever you save a SQL file, providing immediate feedback without interrupting your workflow.&lt;/p>
&lt;h4 id="batch-sql-processing-command">Batch SQL Processing Command&lt;/h4>
&lt;p>For comprehensive project-wide validation, create &lt;code>.claude/commands/scan-sql-files.md&lt;/code>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>Scan and validate all SQL files in the project:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">1.&lt;/span> &lt;span style="font-weight:bold">**Discovery Phase**&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Find all .sql files in project directory
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Categorize by type (queries, migrations, models, tests)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Identify potential Starburst/Delta Lake usage patterns
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">2.&lt;/span> &lt;span style="font-weight:bold">**Validation Phase**&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Run sql-style-guide agent on each file
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Run starburst-validator agent on files with Trino syntax
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Collect all findings into structured report
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">3.&lt;/span> &lt;span style="font-weight:bold">**Reporting Phase**&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Generate summary dashboard showing:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Total files processed
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Style issues found and severity
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Syntax errors and warnings 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Performance optimization opportunities
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Files requiring immediate attention
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">4.&lt;/span> &lt;span style="font-weight:bold">**Action Items**&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Prioritized list of fixes needed
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Automated fix commands where possible
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">-&lt;/span> Manual review recommendations
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>Confirm before starting each phase. Process files in batches of 10 to avoid context overflow.
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Use this command with: &lt;code>claude &amp;quot;/scan-sql-files&amp;quot;&lt;/code>&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="testing-sql-agents-production-ready-validation">Testing SQL Agents: Production-Ready Validation&lt;/h3>
&lt;hr>
&lt;p>Building agents is just the beginning. To make them truly valuable for your team, you need a systematic approach to testing and refinement.&lt;/p>
&lt;h4 id="creating-a-comprehensive-sql-test-suite">Creating a Comprehensive SQL Test Suite&lt;/h4>
&lt;p>Start by building a comprehensive test suite that covers various scenarios your agents will encounter:&lt;/p>
&lt;p>&lt;strong>Unix/Linux/macOS:&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create test directory structure&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>mkdir -p tests/&lt;span style="color:#f92672">{&lt;/span>good,bad,edge-cases&lt;span style="color:#f92672">}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Good examples (should pass validation)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;-- Well-formatted dbt model
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">WITH users_clean AS (
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> SELECT
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> id,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> LOWER(email) AS email,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> created_at
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> FROM {{ ref(&amp;#39;raw_users&amp;#39;) }}
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> WHERE email IS NOT NULL
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">SELECT * FROM users_clean&amp;#34;&lt;/span> &amp;gt; tests/good/dbt_model.sql
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Bad examples (should fail validation)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;select * from users where email like &amp;#39;%@gmail.com&amp;#39;&amp;#34;&lt;/span> &amp;gt; tests/bad/style_issues.sql
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Windows PowerShell:&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-powershell" data-lang="powershell">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create test directory structure&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>mkdir tests; mkdir tests\good; mkdir tests\bad; mkdir tests\edge-cases
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Good examples (should pass validation)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">@&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">-- Well-formatted dbt model
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">WITH users_clean AS (
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> SELECT
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> id,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> LOWER(email) AS email,
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> created_at
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> FROM {{ ref(&amp;#39;raw_users&amp;#39;) }}
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> WHERE email IS NOT NULL
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">SELECT * FROM users_clean
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">&amp;#34;@&lt;/span> | Out-File -FilePath &lt;span style="color:#e6db74">&amp;#34;tests\good\dbt_model.sql&amp;#34;&lt;/span> -Encoding UTF8
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Bad examples (should fail validation)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">&amp;#34;select * from users where email like &amp;#39;%@gmail.com&amp;#39;&amp;#34;&lt;/span> | Out-File -FilePath &lt;span style="color:#e6db74">&amp;#34;tests\bad\style_issues.sql&amp;#34;&lt;/span> -Encoding UTF8
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Windows Command Prompt:&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-cmd" data-lang="cmd">&lt;span style="display:flex;">&lt;span># Create test directory structure
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">mkdir&lt;/span> tests &amp;amp;&amp;amp; &lt;span style="color:#66d9ef">mkdir&lt;/span> tests\good &amp;amp;&amp;amp; &lt;span style="color:#66d9ef">mkdir&lt;/span> tests\bad &amp;amp;&amp;amp; &lt;span style="color:#66d9ef">mkdir&lt;/span> tests\edge-cases
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span># Create files manually or use a text editor for complex SQL examples
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Edge Case Example (All Platforms):&lt;/strong>
Create &lt;code>tests/edge-cases/complex_time_travel.sql&lt;/code> with this content:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-sql" data-lang="sql">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">-- Complex Delta Lake time travel query
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">SELECT&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_total,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> delta_lake.sales.orders 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">VERSION&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">OF&lt;/span> &lt;span style="color:#ae81ff">5&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> date_partition &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2023-01-01&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> customer_region &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;US&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">UNION&lt;/span> &lt;span style="color:#66d9ef">ALL&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">SELECT&lt;/span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> customer_id,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_total,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> order_date 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FROM&lt;/span> delta_lake.sales.orders
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">FOR&lt;/span> &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span> &lt;span style="color:#66d9ef">AS&lt;/span> &lt;span style="color:#66d9ef">OF&lt;/span> &lt;span style="color:#66d9ef">TIMESTAMP&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2023-06-01 00:00:00&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">WHERE&lt;/span> date_partition &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;2023-06-01&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">AND&lt;/span> customer_region &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;EU&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="automated-sql-agent-testing-workflow">Automated SQL Agent Testing Workflow&lt;/h4>
&lt;p>Create a testing command that systematically validates your agents:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Test style guide agent&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the sql-style-guide agent to analyze all files in tests/good/ - these should pass with minimal issues&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the sql-style-guide agent to analyze all files in tests/bad/ - these should identify specific problems&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Test syntax validator&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the starburst-validator agent to check tests/edge-cases/complex_time_travel.sql for Delta Lake best practices&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="measuring-sql-agent-effectiveness">Measuring SQL Agent Effectiveness&lt;/h4>
&lt;p>Track key metrics to understand how well your agents are performing:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Accuracy&lt;/strong>: How often do agents correctly identify real issues?&lt;/li>
&lt;li>&lt;strong>Coverage&lt;/strong>: What percentage of actual problems do they catch?&lt;/li>
&lt;li>&lt;strong>False Positives&lt;/strong>: How often do they flag correct code as problematic?&lt;/li>
&lt;li>&lt;strong>Actionability&lt;/strong>: How clear and helpful are their recommendations?&lt;/li>
&lt;/ol>
&lt;p>Keep a log of agent performance:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span># Agent Performance Log
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## sql-style-guide Agent
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> **Date**: 2025-01-15
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Files Tested**: 25
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Issues Found**: 47
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **False Positives**: 3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Accuracy**: 94%
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Notes**: Occasionally flags dbt macros as style violations
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## starburst-validator Agent 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> **Date**: 2025-01-15
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Files Tested**: 15
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Performance Suggestions**: 12
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Syntax Errors Caught**: 3
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> **Notes**: Excellent at catching missing partition filters
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="iterative-sql-agent-improvement-process">Iterative SQL Agent Improvement Process&lt;/h4>
&lt;p>Based on your testing results, continuously refine your agents:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Analyse Feedback&lt;/strong>: Review false positives and missed issues&lt;/li>
&lt;li>&lt;strong>Update Agent Instructions&lt;/strong>: Refine the validation criteria&lt;/li>
&lt;li>&lt;strong>Add New Test Cases&lt;/strong>: Include edge cases you discover&lt;/li>
&lt;li>&lt;strong>Validate Changes&lt;/strong>: Test updated agents against your full test suite&lt;/li>
&lt;li>&lt;strong>Document Learnings&lt;/strong>: Keep track of what works and what doesn&amp;rsquo;t&lt;/li>
&lt;/ol>
&lt;p>For example, if your style guide agent keeps flagging legitimate dbt macros, you might update its instructions:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-markdown" data-lang="markdown">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">## Additional Considerations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e">&lt;/span>&lt;span style="color:#66d9ef">-&lt;/span> Recognize dbt macros ({{ macro_name() }}) as valid syntax
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Allow longer lines for complex macro calls
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">-&lt;/span> Understand dbt ref() and source() functions
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="real-world-sql-agent-implementation">Real-World SQL Agent Implementation&lt;/h3>
&lt;hr>
&lt;p>Let&amp;rsquo;s walk through a complete example of how these agents work together in a real data engineering project.&lt;/p>
&lt;h4 id="scenario-data-pipeline-code-review-with-claude-code">Scenario: Data Pipeline Code Review with Claude Code&lt;/h4>
&lt;p>Imagine you&amp;rsquo;re working on a data pipeline that processes customer orders using Starburst and Delta Lake. You&amp;rsquo;ve just received a pull request with several SQL files that need review.&lt;/p>
&lt;p>&lt;strong>Step 1: Initial SQL Validation&lt;/strong>&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Run comprehensive validation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use both sql-style-guide and starburst-validator agents to review all .sql files in the current directory. Provide a summary of issues found and prioritize them by severity.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Step 2: Multi-Agent Coordination&lt;/strong>&lt;/p>
&lt;p>The agents work together to provide comprehensive feedback:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Style Guide Agent&lt;/strong> identifies formatting inconsistencies, naming convention violations, and dbt best practice issues&lt;/li>
&lt;li>&lt;strong>Starburst Validator&lt;/strong> catches performance problems, syntax errors, and Delta Lake optimisation opportunities&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Step 3: Automated SQL Fixes&lt;/strong>&lt;/p>
&lt;p>For issues that can be automatically resolved:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Fix style issues with SQLRuff&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>sqruff fix *.sql
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Apply agent suggestions&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the sql-style-guide agent to create corrected versions of files with style issues&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Step 4: Manual SQL Review&lt;/strong>&lt;/p>
&lt;p>For complex issues requiring human judgment:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Get detailed analysis&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use the starburst-validator agent to provide detailed performance optimization recommendations for customer_orders_analysis.sql, including specific code examples&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h4 id="integration-with-existing-data-engineering-tools">Integration with Existing Data Engineering Tools&lt;/h4>
&lt;p>Your Claude Code agents work seamlessly with your existing data engineering stack:&lt;/p>
&lt;p>&lt;strong>dbt Integration&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Configure SQLRuff for dbt + Trino&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># SQLRuff is specifically designed for SQL linting&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;[tool.sqruff]
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">dialect = \&amp;#34;trino\&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">templater = \&amp;#34;dbt\&amp;#34;&amp;#34;&lt;/span> &amp;gt; pyproject.toml
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Validate dbt models&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>claude &lt;span style="color:#e6db74">&amp;#34;Use sql-style-guide agent to validate all models in models/ directory for dbt best practices&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>Git Hooks for SQL Validation&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-bash" data-lang="bash">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Pre-commit validation&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>echo &lt;span style="color:#e6db74">&amp;#34;#!/bin/bash
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">files=\$(git diff --cached --name-only --diff-filter=ACM | grep &amp;#39;\\.sql\$&amp;#39;)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">if [ -n \&amp;#34;\$files\&amp;#34; ]; then
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> claude \&amp;#34;Use sql-style-guide and starburst-validator agents to validate staged SQL files: \$files\&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74">fi&amp;#34;&lt;/span> &amp;gt; .git/hooks/pre-commit
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>chmod +x .git/hooks/pre-commit
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>&lt;strong>CI/CD Pipeline Integration&lt;/strong>:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># GitHub Actions example&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>- &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Validate SQL Files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> claude &amp;#34;Use sql-style-guide and starburst-validator agents to validate all SQL files. Exit with error code if critical issues found.&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;/br>
&lt;/br>
&lt;h3 id="troubleshooting-claude-code-sql-agents">Troubleshooting Claude Code SQL Agents&lt;/h3>
&lt;hr>
&lt;p>Even with well-designed agents, you&amp;rsquo;ll encounter challenges. Here are the most common issues and how to resolve them.&lt;/p>
&lt;h4 id="common-claude-code-installation-problems">Common Claude Code Installation Problems&lt;/h4>
&lt;p>&lt;strong>VSCode Extension Not Loading&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Ensure the &lt;code>code&lt;/code> command is in your PATH&lt;/li>
&lt;li>Run &lt;code>claude&lt;/code> from VSCode&amp;rsquo;s integrated terminal&lt;/li>
&lt;li>On Windows, check for conflicts between WSL and Windows Node.js installations&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Permission Errors&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Configure auto-approval in &lt;code>.claude/settings.json&lt;/code> for trusted operations&lt;/li>
&lt;li>Use &lt;code>--dangerously-skip-permissions&lt;/code> flag for automated scripts (with caution)&lt;/li>
&lt;li>In production, always use manual approval for security&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Context Window Overflow&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Use &lt;code>/clear&lt;/code> command frequently to reset context&lt;/li>
&lt;li>Process large projects in batches&lt;/li>
&lt;li>Monitor the context indicator in Claude Code&amp;rsquo;s status bar&lt;/li>
&lt;/ul>
&lt;h4 id="sql-agent-development-challenges">SQL Agent Development Challenges&lt;/h4>
&lt;p>&lt;strong>Agents Not Being Invoked&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Make agent descriptions action-oriented and specific&lt;/li>
&lt;li>Use clear, unambiguous language in the description field&lt;/li>
&lt;li>Test agent invocation with simple commands first&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Tool Access Issues&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Explicitly configure tool permissions in agent frontmatter&lt;/li>
&lt;li>Verify the agent has access to required tools (Read, Write, Bash)&lt;/li>
&lt;li>Check &lt;code>.claude/settings.json&lt;/code> for permission conflicts&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Inconsistent Results&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Provide detailed, specific instructions in agent definitions&lt;/li>
&lt;li>Use structured output formats to ensure consistency&lt;/li>
&lt;li>Include examples of good and bad patterns in agent instructions&lt;/li>
&lt;/ul>
&lt;h4 id="sql-validation-performance-optimisation">SQL Validation Performance Optimisation&lt;/h4>
&lt;p>&lt;strong>Slow Validation&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Break large queries into smaller validation chunks&lt;/li>
&lt;li>Use lightweight validation for real-time feedback&lt;/li>
&lt;li>Schedule comprehensive validation for batch processing&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Memory Usage&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Clear context regularly during large validation runs&lt;/li>
&lt;li>Process files in smaller batches&lt;/li>
&lt;li>Use the Task tool for clean delegation between agents&lt;/li>
&lt;/ul>
&lt;h4 id="team-collaboration-with-sql-agents">Team Collaboration with SQL Agents&lt;/h4>
&lt;p>&lt;strong>Maintaining Consistency&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Version control all agent definitions and settings&lt;/li>
&lt;li>Document agent capabilities and limitations&lt;/li>
&lt;li>Establish clear guidelines for when to use each agent&lt;/li>
&lt;/ul>
&lt;p>&lt;strong>Training and Adoption&lt;/strong>:&lt;/p>
&lt;ul>
&lt;li>Create simple examples and tutorials for team members&lt;/li>
&lt;li>Start with basic validation and gradually add complexity&lt;/li>
&lt;li>Collect feedback and iterate based on real usage patterns&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h3 id="conclusion-mastering-sql-validation-with-claude-code">Conclusion: Mastering SQL Validation with Claude Code&lt;/h3>
&lt;hr>
&lt;p>You&amp;rsquo;ve now built a comprehensive SQL validation system using Claude Code that can transform how your team approaches code quality. Your custom agents provide automated style checking, syntax validation, and performance optimisation suggestions that would typically require hours of manual review.&lt;/p>
&lt;p>The key benefits you&amp;rsquo;ve implemented include:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Consistent SQL Code Quality&lt;/strong>: Automated enforcement of dbt Labs standards and SQL best practices&lt;/li>
&lt;li>&lt;strong>Platform-Specific Validation&lt;/strong>: Specialised checking for Starburst/Trino and Delta Lake patterns&lt;/li>
&lt;li>&lt;strong>Automated Workflows&lt;/strong>: Real-time validation that integrates seamlessly with your development process&lt;/li>
&lt;li>&lt;strong>Scalable Review Process&lt;/strong>: Batch validation capabilities for large codebases&lt;/li>
&lt;/ul>
&lt;p>Remember that building effective SQL agents is an iterative process. Start with the basic implementations provided in this tutorial, then refine them based on your team&amp;rsquo;s specific needs and the patterns you encounter in your SQL code.&lt;/p>
&lt;p>Your next steps should be:&lt;/p>
&lt;ol>
&lt;li>Deploy these agents in a development environment and test them with your actual SQL files&lt;/li>
&lt;li>Gather feedback from your team and refine the validation criteria&lt;/li>
&lt;li>Integrate the agents into your CI/CD pipeline for automated code review&lt;/li>
&lt;li>Explore additional agents for other aspects of your data engineering workflow&lt;/li>
&lt;/ol>
&lt;p>The foundation you&amp;rsquo;ve built here can be extended to enforce enterprise patterns, or processes - so you might have DBT specific macros that are used for transformation standards, or specific metadata columns that you use on all tables. You might find that there are common performance issues with code, so showing techniques to break automatically make it more performant. Enforce Partition usage in code. Automate or suggest documentation for all the models you create.
The possibilities are endless when you have AI-powered agents working alongside your development process, but as always use with caution. Tools like this are good to save time, but still need validation, review of what it does. Don&amp;rsquo;t blindly use what comes out.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h3 id="resources-and-further-reading">Resources and Further Reading&lt;/h3>
&lt;hr>
&lt;ul>
&lt;li>&lt;a href="https://docs.anthropic.com/claude/docs" target="_blank" rel="noopener">Claude Code Official Documentation&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://github.com/dbt-labs/corp/blob/main/dbt_style_guide.md" target="_blank" rel="noopener">dbt Labs SQL Style Guide&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://docs.starburst.io/" target="_blank" rel="noopener">Starburst Documentation&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://docs.delta.io/" target="_blank" rel="noopener">Delta Lake Documentation&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://github.com/quarylabs/sqruff" target="_blank" rel="noopener">SQLRuff Documentation&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://trino.io/docs/current/sql.html" target="_blank" rel="noopener">Trino SQL Reference&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://docs.anthropic.com/claude/docs/agents" target="_blank" rel="noopener">SQL Agent Best Practices Guide&lt;/a>&lt;/li>
&lt;li>&lt;a href="https://github.com/dbt-labs/dbt-style-guide" target="_blank" rel="noopener">Data Engineering SQL Standards&lt;/a>&lt;/li>
&lt;/ul></content:encoded><category>AI</category><category>Data Engineering</category><category>Claude Code</category><category>AI Agents</category><category>Starburst</category><category>Delta Lake</category><category>Trino</category><category>Data Validation</category><category>dbt</category><category>Data Engineering</category><category>AI Tools</category><category>VSCode</category></item><item><title>AI Simplified: Understanding LLMs, Workflows, and Agents</title><link>https://ghostinthedata.info/posts/2025/2025-05-24-ai-simplified/</link><pubDate>Sat, 24 May 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-05-24-ai-simplified/</guid><author>Chris Hillman</author><description>A clear explanation of AI buzzwords like LLMs, RAG, and AI Agents, with practical examples showing how these technologies work together in real-world applications.</description><content:encoded>&lt;h2 id="ai-buzzwords-demystified">AI Buzzwords Demystified&lt;/h2>
&lt;p>If you&amp;rsquo;ve been following AI developments lately, you&amp;rsquo;ve probably encountered terms like LLMs, RAG, ReAct, and AI Agents. While these technologies are transforming how we interact with AI, the terminology can be overwhelming. In this post, I&amp;rsquo;ll break down these concepts into digestible explanations with practical examples.&lt;/p>
&lt;p>Let&amp;rsquo;s start with the foundation and progressively build up to more complex systems.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="large-language-models-llms-the-foundation">Large Language Models (LLMs): The Foundation&lt;/h2>
&lt;hr>
&lt;p>At the core of today&amp;rsquo;s AI revolution are Large Language Models (LLMs). Popular applications such as ChatGPT and Claude are built on top of these powerful models. They excel at generating and manipulating text based on the prompts we provide.&lt;/p>
&lt;p>The basic operation is simple:&lt;/p>
&lt;p>&lt;code>Input &amp;gt; LLM &amp;gt; Output&lt;/code>&lt;/p>
&lt;p>You ask a question or provide instructions, and the LLM generates a response based on patterns it learned during training. For example, if you ask an LLM to draft an email to Luke Skywalker inviting him for coffee, it can generate a contextually appropriate message.&lt;/p>
&lt;p>However, LLMs have a fundamental limitation: they only know what they&amp;rsquo;ve been trained on. They don&amp;rsquo;t have access to:&lt;/p>
&lt;ul>
&lt;li>Your personal calendar&lt;/li>
&lt;li>Your company&amp;rsquo;s internal documents&lt;/li>
&lt;li>Real-time information (like today&amp;rsquo;s weather)&lt;/li>
&lt;li>The ability to perform actions in the world&lt;/li>
&lt;/ul>
&lt;p>This is where more sophisticated systems come into play.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="ai-workflows-adding-capabilities">AI Workflows: Adding Capabilities&lt;/h2>
&lt;hr>
&lt;p>AI workflows extend LLMs by connecting them to external data sources and tools. This allows them to access information beyond their training data and perform specific tasks.&lt;/p>
&lt;p>Let&amp;rsquo;s continue with our example: What if you want to know when your next coffee chat with Luke is scheduled?&lt;/p>
&lt;p>A basic LLM would fail at this task because it has no access to your calendar. But an AI workflow could:&lt;/p>
&lt;p>&lt;code>Input &amp;gt; LLM &amp;gt; Calendar Search Query &amp;gt; Calendar Results &amp;gt; LLM &amp;gt; Output&lt;/code>&lt;/p>
&lt;p>The workflow:&lt;/p>
&lt;ol>
&lt;li>Interprets your question&lt;/li>
&lt;li>Recognises it needs calendar information&lt;/li>
&lt;li>Searches your calendar&lt;/li>
&lt;li>Uses the results to formulate an answer&lt;/li>
&lt;/ol>
&lt;p>We can make this even more powerful. What if you also wanted to know the weather forecast for your meeting with Luke?&lt;/p>
&lt;p>The workflow would expand:&lt;/p>
&lt;p>&lt;code>Input &amp;gt; LLM &amp;gt; Calendar Search &amp;gt; Weather API Query &amp;gt; LLM &amp;gt; Output&lt;/code>&lt;/p>
&lt;p>This demonstrates the power of AI workflows: they can combine multiple data sources and tools to provide comprehensive responses that a standalone LLM couldn&amp;rsquo;t deliver.&lt;/p>
&lt;/br>
&lt;h3 id="retrieval-augmented-generation-rag-a-special-workflow">Retrieval Augmented Generation (RAG): A Special Workflow&lt;/h3>
&lt;hr>
&lt;p>RAG is a specific type of AI workflow that&amp;rsquo;s become particularly important. It allows LLMs to search through and incorporate information from a knowledge base before generating responses.&lt;/p>
&lt;p>The process works like this:&lt;/p>
&lt;ol>
&lt;li>You ask a question&lt;/li>
&lt;li>The system converts your question into a search query&lt;/li>
&lt;li>Relevant documents are retrieved from a database&lt;/li>
&lt;li>The LLM uses both your question and the retrieved information to generate an answer&lt;/li>
&lt;/ol>
&lt;p>This is especially valuable for companies that want AI systems that can reference their internal documentation, knowledge bases, or specialised information.&lt;/p>
&lt;p>For example, a company might implement RAG to allow employees to query their internal policies:&lt;/p>
&lt;p>&lt;code>&amp;quot;What's our policy on remote work during holidays?&amp;quot;&lt;/code>&lt;/p>
&lt;p>The RAG system would:&lt;/p>
&lt;ol>
&lt;li>Search the company&amp;rsquo;s policy documents&lt;/li>
&lt;li>Find relevant sections about remote work and holidays&lt;/li>
&lt;li>Generate a concise answer based on those specific policies&lt;/li>
&lt;/ol>
&lt;/br>
&lt;/br>
&lt;h2 id="ai-agents-adding-autonomy-and-reasoning">AI Agents: Adding Autonomy and Reasoning&lt;/h2>
&lt;hr>
&lt;p>While workflows are powerful, they still follow predetermined paths. AI agents take this a step further by adding autonomous reasoning and decision-making capabilities.&lt;/p>
&lt;p>The term &amp;ldquo;ReAct&amp;rdquo; (Reasoning + Acting) captures this concept well. AI agents can:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Reason&lt;/strong> about what actions to take&lt;/li>
&lt;li>&lt;strong>Act&lt;/strong> based on their reasoning&lt;/li>
&lt;li>Observe the results&lt;/li>
&lt;li>Reason again based on new information&lt;/li>
&lt;/ol>
&lt;p>Let&amp;rsquo;s look at a practical example. Imagine a content creation workflow:&lt;/p>
&lt;ol>
&lt;li>Find relevant URLs on a topic&lt;/li>
&lt;li>Summarise the content from those URLs&lt;/li>
&lt;li>Create a LinkedIn post based on the summaries&lt;/li>
&lt;/ol>
&lt;p>In a traditional workflow, human intervention might be needed to verify the quality of the post before publishing. An AI agent system could eliminate this by adding a quality control step:&lt;/p>
&lt;ol>
&lt;li>Find relevant URLs (using Agent A)&lt;/li>
&lt;li>Summarise those URLs (using Agent A)&lt;/li>
&lt;li>Create a LinkedIn post draft (using Agent A)&lt;/li>
&lt;li>Evaluate if the post meets quality criteria (using Agent B)&lt;/li>
&lt;li>If not, revise the post until it does&lt;/li>
&lt;/ol>
&lt;p>This demonstrates a key aspect of AI agents: they can collaborate and check each other&amp;rsquo;s work, similar to how humans might review each other&amp;rsquo;s contributions.&lt;/p>
&lt;p>The more reviewing agents, the more chance there is that it will be accurate.&lt;/p>
&lt;/br>
&lt;h3 id="specialised-ai-agents">Specialised AI Agents&lt;/h3>
&lt;hr>
&lt;p>The most powerful AI agent systems use specialised agents for specific tasks. Think of this as having a team of experts, each with their own specialty.&lt;/p>
&lt;p>For example, a fraud detection system might include:&lt;/p>
&lt;ul>
&lt;li>A data gathering agent that collects transaction information&lt;/li>
&lt;li>A pattern recognition agent trained specifically on fraudulent transaction patterns&lt;/li>
&lt;li>A risk assessment agent that evaluates potential impact, understanding the policies and risk position of your company&lt;/li>
&lt;li>A notification agent that alerts appropriate personnel&lt;/li>
&lt;/ul>
&lt;p>Each agent is optimised for its specific task, creating a system that&amp;rsquo;s more effective than a single general-purpose agent would be.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="putting-it-all-together">Putting It All Together&lt;/h2>
&lt;hr>
&lt;p>Understanding the progression from LLMs to workflows to agents helps clarify how these systems become increasingly powerful:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>LLMs&lt;/strong> provide the foundational ability to understand and generate human language&lt;/li>
&lt;li>&lt;strong>AI Workflows&lt;/strong> connect LLMs to external data and tools to expand their capabilities&lt;/li>
&lt;li>&lt;strong>AI Agents&lt;/strong> add autonomous reasoning and decision-making to create systems that can adapt and self-improve&lt;/li>
&lt;/ul>
&lt;p>As these technologies continue to evolve, we&amp;rsquo;ll see increasingly sophisticated applications that combine these approaches to solve complex problems.&lt;/p>
&lt;p>The most exciting developments aren&amp;rsquo;t about any single technology, but rather how they can be integrated to create systems that augment human capabilities in ways we&amp;rsquo;re just beginning to explore.&lt;/p></content:encoded><category>AI</category><category>Leadership</category><category>AI Concepts</category><category>ChatGPT</category><category>Claude</category><category>LLM</category><category>AI Workflows</category><category>RAG</category><category>AI Agents</category></item><item><title>The One Simple Secret to Effective AI Prompting</title><link>https://ghostinthedata.info/posts/2025/2025-05-11-ai-prompting-secrets/</link><pubDate>Sun, 11 May 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-05-11-ai-prompting-secrets/</guid><author>Chris Hillman</author><description>Forget complicated prompting techniques - AI is best at roleplaying. Learn how treating AI like an improv actor leads to better results than traditional information extraction approaches.</description><content:encoded>&lt;h2 id="forget-everything-youve-learned-about-ai-prompting">Forget Everything You&amp;rsquo;ve Learned About AI Prompting&lt;/h2>
&lt;hr>
&lt;p>There&amp;rsquo;s a sea of articles out there about &amp;ldquo;how to talk to AI&amp;rdquo; or &amp;ldquo;the perfect prompt structure.&amp;rdquo; Frameworks, formulas, special keywords—it can get overwhelming.&lt;/p>
&lt;p>But what if I told you that you could forget all of it?&lt;/p>
&lt;p>That&amp;rsquo;s right. All those complicated prompting techniques might be unnecessary, because there&amp;rsquo;s one fundamental principle that works better than anything else: &lt;strong>AI excels at roleplaying&lt;/strong>.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="stop-treating-ai-like-a-search-engine">Stop Treating AI Like a Search Engine&lt;/h2>
&lt;hr>
&lt;p>Here&amp;rsquo;s where most people go wrong—they approach AI like it&amp;rsquo;s Google. They try to extract information with perfectly structured queries, hoping to mine the AI for facts and figures.&lt;/p>
&lt;p>But that&amp;rsquo;s not playing to the strengths of today&amp;rsquo;s language models.&lt;/p>
&lt;p>Instead of treating AI like a database you&amp;rsquo;re querying, try something different: &lt;strong>talk to it like it&amp;rsquo;s an improv actor waiting for a character to play&lt;/strong>.&lt;/p>
&lt;p>Why does this work? Because under the hood, AI doesn&amp;rsquo;t store information like traditional databases. These models work with patterns, correlations, and learned relationships between concepts. They don&amp;rsquo;t have a fixed identity or worldview—they can adapt to whatever role you assign them.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-roleplaying-approach-in-action">The Roleplaying Approach in Action&lt;/h2>
&lt;hr>
&lt;p>Think about the difference between these two approaches:&lt;/p>
&lt;p>&lt;strong>Traditional approach:&lt;/strong>
&amp;ldquo;Give me three facts about biology that would interest a teenager.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Roleplaying approach:&lt;/strong>
&amp;ldquo;You&amp;rsquo;re a world-renowned biologist who&amp;rsquo;s known for making science exciting for young people. My 16-year-old nephew thinks biology is boring. You have two minutes to change his mind with fascinating insights. What would you say?&amp;rdquo;&lt;/p>
&lt;p>The second approach gives the AI a character to embody, an audience to connect with, and a goal to achieve. You&amp;rsquo;re not just extracting information—you&amp;rsquo;re setting the stage for a performance.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="why-this-works-better">Why This Works Better&lt;/h2>
&lt;hr>
&lt;p>When you frame your interaction as a roleplaying scenario, several things happen:&lt;/p>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>You get more tailored responses&lt;/strong> - The AI adapts its tone, vocabulary, and content selection to fit the character you&amp;rsquo;ve assigned it.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Your responses feel more natural&lt;/strong> - Instead of listing facts, the AI tries to communicate as a human with the given expertise would.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>It solves the context problem&lt;/strong> - By establishing who&amp;rsquo;s talking to whom and why, you give the AI crucial context for shaping appropriate responses.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>You unlock creativity&lt;/strong> - Roleplaying creates space for the AI to generate novel perspectives and approaches.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;/br>
&lt;/br>
&lt;h2 id="examples-that-make-the-difference">Examples That Make the Difference&lt;/h2>
&lt;hr>
&lt;p>Let&amp;rsquo;s look at a few more examples of how the roleplaying approach transforms interactions:&lt;/p>
&lt;p>&lt;strong>Instead of:&lt;/strong>
&amp;ldquo;Write a summary of the benefits of meditation.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Try:&lt;/strong>
&amp;ldquo;You&amp;rsquo;re a mindfulness coach with 20 years of experience. One of your clients keeps asking why they should bother with meditation when they&amp;rsquo;re so busy. Write a compassionate email explaining the benefits in a way that would resonate with a sceptical, overworked professional.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Instead of:&lt;/strong>
&amp;ldquo;Give me pros and cons of remote work.&amp;rdquo;&lt;/p>
&lt;p>&lt;strong>Try:&lt;/strong>
&amp;ldquo;You&amp;rsquo;re a workplace culture consultant who&amp;rsquo;s helped companies both succeed and fail at remote work transitions. A small business owner has asked for your balanced perspective on implementing remote work. What insights would you share based on your experience?&amp;rdquo;&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-versatility-of-ai-roleplay">The Versatility of AI Roleplay&lt;/h2>
&lt;hr>
&lt;p>The beauty of the roleplay approach is how adaptable it is. You can ask the AI to take on virtually any perspective:&lt;/p>
&lt;ul>
&lt;li>A scientist explaining complex ideas simply&lt;/li>
&lt;li>A medieval bard recounting historical events&lt;/li>
&lt;li>A friendly teacher helping with homework&lt;/li>
&lt;li>A debate coach showing both sides of an argument&lt;/li>
&lt;li>A colleague brainstorming solutions to a work problem&lt;/li>
&lt;/ul>
&lt;p>Each role creates a different lens through which the AI processes and presents information, giving you a rich variety of perspectives and communication styles.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="moving-beyond-information-extraction">Moving Beyond Information Extraction&lt;/h2>
&lt;hr>
&lt;p>This approach shifts your relationship with AI from extraction to collaboration. Rather than just pulling information out of the system, you&amp;rsquo;re creating a dynamic where the AI becomes a virtual partner adapted to your specific needs.&lt;/p>
&lt;p>This is particularly powerful when you need:&lt;/p>
&lt;ul>
&lt;li>Explanations tailored to specific audiences&lt;/li>
&lt;li>Creative solutions to problems&lt;/li>
&lt;li>Different perspectives on a complex issue&lt;/li>
&lt;li>Engaging content with a consistent voice&lt;/li>
&lt;li>Specialised expertise you don&amp;rsquo;t personally have&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;h2 id="try-it-yourself">Try It Yourself&lt;/h2>
&lt;hr>
&lt;p>The next time you interact with ChatGPT, Claude or another AI assistant, experiment with the roleplaying approach:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Define a character&lt;/strong> with expertise relevant to your question&lt;/li>
&lt;li>&lt;strong>Establish an audience&lt;/strong> (who the AI is speaking to)&lt;/li>
&lt;li>&lt;strong>Provide a purpose&lt;/strong> for the communication&lt;/li>
&lt;li>&lt;strong>Set constraints&lt;/strong> if helpful (time, format, tone)&lt;/li>
&lt;/ol>
&lt;p>Then watch how the quality and relevance of your responses transform.&lt;/p>
&lt;/br>
&lt;/br>
&lt;h2 id="the-bottom-line">The Bottom Line&lt;/h2>
&lt;hr>
&lt;p>AI isn&amp;rsquo;t a database—it&amp;rsquo;s a chameleon waiting to take on whatever colour you suggest. When you understand this fundamental nature, you unlock its greatest potential.&lt;/p>
&lt;p>So forget the complicated prompting frameworks. The most powerful technique is also the simplest: treat AI like an improv actor, and you&amp;rsquo;ll get performances tailored exactly to your needs.&lt;/p></content:encoded><category>AI</category><category>Leadership</category><category>AI Prompting</category><category>ChatGPT</category><category>Claude</category><category>LLM</category><category>AI Communication</category><category>Roleplaying</category><category>AI Productivity</category></item><item><title>Leveraging LLMs for Business Impact: Part 2 - Building an AI Data Engineer Agent</title><link>https://ghostinthedata.info/posts/2025/2025-03-08-leveraging-llm-for-business-impact-part-2/</link><pubDate>Sat, 08 Mar 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-03-08-leveraging-llm-for-business-impact-part-2/</guid><author>Chris Hillman</author><description>A step-by-step guide to implementing an AI-powered code reviewer for data engineering teams that enforces best practices and consistency across your codebase.</description><content:encoded>&lt;h2 id="introduction">Introduction&lt;/h2>
&lt;p>In &lt;a href="https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/" target="_blank" rel="noopener">Part 1 of this series&lt;/a>, we explored the theoretical foundations of Large Language Models (LLMs), Retrieval Augmented Generation (RAG), and vector databases. Now, it&amp;rsquo;s time to put theory into practice. This is going to be a long read, so grab some coffee, and one (couple) of your favourite biscuits.&lt;/p>
&lt;p>One use case for leveraging LLM&amp;rsquo;s, is creating of a Agent - a Senior Data Engineer AI that automatically reviews Pull Requests in your data engineering projects.
This agent will be that nit picky Data Engineer that enforces SQL formatting standards, ensure naming and data type consistency, validate data quality checks, and suggest improvements based on best practices. By integrating this into your GitHub workflow, you can maintain higher code quality, accelerate onboarding for new team members, and reduce the burden of manual code reviews.&lt;/p>
&lt;p>This tutorial walks through the complete implementation process from setting up the necessary API keys to crafting the AI agent and configuring GitHub Actions to run it automatically on every Pull Request.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="project-overview">Project Overview&lt;/h2>
&lt;/br>
&lt;p>Our AI-powered code review system will:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Monitor GitHub for new Pull Requests&lt;/strong> in your data repository&lt;/li>
&lt;li>&lt;strong>Extract changed files&lt;/strong> for analysis&lt;/li>
&lt;li>&lt;strong>Retrieve relevant context&lt;/strong> about your code standards and existing codebase&lt;/li>
&lt;li>&lt;strong>Analyse changes&lt;/strong> using a custom-prompted LLM&lt;/li>
&lt;li>&lt;strong>Generate detailed feedback&lt;/strong> on code quality, standards compliance, and potential improvements&lt;/li>
&lt;li>&lt;strong>Post review comments&lt;/strong> directly on the Pull Request&lt;/li>
&lt;/ol>
&lt;p>This solution combines GitHub APIs, LLM capabilities, and custom prompt engineering to create an automated reviewer that thinks like a senior data engineer on your team.&lt;/p>
&lt;figure>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-08-leveraging-llm-for-business-impact-part-2/images/ai-code-reviewer-workflow.png" width="400x">
&lt;/figure>

&lt;p>Although this does offer a solution, there is plenty of things that could be done to make this more robust, and far more superior. So please use this more for educational purposes, and a demonstration.&lt;/p>
&lt;/br>
&lt;h3 id="use-cases-and-benefits">Use Cases and Benefits&lt;/h3>
&lt;p>This implementation addresses several key challenges in data engineering teams:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Consistency Enforcement&lt;/strong>: Ensures all database objects follow the same naming conventions and data types&lt;/li>
&lt;li>&lt;strong>Knowledge Transfer&lt;/strong>: Helps junior engineers learn best practices through contextual feedback&lt;/li>
&lt;li>&lt;strong>Review Acceleration&lt;/strong>: Handles routine checks automatically, letting human reviewers focus on architecture and business logic&lt;/li>
&lt;li>&lt;strong>Standards Documentation&lt;/strong>: Indirectly builds a living set of standards through consistent application&lt;/li>
&lt;li>&lt;strong>Technical Debt Prevention&lt;/strong>: Catches potential issues before they enter the codebase&lt;/li>
&lt;/ul>
&lt;p>If setup correctly you can leverage a vector database to store context and improve for your business process over time.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-1-setting-up-the-required-api-keys">Step 1: Setting Up the Required API Keys&lt;/h2>
&lt;/br>
&lt;p>To build our system, we&amp;rsquo;ll need authentication tokens for GitHub and an LLM provider. Here&amp;rsquo;s how to set these up securely:&lt;/p>
&lt;/br>
&lt;h3 id="github-personal-access-token">GitHub Personal Access Token&lt;/h3>
&lt;ol>
&lt;li>Log in to your GitHub account and navigate to &lt;strong>Settings&lt;/strong> → &lt;strong>Developer settings&lt;/strong> → &lt;strong>Personal access tokens&lt;/strong> → &lt;strong>Tokens (classic)&lt;/strong>&lt;/li>
&lt;li>Click &lt;strong>Generate new token&lt;/strong> and select &lt;strong>Generate new token (classic)&lt;/strong>&lt;/li>
&lt;li>Name your token something descriptive like &amp;ldquo;AI Code Reviewer&amp;rdquo;&lt;/li>
&lt;li>Set the expiration according to your needs&lt;/li>
&lt;li>Select the following permissions:
&lt;ul>
&lt;li>&lt;code>repo&lt;/code> (Full control of private repositories)&lt;/li>
&lt;li>&lt;code>workflow&lt;/code> (Update GitHub Action workflows)&lt;/li>
&lt;li>&lt;code>pull_requests&lt;/code> (Access to pull requests)&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>Click &lt;strong>Generate token&lt;/strong> and copy the token immediately (you won&amp;rsquo;t be able to see it again)&lt;/li>
&lt;/ol>
&lt;/br>
&lt;h3 id="llm-api-key">LLM API Key&lt;/h3>
&lt;p>You&amp;rsquo;ll need access to a capable LLM. We&amp;rsquo;ll provide instructions for both Anthropic&amp;rsquo;s Claude and OpenAI&amp;rsquo;s GPT-4:&lt;/p>
&lt;h4 id="anthropic-claude-api-key">Anthropic Claude API Key:&lt;/h4>
&lt;ol>
&lt;li>Visit &lt;a href="https://console.anthropic.com/" target="_blank" rel="noopener">Anthropic&amp;rsquo;s Console&lt;/a>&lt;/li>
&lt;li>Create or log in to your account&lt;/li>
&lt;li>Navigate to &lt;strong>API Keys&lt;/strong> in the left sidebar&lt;/li>
&lt;li>Click &lt;strong>Create API Key&lt;/strong>&lt;/li>
&lt;li>Name your key (e.g., &amp;ldquo;GitHub Code Reviewer&amp;rdquo;)&lt;/li>
&lt;li>Copy and securely store your API key&lt;/li>
&lt;/ol>
&lt;h4 id="openai-api-key-alternative">OpenAI API Key (Alternative):&lt;/h4>
&lt;ol>
&lt;li>Visit &lt;a href="https://platform.openai.com/" target="_blank" rel="noopener">OpenAI&amp;rsquo;s Platform&lt;/a>&lt;/li>
&lt;li>Sign in to your account&lt;/li>
&lt;li>Navigate to &lt;strong>API Keys&lt;/strong>&lt;/li>
&lt;li>Click &lt;strong>Create new secret key&lt;/strong>&lt;/li>
&lt;li>Name your key and click &lt;strong>Create&lt;/strong>&lt;/li>
&lt;li>Copy and securely store the API key&lt;/li>
&lt;/ol>
&lt;/br>
&lt;h3 id="storing-secrets-in-github-repository">Storing Secrets in GitHub Repository&lt;/h3>
&lt;p>Now we&amp;rsquo;ll add these secrets to your GitHub repository:&lt;/p>
&lt;ol>
&lt;li>Navigate to your repository on GitHub&lt;/li>
&lt;li>Go to &lt;strong>Settings&lt;/strong> → &lt;strong>Secrets and variables&lt;/strong> → &lt;strong>Actions&lt;/strong>&lt;/li>
&lt;li>Click &lt;strong>New repository secret&lt;/strong>&lt;/li>
&lt;li>Add the following secrets:
&lt;ul>
&lt;li>&lt;code>GH_TOKEN&lt;/code>: Your GitHub Personal Access Token&lt;/li>
&lt;li>&lt;code>ANTHROPIC_API_KEY&lt;/code> or &lt;code>OPENAI_API_KEY&lt;/code>: Your LLM API key&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ol>
&lt;p>These secrets will be accessible in our GitHub Actions workflow without being exposed in logs or to unauthorised users.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-2-creating-the-ai-code-reviewer-script">Step 2: Creating the AI Code Reviewer Script&lt;/h2>
&lt;/br>
&lt;p>We&amp;rsquo;ll create a Python script that pulls code from a PR, analyses it with an LLM, and posts comments back to GitHub. Here&amp;rsquo;s the implementation:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> requests
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> anthropic &lt;span style="color:#75715e"># or openai&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> base64
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> github &lt;span style="color:#f92672">import&lt;/span> Github
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> pathlib &lt;span style="color:#f92672">import&lt;/span> Path
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize clients&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>github_token &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;GH_TOKEN&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>g &lt;span style="color:#f92672">=&lt;/span> Github(github_token)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Choose your LLM provider&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>USE_CLAUDE &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#66d9ef">True&lt;/span> &lt;span style="color:#75715e"># Set to False to use OpenAI&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> USE_CLAUDE:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> anthropic_key &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;ANTHROPIC_API_KEY&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> client &lt;span style="color:#f92672">=&lt;/span> anthropic&lt;span style="color:#f92672">.&lt;/span>Anthropic(api_key&lt;span style="color:#f92672">=&lt;/span>anthropic_key)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">import&lt;/span> openai
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> openai&lt;span style="color:#f92672">.&lt;/span>api_key &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;OPENAI_API_KEY&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># GitHub repository information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>repo_owner &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;GITHUB_REPOSITORY_OWNER&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>repo_name &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;GITHUB_REPOSITORY&amp;#34;&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#34;/&amp;#34;&lt;/span>)[&lt;span style="color:#ae81ff">1&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>pr_number &lt;span style="color:#f92672">=&lt;/span> int(os&lt;span style="color:#f92672">.&lt;/span>environ[&lt;span style="color:#e6db74">&amp;#34;PR_NUMBER&amp;#34;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>repository &lt;span style="color:#f92672">=&lt;/span> g&lt;span style="color:#f92672">.&lt;/span>get_repo(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>repo_owner&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">/&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>repo_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>pull_request &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_pull(pr_number)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">get_sql_standards&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Retrieve SQL standards from repository documentation&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards_file_content &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;docs/sql_standards.md&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(standards_file_content&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># If no standards file exists, use default standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # SQL Standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Naming Conventions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Tables: snake_case, plural (e.g., customer_orders)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Columns: snake_case, singular (e.g., customer_id)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Primary keys: id (e.g., customer_id)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Foreign keys: Same name as referenced PK (e.g., customer_id)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Boolean columns: is_* or has_* prefix (e.g., is_active)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Date columns: _date suffix (e.g., created_date or created_date_id)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Formatting
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Keywords in UPPERCASE
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Indentation of 4 spaces
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Commas at the end of lines
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - One statement per line
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Data Types
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - IDs: INTEGER or BIGINT
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Names: VARCHAR
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Flags: BOOLEAN
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Dates: DATE
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Timestamps: TIMESTAMP(6)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## DBT Practices
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Models must be idempotent
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Include appropriate tests for primary keys or unique keys
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Add descriptions to all models and columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> standards
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">get_naming_dictionary&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Build a dictionary of standard column names and their data types from the codebase&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict &lt;span style="color:#f92672">=&lt;/span> {}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check SQL and DBT files in the repository&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_extensions &lt;span style="color:#f92672">=&lt;/span> [&lt;span style="color:#e6db74">&amp;#34;.sql&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;.yml&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> contents &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Process files recursively&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">while&lt;/span> contents:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> contents&lt;span style="color:#f92672">.&lt;/span>pop(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>type &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;dir&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> contents&lt;span style="color:#f92672">.&lt;/span>extend(repository&lt;span style="color:#f92672">.&lt;/span>get_contents(file_content&lt;span style="color:#f92672">.&lt;/span>path))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> any(file_content&lt;span style="color:#f92672">.&lt;/span>name&lt;span style="color:#f92672">.&lt;/span>endswith(ext) &lt;span style="color:#66d9ef">for&lt;/span> ext &lt;span style="color:#f92672">in&lt;/span> sql_extensions):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract DDL statements and column names&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_text &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(file_content&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># This is a simplified approach - a real implementation would use SQL parsing&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> line &lt;span style="color:#f92672">in&lt;/span> file_text&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;CREATE TABLE&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>upper() &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#e6db74">&amp;#34;column:&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column definitions - this is simplified&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;INTEGER&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>split()[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip(&lt;span style="color:#e6db74">&amp;#34;,&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>lower()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict[col_name] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;INTEGER&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>split()[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip(&lt;span style="color:#e6db74">&amp;#34;,&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>lower()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict[col_name] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> &lt;span style="color:#e6db74">&amp;#34;DATE&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> line&lt;span style="color:#f92672">.&lt;/span>split()[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip(&lt;span style="color:#e6db74">&amp;#34;,&amp;#34;&lt;/span>)&lt;span style="color:#f92672">.&lt;/span>lower()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict[col_name] &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;DATE&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add more data types as needed&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error building naming dictionary: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add some default mappings if the dictionary is empty&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> naming_dict:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;account_number&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;customer_id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;email_address&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;created_date&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;DATE&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;is_active&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;BOOLEAN&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;transaction_amount&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;DECIMAL&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;product_name&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;VARCHAR&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;order_id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;INTEGER&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> naming_dict
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">analyze_file&lt;/span>(file_path, file_content, sql_standards, naming_dict):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Submit file to LLM for analysis&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Construct the prompt for the LLM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> prompt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You are an expert Senior Data Engineer responsible for reviewing SQL and data transformation code. 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You&amp;#39;re reviewing a pull request and need to provide detailed, actionable feedback.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # SQL STANDARDS AND BEST PRACTICES
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_standards&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # KNOWN COLUMN NAMING PATTERNS AND DATA TYPES
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>json&lt;span style="color:#f92672">.&lt;/span>dumps(naming_dict, indent&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">4&lt;/span>)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # FILE TO REVIEW
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Path: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Please review this code thoroughly and provide feedback on:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 1. SQL formatting and adherence to our standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 2. Naming consistency (compared to our naming dictionary)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 3. Data type consistency
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 4. Idempotency issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 5. Missing data quality checks or tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 6. Proper use of dbt_valid_from and dbt_valid_to for time-based models
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 7. Performance concerns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 8. Any other issues or suggestions for improvement
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Format your response as:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Summary
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> A brief overview of the code quality (1-2 sentences)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Describe each issue you find with line numbers and specific recommendations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Suggestions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Provide constructive suggestions for improvement
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Proposed Tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If relevant, suggest tests that would improve data quality
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Focus on being helpful and educational. If something is done well, mention that too.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Call the appropriate LLM API&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> USE_CLAUDE:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> client&lt;span style="color:#f92672">.&lt;/span>messages&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;claude-3-opus-20240229&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_tokens&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">4000&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0.2&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;You are a Senior Data Engineer who specializes in code review. You provide constructive, specific feedback focused on data engineering best practices.&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>content[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> openai&lt;span style="color:#f92672">.&lt;/span>ChatCompletion&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;gpt-4-turbo&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0.2&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;system&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;You are a Senior Data Engineer who specializes in code review. You provide constructive, specific feedback focused on data engineering best practices.&amp;#34;&lt;/span>},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>choices[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>message&lt;span style="color:#f92672">.&lt;/span>content
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">post_review_comment&lt;/span>(pull_request, body):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Post a review comment to the pull request&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pull_request&lt;span style="color:#f92672">.&lt;/span>create_issue_comment(body)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Posted review comment to PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr_number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error posting comment: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">main&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get SQL standards and naming dictionary&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards &lt;span style="color:#f92672">=&lt;/span> get_sql_standards()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_dict &lt;span style="color:#f92672">=&lt;/span> get_naming_dictionary()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get the files changed in the PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> files &lt;span style="color:#f92672">=&lt;/span> pull_request&lt;span style="color:#f92672">.&lt;/span>get_files()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Track all feedback to post in one comment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;# AI Data Engineer Review for PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr_number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze each file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file &lt;span style="color:#f92672">in&lt;/span> files:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Only process SQL and DBT files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#f92672">.&lt;/span>endswith((&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.dbt&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Analyzing &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> requests&lt;span style="color:#f92672">.&lt;/span>get(file&lt;span style="color:#f92672">.&lt;/span>raw_url)&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze the file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> feedback &lt;span style="color:#f92672">=&lt;/span> analyze_file(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content, sql_standards, naming_dict)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;## File: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>feedback&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">---&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add a conclusion&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> **Note**: This review was generated by an AI assistant. While it strives for accuracy, always use your judgment when implementing suggestions.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If you have feedback on how to improve these automated reviews, please let us know!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Post the combined review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> post_review_comment(pull_request, all_feedback)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> __name__ &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;__main__&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> main()
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Let&amp;rsquo;s understand what this script does:&lt;/p>
&lt;ol>
&lt;li>It authenticates with GitHub and the LLM API of your choice (Claude or GPT-4)&lt;/li>
&lt;li>Extracts the SQL standards from your repository (or uses defaults)&lt;/li>
&lt;li>Builds a dictionary of common column names and data types from your codebase&lt;/li>
&lt;li>For each changed SQL or DBT file in the PR:
&lt;ul>
&lt;li>Retrieves the file content&lt;/li>
&lt;li>Sends it to the LLM for analysis with a detailed prompt&lt;/li>
&lt;li>Collects the feedback&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>Posts a comprehensive review comment to the PR&lt;/li>
&lt;/ol>
&lt;p>This script combines several powerful techniques:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Contextual prompting&lt;/strong>: It gives the LLM your specific SQL standards and naming conventions&lt;/li>
&lt;li>&lt;strong>Domain-specific guidance&lt;/strong>: The prompt focuses on data engineering concerns like idempotency and SCD tracking&lt;/li>
&lt;li>&lt;strong>Actionable feedback&lt;/strong>: It asks for specific line numbers and recommendations&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-3-creating-the-github-action-workflow">Step 3: Creating the GitHub Action Workflow&lt;/h2>
&lt;/br>
&lt;p>Now let&amp;rsquo;s set up the GitHub Action workflow to run this script automatically on every PR. Create a file at &lt;code>.github/workflows/ai-code-review.yml&lt;/code> in your repository:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">AI Data Engineer Code Review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">on&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pull_request&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">types&lt;/span>: [&lt;span style="color:#ae81ff">opened, synchronize]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">paths&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**.sql&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**/models/**.yml&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**/dbt_project.yml&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">jobs&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">review&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">runs-on&lt;/span>: &lt;span style="color:#ae81ff">ubuntu-latest&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">permissions&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">contents&lt;/span>: &lt;span style="color:#ae81ff">read&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pull-requests&lt;/span>: &lt;span style="color:#ae81ff">write&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">steps&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Checkout repository&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">uses&lt;/span>: &lt;span style="color:#ae81ff">actions/checkout@v3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">with&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">fetch-depth&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Set up Python&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">uses&lt;/span>: &lt;span style="color:#ae81ff">actions/setup-python@v4&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">with&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">python-version&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;3.10&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Install dependencies&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> python -m pip install --upgrade pip
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> pip install PyGithub requests anthropic openai&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Run AI code review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">env&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">GH_TOKEN&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.GH_TOKEN }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">ANTHROPIC_API_KEY&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.ANTHROPIC_API_KEY }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">OPENAI_API_KEY&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.OPENAI_API_KEY }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">PR_NUMBER&lt;/span>: &lt;span style="color:#ae81ff">${{ github.event.pull_request.number }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">GITHUB_REPOSITORY&lt;/span>: &lt;span style="color:#ae81ff">${{ github.repository }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">GITHUB_REPOSITORY_OWNER&lt;/span>: &lt;span style="color:#ae81ff">${{ github.repository_owner }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # Create the code review script
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> cat &amp;gt; ai_code_review.py &amp;lt;&amp;lt; &amp;#39;EOF&amp;#39;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # Paste the entire Python script from Step 2 here
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> EOF
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # Run the script
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> python ai_code_review.py&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This workflow:&lt;/p>
&lt;ol>
&lt;li>Triggers whenever a PR is opened or updated&lt;/li>
&lt;li>Only runs when SQL, or YAML files are changed&lt;/li>
&lt;li>Sets up Python and installs the required dependencies&lt;/li>
&lt;li>Creates the review script (by embedding it in the workflow)&lt;/li>
&lt;li>Runs the script with the necessary environment variables&lt;/li>
&lt;/ol>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-4-implementing-rag-for-contextual-code-review">Step 4: Implementing RAG for Contextual Code Review&lt;/h2>
&lt;/br>
&lt;p>The real power of our AI code reviewer comes from using Retrieval Augmented Generation (RAG) to give the AI contextual understanding of your codebase, conventions, and standards. Rather than relying on hardcoded rules, we&amp;rsquo;ll build a vector database that understands your organisation&amp;rsquo;s patterns and practices.&lt;/p>
&lt;p>Here&amp;rsquo;s how we&amp;rsquo;ll approach this:&lt;/p>
&lt;h3 id="creating-and-maintaining-a-vector-database-of-codebase-knowledge">Creating and Maintaining a Vector Database of Codebase Knowledge&lt;/h3>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> numpy &lt;span style="color:#66d9ef">as&lt;/span> np
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> json
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> base64
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> datetime &lt;span style="color:#f92672">import&lt;/span> datetime
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pinecone
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> sentence_transformers &lt;span style="color:#f92672">import&lt;/span> SentenceTransformer
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize the embedding model&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>model &lt;span style="color:#f92672">=&lt;/span> SentenceTransformer(&lt;span style="color:#e6db74">&amp;#39;all-MiniLM-L6-v2&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>EMBEDDING_DIMENSION &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">384&lt;/span> &lt;span style="color:#75715e"># Dimension for all-MiniLM-L6-v2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize Pinecone&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PINECONE_API_KEY &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#34;PINECONE_API_KEY&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PINECONE_ENVIRONMENT &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#34;PINECONE_ENVIRONMENT&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;gcp-starter&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>INDEX_NAME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;data-engineer-reviewer&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>pinecone&lt;span style="color:#f92672">.&lt;/span>init(api_key&lt;span style="color:#f92672">=&lt;/span>PINECONE_API_KEY, environment&lt;span style="color:#f92672">=&lt;/span>PINECONE_ENVIRONMENT)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create index if it doesn&amp;#39;t exist&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> INDEX_NAME &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> pinecone&lt;span style="color:#f92672">.&lt;/span>list_indexes():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pinecone&lt;span style="color:#f92672">.&lt;/span>create_index(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> name&lt;span style="color:#f92672">=&lt;/span>INDEX_NAME,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dimension&lt;span style="color:#f92672">=&lt;/span>EMBEDDING_DIMENSION,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metric&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;cosine&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> shards&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Connect to the index&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>index &lt;span style="color:#f92672">=&lt;/span> pinecone&lt;span style="color:#f92672">.&lt;/span>Index(INDEX_NAME)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">extract_code_knowledge&lt;/span>(repository):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Extract and process knowledge from repository files&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">&amp;#34;Beginning knowledge extraction from repository...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Lists to store our embeddings and metadata&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_files_processed &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get repository contents recursively&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> contents &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_extensions &lt;span style="color:#f92672">=&lt;/span> [&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.dbt&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">while&lt;/span> contents:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> contents&lt;span style="color:#f92672">.&lt;/span>pop(&lt;span style="color:#ae81ff">0&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>type &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;dir&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dir_contents &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_contents(file_content&lt;span style="color:#f92672">.&lt;/span>path)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> contents&lt;span style="color:#f92672">.&lt;/span>extend(dir_contents)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error accessing directory &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">continue&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> any(file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>endswith(ext) &lt;span style="color:#66d9ef">for&lt;/span> ext &lt;span style="color:#f92672">in&lt;/span> sql_extensions):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Process SQL and DBT files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_text &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(file_content&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_files_processed &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column definitions and their data types&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract table creation patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> create_table_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;CREATE\s+TABLE\s+(\w+)\s*\((.*?)\)&amp;#34;&lt;/span>, file_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> table_name, columns_text &lt;span style="color:#f92672">in&lt;/span> create_table_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> columns &lt;span style="color:#f92672">=&lt;/span> columns_text&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> col &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>startswith(&lt;span style="color:#e6db74">&amp;#39;--&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_parts &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(col_parts) &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> col_parts[&lt;span style="color:#ae81ff">0&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_type &lt;span style="color:#f92672">=&lt;/span> col_parts[&lt;span style="color:#ae81ff">1&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; is defined with data type &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_type&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; in table &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>table_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dt_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_files_processed&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;column_name&amp;#34;&lt;/span>: col_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;data_type&amp;#34;&lt;/span>: col_type,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;table_definition&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column references in SELECT statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> select_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SELECT\s+(.*?)\s+FROM&amp;#34;&lt;/span>, file_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> select_clause &lt;span style="color:#f92672">in&lt;/span> select_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;*&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> select_clause: &lt;span style="color:#75715e"># Skip SELECT *&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> columns &lt;span style="color:#f92672">=&lt;/span> select_clause&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> col &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>startswith(&lt;span style="color:#e6db74">&amp;#39;--&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Handle aliasing with &amp;#39;AS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_parts &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> col_parts[&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;.&amp;#39;&lt;/span>)[&lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column name &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; is used in SELECT statement in file &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;np_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_files_processed&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;column_name&amp;#34;&lt;/span>: col_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;select_statement&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract DBT-specific patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>path &lt;span style="color:#f92672">or&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get a snippet of the file for context&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> file_text[:&lt;span style="color:#ae81ff">1000&lt;/span>] &lt;span style="color:#75715e"># First 1000 chars for context&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_file_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_files_processed&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;dbt_file&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for specific DBT patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;{{ config(&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_text:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config_pattern &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;{{[\s]*config\((.*?)\)[\s]*}}&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(config_pattern, file_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> config_content &lt;span style="color:#f92672">in&lt;/span> config_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;DBT config in &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>config_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_config_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_files_processed&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;dbt_config&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error processing file &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract SQL standards if they exist&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards_file &lt;span style="color:#f92672">=&lt;/span> repository&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;docs/sql_standards.md&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards_text &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(standards_file&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Split into sections for better retrieval granularity&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sections &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;#{2,3}\s+&amp;#39;&lt;/span>, standards_text)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, section &lt;span style="color:#f92672">in&lt;/span> enumerate(sections):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> section&lt;span style="color:#f92672">.&lt;/span>strip():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> section&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;std_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;section_index&amp;#34;&lt;/span>: i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># If no standards file, use some default standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> default_standards &lt;span style="color:#f92672">=&lt;/span> [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use snake_case for all database object names&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Primary keys should be named &amp;#39;id&amp;#39; or &amp;#39;table_name_id&amp;#39;&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Foreign keys should match the referenced column name&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use UPPERCASE for SQL keywords&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Indent with 2 spaces&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Place each column on its own line in SELECT statements&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Include a WHERE clause to avoid full table scans&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use consistent data types for similar fields across tables&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, standard &lt;span style="color:#f92672">in&lt;/span> enumerate(default_standards):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(standard)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;default_std_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: standard,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;default_standards&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;section_index&amp;#34;&lt;/span>: i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Update the Pinecone index with the extracted knowledge&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> update_pinecone_index(sql_standards_data, naming_patterns_data, data_type_patterns_data, dbt_patterns_data)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Knowledge extraction complete. Processed &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>sql_files_processed&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> SQL files.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Extracted: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(sql_standards_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> standard sections, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(naming_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> naming patterns, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(data_type_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> data type patterns, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(dbt_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> DBT patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">update_pinecone_index&lt;/span>(sql_standards, naming_patterns, data_type_patterns, dbt_patterns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Update the Pinecone index with extracted knowledge&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Pinecone allows bulk upserts for better performance&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># We&amp;#39;ll tag vectors by their type for filtering later&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First, delete existing vectors by namespace&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data_types&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert SQL standards data&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sql_standards:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>sql_standards, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert naming patterns data&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> naming_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Pinecone has limits on batch size, so chunk if needed&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk_size &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(naming_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> naming_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert data type patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> data_type_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(data_type_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> data_type_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data_types&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert DBT patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dbt_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(dbt_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> dbt_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">query_repository_knowledge&lt;/span>(query, namespace, top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">5&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Query the Pinecone index for relevant knowledge&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> query_embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(query)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> index&lt;span style="color:#f92672">.&lt;/span>query(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> vector&lt;span style="color:#f92672">=&lt;/span>query_embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>top_k,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> include_metadata&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> namespace&lt;span style="color:#f92672">=&lt;/span>namespace
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract the text from the metadata&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results&lt;span style="color:#f92672">.&lt;/span>matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> retrieved_texts &lt;span style="color:#f92672">=&lt;/span> [&lt;span style="color:#66d9ef">match&lt;/span>&lt;span style="color:#f92672">.&lt;/span>metadata[&lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>] &lt;span style="color:#66d9ef">for&lt;/span> &lt;span style="color:#66d9ef">match&lt;/span> &lt;span style="color:#f92672">in&lt;/span> results&lt;span style="color:#f92672">.&lt;/span>matches]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> retrieved_texts
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> []
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then update the &lt;code>main()&lt;/code> function to use our RAG-based approach:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">main&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First, extract knowledge from the repository to build/update the vector database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># This could be done less frequently, such as on a schedule rather than for every PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> extract_code_knowledge(repository)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get the files changed in the PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> files &lt;span style="color:#f92672">=&lt;/span> pull_request&lt;span style="color:#f92672">.&lt;/span>get_files()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Track all feedback to post in one comment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;# AI Data Engineer Review for PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr_number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Initialize metrics summary&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_consistency_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_best_practice_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze each file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file &lt;span style="color:#f92672">in&lt;/span> files:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Only process SQL and DBT files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#f92672">.&lt;/span>endswith((&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.dbt&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Analyzing &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> requests&lt;span style="color:#f92672">.&lt;/span>get(file&lt;span style="color:#f92672">.&lt;/span>raw_url)&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get primary feedback using RAG-powered analysis&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> primary_feedback &lt;span style="color:#f92672">=&lt;/span> analyze_file(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Verify the feedback with a second agent&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> verification &lt;span style="color:#f92672">=&lt;/span> verify_feedback(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content, primary_feedback)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze the review to identify specific categories of issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> issues_analysis &lt;span style="color:#f92672">=&lt;/span> analyze_review_for_metrics(primary_feedback)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Update metrics&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> issues_analysis[&lt;span style="color:#e6db74">&amp;#34;total_issues&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;naming_consistency_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> issues_analysis[&lt;span style="color:#e6db74">&amp;#34;naming_issues&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;dbt_best_practice_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> issues_analysis[&lt;span style="color:#e6db74">&amp;#34;dbt_issues&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine feedback with verification&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_feedback &lt;span style="color:#f92672">=&lt;/span> primary_feedback
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> verification:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>verification&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;## File: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_feedback&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">---&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add a conclusion&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> **Note**: This review was generated by an AI assistant. While it strives for accuracy, always use your judgment when implementing suggestions.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If you have feedback on how to improve these automated reviews, please let us know!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Post the combined review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> post_review_comment(pull_request, all_feedback)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Also modify the &lt;code>analyze_file&lt;/code> function to leverage this contextual knowledge:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">analyze_file&lt;/span>(file_path, file_content):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Submit file to LLM for analysis using RAG context&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build RAG context from the vector database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> rag_context &lt;span style="color:#f92672">=&lt;/span> build_rag_context_for_review(file_path, file_content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Construct the prompt for the LLM&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> prompt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You are an expert Senior Data Engineer responsible for reviewing SQL and data transformation code. 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You&amp;#39;re reviewing a pull request and need to provide detailed, actionable feedback.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # CONTEXT FROM CODEBASE ANALYSIS
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>rag_context&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # FILE TO REVIEW
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Path: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Please review this code thoroughly based on the provided context. Focus your feedback on:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 1. SQL formatting and adherence to our standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 2. Naming consistency with existing patterns in our codebase
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 3. Data type consistency with how we typically define these columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 4. Idempotency issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 5. Missing data quality checks or tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 6. If DBT code, proper use of dbt_valid_from and dbt_valid_to for time-based models
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 7. Performance concerns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 8. Any other issues or suggestions for improvement
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Format your response as:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Summary
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> A brief overview of the code quality (1-2 sentences)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Describe each issue you find with line numbers and specific recommendations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Suggestions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Provide constructive suggestions for improvement
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Proposed Tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If relevant, suggest tests that would improve data quality
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Focus on being helpful and educational. If something is done well, mention that too.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Call the appropriate LLM API&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> USE_CLAUDE:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> client&lt;span style="color:#f92672">.&lt;/span>messages&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;claude-3-opus-20240229&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_tokens&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">4000&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;You are a Senior Data Engineer who specializes in code review. You provide constructive, specific feedback focused on data engineering best practices.&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>content[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> openai&lt;span style="color:#f92672">.&lt;/span>ChatCompletion&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;gpt-4&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;system&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;You are a Senior Data Engineer who specializes in code review. You provide constructive, specific feedback focused on data engineering best practices.&amp;#34;&lt;/span>},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>choices[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>message&lt;span style="color:#f92672">.&lt;/span>content
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>The function to build the RAG context for each review:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">build_rag_context_for_review&lt;/span>(file_path, file_content):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Build a comprehensive RAG context for the file being reviewed&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Determine the type of file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_sql &lt;span style="color:#f92672">=&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_dbt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_path &lt;span style="color:#f92672">or&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Always include SQL standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;SQL coding standards and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">5&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sql_standards_results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# SQL STANDARDS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> sql_standards_results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column names from the file to query for naming patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_sql:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Simple regex to extract column names from SELECT statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> select_clauses &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SELECT\s+(.*?)\s+FROM&amp;#34;&lt;/span>, file_content, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> clause &lt;span style="color:#f92672">in&lt;/span> select_clauses:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;*&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> clause:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> cols &lt;span style="color:#f92672">=&lt;/span> clause&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> cols:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span>)[&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;.&amp;#39;&lt;/span>)[&lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names&lt;span style="color:#f92672">.&lt;/span>append(col)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for naming patterns information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_name &lt;span style="color:#f92672">in&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column named &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> naming_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# COLUMN NAMING PATTERNS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(naming_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for DBT patterns if applicable&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_dbt:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for specific DBT features in the file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ config(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT config block&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ ref(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT ref function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ source(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT source function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ dbt_utils&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT utils package&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;incremental&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;incremental model&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> feature &lt;span style="color:#f92672">in&lt;/span> dbt_features:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;DBT &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>feature&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> patterns and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dbt_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# DBT PATTERNS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(dbt_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine all context sections&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#f92672">.&lt;/span>join(context_sections)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This Pinecone implementation provides several advantages:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Scalability&lt;/strong>: Pinecone can handle millions of vectors and high query throughput&lt;/li>
&lt;li>&lt;strong>Managed Service&lt;/strong>: No need to worry about infrastructure management&lt;/li>
&lt;li>&lt;strong>Low Latency&lt;/strong>: Optimised for fast query responses&lt;/li>
&lt;li>&lt;strong>Namespaces&lt;/strong>: Built-in support for organising vectors by category (SQL standards, naming patterns, etc.)&lt;/li>
&lt;/ol>
&lt;p>Using this RAG approach with Pinecone gives the AI reviewer true contextual awareness of your specific codebase patterns and standards without the need for hardcoded rules.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-5-adding-multi-agent-verification-for-improved-reliability">Step 5: Adding Multi-Agent Verification for Improved Reliability&lt;/h2>
&lt;/br>
&lt;p>To reduce hallucinations and improve the reliability of our reviews, we can implement a multi-agent verification system that uses a second &amp;ldquo;Critic&amp;rdquo; agent to verify the primary agent&amp;rsquo;s findings:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">verify_feedback&lt;/span>(file_path, file_content, primary_feedback):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Use a second LLM as a &amp;#34;Critic&amp;#34; to verify the primary agent&amp;#39;s feedback
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> and reduce hallucinations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build a simpler RAG context focused on verifying factual claims&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_type &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;SQL&amp;#34;&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">else&lt;/span> &lt;span style="color:#e6db74">&amp;#34;DBT&amp;#34;&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_path &lt;span style="color:#66d9ef">else&lt;/span> &lt;span style="color:#e6db74">&amp;#34;YAML&amp;#34;&lt;/span> &lt;span style="color:#66d9ef">if&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">else&lt;/span> &lt;span style="color:#e6db74">&amp;#34;Other&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> verification_prompt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You are a Critical Reviewer whose job is to verify another AI&amp;#39;s code review for accuracy.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You should check if the review is factually correct and relevant to the code.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # FILE REVIEWED (&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_type&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Path: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # ORIGINAL REVIEW
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>primary_feedback&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Please verify this review and note:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 1. Any hallucinations or incorrect statements about the code
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 2. Any misinterpretations of SQL syntax or data engineering concepts
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 3. Any feedback that seems contradictory or inconsistent
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Only mention issues that are clearly wrong - do not repeat correct feedback.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If the review seems accurate, just say &amp;#34;The review appears accurate.&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">analyze_review_for_metrics&lt;/span>(review_text):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Analyze the review content to extract metrics about issues found&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;total_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;data_type_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;performance_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;test_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;formatting_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Count total issues by looking for &amp;#34;##&amp;#34; sections&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> issues_section &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;## Issues\s+(.*?)(?=##|$)&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> issues_section:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> issues_text &lt;span style="color:#f92672">=&lt;/span> issues_section&lt;span style="color:#f92672">.&lt;/span>group(&lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Count bullet points or numbered items&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> items &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;^(?:\d+\.|-|\*)\s+&amp;#34;&lt;/span>, issues_text, re&lt;span style="color:#f92672">.&lt;/span>MULTILINE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;total_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(items)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for naming-related issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming|column name|inconsistent&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming|column name|inconsistent&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;naming_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(naming_matches)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for data type issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data type|varchar|integer|decimal|timestamp&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data type|varchar|integer|decimal|timestamp&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;data_type_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(data_type_matches)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for performance issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;performance|slow|optimization|index|partition&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> performance_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;performance|slow|optimization|index|partition&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;performance_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(performance_matches)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for DBT-specific issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt|incremental|model|materialization|ref|source&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt|incremental|model|materialization|ref|source&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;dbt_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(dbt_matches)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for test-related issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;test|assert|validation|quality&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> test_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;test|assert|validation|quality&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> analysis[&lt;span style="color:#e6db74">&amp;#34;test_issues&amp;#34;&lt;/span>] &lt;span style="color:#f92672">=&lt;/span> len(test_matches)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check for formatting issues&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>search(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;format|indent|spaces|uppercase|lowercase&amp;#34;&lt;/span>, review_text, re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> formatting_matches &lt;span style="color:#f92672">=&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Call the LLM API (using a different model if possible)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> USE_CLAUDE:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> client&lt;span style="color:#f92672">.&lt;/span>messages&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;claude-3-sonnet-20240229&amp;#34;&lt;/span>, &lt;span style="color:#75715e"># Using a different model&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> max_tokens&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">2000&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;You are a Critical Reviewer who verifies the accuracy of code reviews. You only point out factual errors and hallucinations.&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: verification_prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> verification &lt;span style="color:#f92672">=&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>content[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> openai&lt;span style="color:#f92672">.&lt;/span>ChatCompletion&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;gpt-4-turbo&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;system&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;You are a Critical Reviewer who verifies the accuracy of code reviews. You only point out factual errors and hallucinations.&amp;#34;&lt;/span>},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: verification_prompt}
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> verification &lt;span style="color:#f92672">=&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>choices[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>message&lt;span style="color:#f92672">.&lt;/span>content
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Only include verification notes if issues were found&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;appears accurate&amp;#34;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> verification&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Verification Notes
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>verification&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Then update the main function to include this verification step:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">build_rag_context_for_review&lt;/span>(file_path, file_content):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Build a comprehensive RAG context for the file being reviewed&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Determine the type of file and construct appropriate queries&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_sql &lt;span style="color:#f92672">=&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_dbt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_path &lt;span style="color:#f92672">or&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Always include SQL standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;SQL coding standards and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">5&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sql_standards_results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# SQL STANDARDS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> sql_standards_results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column names from the file to query for naming patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_sql:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Simple regex to extract column names from SELECT statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># In a production system, use a proper SQL parser&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> select_clauses &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SELECT\s+(.*?)\s+FROM&amp;#34;&lt;/span>, file_content, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> clause &lt;span style="color:#f92672">in&lt;/span> select_clauses:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;*&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> clause:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> cols &lt;span style="color:#f92672">=&lt;/span> clause&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> cols:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span>)[&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;.&amp;#39;&lt;/span>)[&lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names&lt;span style="color:#f92672">.&lt;/span>append(col)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Also extract from CREATE TABLE statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> create_table_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;CREATE\s+TABLE\s+\w+\s*\((.*?)\)&amp;#34;&lt;/span>, file_content, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> columns_text &lt;span style="color:#f92672">in&lt;/span> create_table_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> cols &lt;span style="color:#f92672">=&lt;/span> columns_text&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> cols:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> col &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>startswith(&lt;span style="color:#e6db74">&amp;#39;--&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_parts &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(col_parts) &lt;span style="color:#f92672">&amp;gt;=&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names&lt;span style="color:#f92672">.&lt;/span>append(col_parts[&lt;span style="color:#ae81ff">0&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for naming patterns information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_name &lt;span style="color:#f92672">in&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column named &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>] &lt;span style="color:#f92672">and&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> naming_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# COLUMN NAMING PATTERNS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(naming_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for data type information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_name &lt;span style="color:#f92672">in&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Data type for column &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;data_types&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>] &lt;span style="color:#f92672">and&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> data_type_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# DATA TYPE PATTERNS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(data_type_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for DBT patterns if applicable&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_dbt:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for specific DBT features in the file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ config(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT config block&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ ref(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT ref function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ source(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT source function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ dbt_utils&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT utils package&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;incremental&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;incremental model&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> feature &lt;span style="color:#f92672">in&lt;/span> dbt_features:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;DBT &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>feature&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> patterns and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>] &lt;span style="color:#f92672">and&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results[&lt;span style="color:#e6db74">&amp;#34;documents&amp;#34;&lt;/span>][&lt;span style="color:#ae81ff">0&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dbt_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# DBT PATTERNS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(dbt_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine all context sections&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#f92672">.&lt;/span>join(context_sections)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">main&lt;/span>():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First, extract knowledge from the repository to build/update the vector database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># This could be done less frequently, such as on a schedule rather than for every PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> extract_code_knowledge(repository)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get the files changed in the PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> files &lt;span style="color:#f92672">=&lt;/span> pull_request&lt;span style="color:#f92672">.&lt;/span>get_files()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Track all feedback to post in one comment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;# AI Data Engineer Review for PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr_number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Initialize metrics summary&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_consistency_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_best_practice_issues&amp;#34;&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze each file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file &lt;span style="color:#f92672">in&lt;/span> files:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Only process SQL and DBT files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#f92672">.&lt;/span>endswith((&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.dbt&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> print(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Analyzing &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> requests&lt;span style="color:#f92672">.&lt;/span>get(file&lt;span style="color:#f92672">.&lt;/span>raw_url)&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get primary feedback&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> primary_feedback &lt;span style="color:#f92672">=&lt;/span> analyze_file(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content, sql_standards, naming_dict, repo_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Verify the feedback&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> verification &lt;span style="color:#f92672">=&lt;/span> verify_feedback(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content, primary_feedback, sql_standards)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Check custom rules&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_issues &lt;span style="color:#f92672">=&lt;/span> check_custom_rules(file_content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Analyze naming consistency&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> consistency_result &lt;span style="color:#f92672">=&lt;/span> analyze_naming_consistency(file_content, naming_dict)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build custom rules and metrics section&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> custom_issues &lt;span style="color:#f92672">or&lt;/span> consistency_result[&lt;span style="color:#e6db74">&amp;#34;inconsistent_columns&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;## Custom Rules and Metrics&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;**Naming Consistency Score**: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>consistency_result[&lt;span style="color:#e6db74">&amp;#39;score&amp;#39;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">%&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> consistency_result[&lt;span style="color:#e6db74">&amp;#34;inconsistent_columns&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;**Inconsistent Column Names/Types**:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> consistency_result[&lt;span style="color:#e6db74">&amp;#34;inconsistent_columns&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> custom_issues:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;**Custom Rule Violations**:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> issue &lt;span style="color:#f92672">in&lt;/span> custom_issues:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>issue&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> custom_section &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine all feedback sections&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_feedback &lt;span style="color:#f92672">=&lt;/span> primary_feedback
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> custom_section:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>custom_section&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> verification:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>verification&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Update metrics&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> len(custom_issues)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;avg_naming_consistency&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> consistency_result[&lt;span style="color:#e6db74">&amp;#34;score&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;custom_rule_violations&amp;#34;&lt;/span>] &lt;span style="color:#f92672">+=&lt;/span> len(custom_issues)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;## File: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_feedback&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">---&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Calculate average naming consistency&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>] &lt;span style="color:#f92672">&amp;gt;&lt;/span> &lt;span style="color:#ae81ff">0&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics[&lt;span style="color:#e6db74">&amp;#34;avg_naming_consistency&amp;#34;&lt;/span>] &lt;span style="color:#f92672">/=&lt;/span> metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add metrics summary&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics_summary &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Review Metrics
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Files Analyzed: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Issues Found: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>metrics[&lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Average Naming Consistency: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>metrics[&lt;span style="color:#e6db74">&amp;#34;avg_naming_consistency&amp;#34;&lt;/span>]&lt;span style="color:#e6db74">:&lt;/span>&lt;span style="color:#e6db74">.1f&lt;/span>&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">%
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> - Custom Rule Violations: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>metrics[&lt;span style="color:#e6db74">&amp;#34;custom_rule_violations&amp;#34;&lt;/span>]&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> metrics_summary
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Add a conclusion&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> all_feedback &lt;span style="color:#f92672">+=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> **Note**: This review was generated by an AI assistant. While it strives for accuracy, always use your judgment when implementing suggestions.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If you have feedback on how to improve these automated reviews, please let us know!
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Post the combined review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> post_review_comment(pull_request, all_feedback)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This multi-agent approach helps ensure that the reviews your team receives are more accurate and reliable. The second agent acts as a fact-checker, flagging any hallucinations or misinterpretations.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="step-6-implementing-github-action-using-your-existing-framework">Step 6: Implementing GitHub Action Using Your Existing Framework&lt;/h2>
&lt;/br>
&lt;p>Now that we&amp;rsquo;ve established our vector database for contextual code reviews, let&amp;rsquo;s integrate it with GitHub Actions using your existing framework. We&amp;rsquo;ll build on the generate_comment.py script and autofeedback.yml workflow you&amp;rsquo;ve already created.&lt;/p>
&lt;p>First, let&amp;rsquo;s update the generate_comment.py file to use our RAG-powered approach:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> os
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> sys
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> requests
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> logging
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> re
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> base64
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">import&lt;/span> pinecone
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> openai &lt;span style="color:#f92672">import&lt;/span> OpenAI
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> github &lt;span style="color:#f92672">import&lt;/span> Github
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">from&lt;/span> sentence_transformers &lt;span style="color:#f92672">import&lt;/span> SentenceTransformer
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Setup logging&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>logger &lt;span style="color:#f92672">=&lt;/span> logging&lt;span style="color:#f92672">.&lt;/span>getLogger()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>formatter &lt;span style="color:#f92672">=&lt;/span> logging&lt;span style="color:#f92672">.&lt;/span>Formatter(&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#e6db74">%(message)s&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>stream_handler &lt;span style="color:#f92672">=&lt;/span> logging&lt;span style="color:#f92672">.&lt;/span>StreamHandler(sys&lt;span style="color:#f92672">.&lt;/span>stdout)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>stream_handler&lt;span style="color:#f92672">.&lt;/span>setFormatter(formatter)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>logger&lt;span style="color:#f92672">.&lt;/span>addHandler(stream_handler)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>logger&lt;span style="color:#f92672">.&lt;/span>setLevel(logging&lt;span style="color:#f92672">.&lt;/span>INFO)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Environment variables&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>GIT_TOKEN &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#39;GIT_TOKEN&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>GIT_REPO &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#39;GITHUB_REPOSITORY&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>OPENAI_API_KEY &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#34;OPENAI_API_KEY&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PINECONE_API_KEY &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#34;PINECONE_API_KEY&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>PINECONE_ENVIRONMENT &lt;span style="color:#f92672">=&lt;/span> os&lt;span style="color:#f92672">.&lt;/span>environ&lt;span style="color:#f92672">.&lt;/span>get(&lt;span style="color:#e6db74">&amp;#34;PINECONE_ENVIRONMENT&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;gcp-starter&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> OPENAI_API_KEY &lt;span style="color:#f92672">is&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">ValueError&lt;/span>(&lt;span style="color:#e6db74">&amp;#34;You need to specify OPENAI_API_KEY environment variable!&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> PINECONE_API_KEY &lt;span style="color:#f92672">is&lt;/span> &lt;span style="color:#66d9ef">None&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">raise&lt;/span> &lt;span style="color:#a6e22e">ValueError&lt;/span>(&lt;span style="color:#e6db74">&amp;#34;You need to specify PINECONE_API_KEY environment variable!&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize OpenAI client&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>client &lt;span style="color:#f92672">=&lt;/span> OpenAI(api_key&lt;span style="color:#f92672">=&lt;/span>OPENAI_API_KEY)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize embedding model&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>model &lt;span style="color:#f92672">=&lt;/span> SentenceTransformer(&lt;span style="color:#e6db74">&amp;#39;all-MiniLM-L6-v2&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>EMBEDDING_DIMENSION &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">384&lt;/span> &lt;span style="color:#75715e"># Dimension for all-MiniLM-L6-v2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Initialize Pinecone&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>INDEX_NAME &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;data-engineer-reviewer&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>pinecone&lt;span style="color:#f92672">.&lt;/span>init(api_key&lt;span style="color:#f92672">=&lt;/span>PINECONE_API_KEY, environment&lt;span style="color:#f92672">=&lt;/span>PINECONE_ENVIRONMENT)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Create index if it doesn&amp;#39;t exist&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> INDEX_NAME &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> pinecone&lt;span style="color:#f92672">.&lt;/span>list_indexes():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pinecone&lt;span style="color:#f92672">.&lt;/span>create_index(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> name&lt;span style="color:#f92672">=&lt;/span>INDEX_NAME,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dimension&lt;span style="color:#f92672">=&lt;/span>EMBEDDING_DIMENSION,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metric&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;cosine&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> shards&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># Connect to the index&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>index &lt;span style="color:#f92672">=&lt;/span> pinecone&lt;span style="color:#f92672">.&lt;/span>Index(INDEX_NAME)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">get_pr_info&lt;/span>(pr_number):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Get PR files and information&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> g &lt;span style="color:#f92672">=&lt;/span> Github(GIT_TOKEN)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> repo &lt;span style="color:#f92672">=&lt;/span> g&lt;span style="color:#f92672">.&lt;/span>get_repo(GIT_REPO)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pr &lt;span style="color:#f92672">=&lt;/span> repo&lt;span style="color:#f92672">.&lt;/span>get_pull(pr_number)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> files &lt;span style="color:#f92672">=&lt;/span> pr&lt;span style="color:#f92672">.&lt;/span>get_files()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Filter for data engineering files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_files &lt;span style="color:#f92672">=&lt;/span> [f &lt;span style="color:#66d9ef">for&lt;/span> f &lt;span style="color:#f92672">in&lt;/span> files &lt;span style="color:#66d9ef">if&lt;/span> f&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#f92672">.&lt;/span>endswith((&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.dbt&amp;#39;&lt;/span>))]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;pr&amp;#34;&lt;/span>: pr,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;repo&amp;#34;&lt;/span>: repo,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;data_files&amp;#34;&lt;/span>: data_files
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">extract_repository_knowledge&lt;/span>(repo):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Extract knowledge from the repository to populate the vector database&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">&amp;#34;Extracting knowledge from repository...&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Lists to store our embeddings and metadata&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> data_type_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First, try to find SQL standards document&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards_file &lt;span style="color:#f92672">=&lt;/span> repo&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;docs/sql_standards.md&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> standards_text &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(standards_file&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Split into sections for better retrieval granularity&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sections &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#39;#{2,3}\s+&amp;#39;&lt;/span>, standards_text)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, section &lt;span style="color:#f92672">in&lt;/span> enumerate(sections):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> section&lt;span style="color:#f92672">.&lt;/span>strip():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> section&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;std_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;section_index&amp;#34;&lt;/span>: i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found SQL standards document with &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(sections)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> sections&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># If no standards file, use default standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">&amp;#34;No SQL standards document found, using defaults&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> default_standards &lt;span style="color:#f92672">=&lt;/span> [
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use snake_case for all database object names&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Primary keys should be named &amp;#39;id&amp;#39; or &amp;#39;table_name_id&amp;#39;&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Foreign keys should match the referenced column name&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use UPPERCASE for SQL keywords&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Indent with 2 spaces&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Place each column on its own line in SELECT statements&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Include a WHERE clause to avoid full table scans&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;Use consistent data types for similar fields across tables&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i, standard &lt;span style="color:#f92672">in&lt;/span> enumerate(default_standards):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(standard)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;default_std_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>i&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: standard,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;default_standards&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;section_index&amp;#34;&lt;/span>: i
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Find existing models and their patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for DBT models directory&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> models_contents &lt;span style="color:#f92672">=&lt;/span> repo&lt;span style="color:#f92672">.&lt;/span>get_contents(&lt;span style="color:#e6db74">&amp;#34;models&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model_files &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Recursive function to collect all SQL files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">collect_sql_files&lt;/span>(contents):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> item &lt;span style="color:#f92672">in&lt;/span> contents:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> item&lt;span style="color:#f92672">.&lt;/span>type &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;dir&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> collect_sql_files(repo&lt;span style="color:#f92672">.&lt;/span>get_contents(item&lt;span style="color:#f92672">.&lt;/span>path))
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">elif&lt;/span> item&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>endswith((&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model_files&lt;span style="color:#f92672">.&lt;/span>append(item)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> collect_sql_files(models_contents)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Found &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(model_files)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> model files to analyze&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Sample a subset of files to analyze (to avoid API rate limits)&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_sample &lt;span style="color:#f92672">=&lt;/span> model_files[:&lt;span style="color:#ae81ff">50&lt;/span>] &lt;span style="color:#75715e"># Adjust based on your needs&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file_content &lt;span style="color:#f92672">in&lt;/span> file_sample:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_text &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(file_content&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column information from SQL files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column references in SELECT statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> select_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SELECT\s+(.*?)\s+FROM&amp;#34;&lt;/span>, file_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> select_clause &lt;span style="color:#f92672">in&lt;/span> select_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;*&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> select_clause: &lt;span style="color:#75715e"># Skip SELECT *&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> columns &lt;span style="color:#f92672">=&lt;/span> select_clause&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> columns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> col &lt;span style="color:#f92672">and&lt;/span> &lt;span style="color:#f92672">not&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>startswith(&lt;span style="color:#e6db74">&amp;#39;--&amp;#39;&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Handle aliasing with &amp;#39;AS&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_parts &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span>, &lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> col_parts[&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col_name &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;.&amp;#39;&lt;/span>)[&lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column name &amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39; is used in SELECT statement in file &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;np_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(naming_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;column_name&amp;#34;&lt;/span>: col_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;select_statement&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract DBT-specific patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>path:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get a snippet of the file for context&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> file_text[:&lt;span style="color:#ae81ff">1000&lt;/span>] &lt;span style="color:#75715e"># First 1000 chars for context&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_file_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(dbt_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;dbt_file&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for specific DBT patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;{{ config(&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_text:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config_pattern &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;{{[\s]*config\((.*?)\)[\s]*}}&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> config_matches &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(config_pattern, file_text, re&lt;span style="color:#f92672">.&lt;/span>DOTALL)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> config_content &lt;span style="color:#f92672">in&lt;/span> config_matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> text &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;DBT config in &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>config_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(text)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_patterns_data&lt;span style="color:#f92672">.&lt;/span>append({
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;id&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_config_&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(dbt_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;values&amp;#34;&lt;/span>: embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;metadata&amp;#34;&lt;/span>: {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>: text,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;file_path&amp;#34;&lt;/span>: file_content&lt;span style="color:#f92672">.&lt;/span>path,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;source&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;dbt_config&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> })
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error processing file &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#f92672">.&lt;/span>path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>warning(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Could not process models directory: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Update the Pinecone index with the extracted knowledge&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> update_pinecone_index(sql_standards_data, naming_patterns_data, data_type_patterns_data, dbt_patterns_data)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Knowledge extraction complete.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Indexed: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(sql_standards_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> standard sections, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(naming_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> naming patterns, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(data_type_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> data type patterns, &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>len(dbt_patterns_data)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> DBT patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">update_pinecone_index&lt;/span>(sql_standards, naming_patterns, data_type_patterns, dbt_patterns):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Update the Pinecone index with extracted knowledge&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># First, delete existing vectors by namespace&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data_types&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>delete(delete_all&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert SQL standards data&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sql_standards:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>sql_standards, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert naming patterns data&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> naming_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Pinecone has limits on batch size, so chunk if needed&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk_size &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#ae81ff">100&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(naming_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> naming_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert data type patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> data_type_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(data_type_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> data_type_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;data_types&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Upsert DBT patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dbt_patterns:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> i &lt;span style="color:#f92672">in&lt;/span> range(&lt;span style="color:#ae81ff">0&lt;/span>, len(dbt_patterns), chunk_size):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> chunk &lt;span style="color:#f92672">=&lt;/span> dbt_patterns[i:i &lt;span style="color:#f92672">+&lt;/span> chunk_size]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> index&lt;span style="color:#f92672">.&lt;/span>upsert(vectors&lt;span style="color:#f92672">=&lt;/span>chunk, namespace&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error updating Pinecone index: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">query_repository_knowledge&lt;/span>(query, namespace, top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">5&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Query the Pinecone index for relevant knowledge&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> query_embedding &lt;span style="color:#f92672">=&lt;/span> model&lt;span style="color:#f92672">.&lt;/span>encode(query)&lt;span style="color:#f92672">.&lt;/span>tolist()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> index&lt;span style="color:#f92672">.&lt;/span>query(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> vector&lt;span style="color:#f92672">=&lt;/span>query_embedding,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>top_k,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> include_metadata&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> namespace&lt;span style="color:#f92672">=&lt;/span>namespace
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract the text from the metadata&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results&lt;span style="color:#f92672">.&lt;/span>matches:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> retrieved_texts &lt;span style="color:#f92672">=&lt;/span> [&lt;span style="color:#66d9ef">match&lt;/span>&lt;span style="color:#f92672">.&lt;/span>metadata[&lt;span style="color:#e6db74">&amp;#34;text&amp;#34;&lt;/span>] &lt;span style="color:#66d9ef">for&lt;/span> &lt;span style="color:#66d9ef">match&lt;/span> &lt;span style="color:#f92672">in&lt;/span> results&lt;span style="color:#f92672">.&lt;/span>matches]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> retrieved_texts
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error querying Pinecone: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">build_rag_context_for_review&lt;/span>(file_path, file_content):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Build a comprehensive RAG context for the file being reviewed&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Determine the type of file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_sql &lt;span style="color:#f92672">=&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.sql&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> is_dbt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#39;models/&amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_path &lt;span style="color:#f92672">or&lt;/span> file_path&lt;span style="color:#f92672">.&lt;/span>endswith(&lt;span style="color:#e6db74">&amp;#39;.yml&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Always include SQL standards&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sql_standards_results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;SQL coding standards and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;sql_standards&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">5&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> sql_standards_results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# SQL STANDARDS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> sql_standards_results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract column names from the file to query for naming patterns&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_sql:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Simple regex to extract column names from SELECT statements&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> select_clauses &lt;span style="color:#f92672">=&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>findall(&lt;span style="color:#e6db74">r&lt;/span>&lt;span style="color:#e6db74">&amp;#34;SELECT\s+(.*?)\s+FROM&amp;#34;&lt;/span>, file_content, re&lt;span style="color:#f92672">.&lt;/span>DOTALL &lt;span style="color:#f92672">|&lt;/span> re&lt;span style="color:#f92672">.&lt;/span>IGNORECASE)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> clause &lt;span style="color:#f92672">in&lt;/span> select_clauses:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39;*&amp;#39;&lt;/span> &lt;span style="color:#f92672">not&lt;/span> &lt;span style="color:#f92672">in&lt;/span> clause:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> cols &lt;span style="color:#f92672">=&lt;/span> clause&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;,&amp;#39;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col &lt;span style="color:#f92672">in&lt;/span> cols:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>upper():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39; AS &amp;#39;&lt;/span>)[&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">else&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> col &lt;span style="color:#f92672">=&lt;/span> col&lt;span style="color:#f92672">.&lt;/span>split(&lt;span style="color:#e6db74">&amp;#39;.&amp;#39;&lt;/span>)[&lt;span style="color:#f92672">-&lt;/span>&lt;span style="color:#ae81ff">1&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>strip()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> column_names&lt;span style="color:#f92672">.&lt;/span>append(col)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for naming patterns information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> col_name &lt;span style="color:#f92672">in&lt;/span> column_names:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Column named &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>col_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> naming_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> naming_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# COLUMN NAMING PATTERNS&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(naming_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Query for DBT patterns if applicable&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_dbt:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for specific DBT features in the file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ config(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT config block&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ ref(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT ref function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ source(&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT source function&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;{{ dbt_utils&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;DBT utils package&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;incremental&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> file_content&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_features&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;incremental model&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> feature &lt;span style="color:#f92672">in&lt;/span> dbt_features:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> results &lt;span style="color:#f92672">=&lt;/span> query_repository_knowledge(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;DBT &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>feature&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74"> patterns and best practices&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_patterns&amp;#34;&lt;/span>, 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> top_k&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">3&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc &lt;span style="color:#f92672">in&lt;/span> results:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> dbt_context&lt;span style="color:#f92672">.&lt;/span>append(doc)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> dbt_context:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">&amp;#34;# DBT PATTERNS AND BEST PRACTICES&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> context_sections&lt;span style="color:#f92672">.&lt;/span>extend(dbt_context)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Combine all context sections&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#f92672">.&lt;/span>join(context_sections)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">get_feedback&lt;/span>(file_path, file_content):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Generate AI review feedback using RAG context&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Build RAG context from the vector database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> rag_context &lt;span style="color:#f92672">=&lt;/span> build_rag_context_for_review(file_path, file_content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># System prompt for the Senior Data Engineer persona&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> system_prompt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You are an expert Senior Data Engineer responsible for reviewing SQL and data transformation code.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> You have extensive knowledge of SQL, DBT, data modeling, and data engineering best practices.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Your feedback should be detailed, specific, and constructive, focusing on:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 1. SQL formatting and adherence to standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 2. Naming consistency with existing patterns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 3. Data type consistency
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 4. Idempotency issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 5. Missing data quality checks or tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 6. Proper time-based modeling (SCD) implementation
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 7. Performance optimization opportunities
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Always provide line numbers and specific suggestions for improvement.
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># User prompt with the RAG context and file to review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> user_prompt &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> I need you to review a data engineering file. Here&amp;#39;s some context about our codebase standards and patterns:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>rag_context&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> # FILE TO REVIEW
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Path: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_content&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ```
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Please provide a comprehensive review focusing on:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 1. SQL formatting and adherence to our standards
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 2. Naming consistency with existing patterns in our codebase
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 3. Data type consistency with how we typically define these columns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 4. Idempotency issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 5. Missing data quality checks or tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 6. If it&amp;#39;s DBT code, proper use of dbt_valid_from and dbt_valid_to for time-based models
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 7. Performance concerns
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Format your response as:
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Summary
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> A brief overview of the code quality (1-2 sentences)
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Issues
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Describe each issue you find with line numbers and specific recommendations
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Suggestions
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> Provide constructive suggestions for improvement
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> 
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> ## Proposed Tests
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> If relevant, suggest tests that would improve data quality
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> &amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Call the OpenAI API&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> response &lt;span style="color:#f92672">=&lt;/span> client&lt;span style="color:#f92672">.&lt;/span>chat&lt;span style="color:#f92672">.&lt;/span>completions&lt;span style="color:#f92672">.&lt;/span>create(
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> model&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#e6db74">&amp;#34;gpt-4&amp;#34;&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> messages&lt;span style="color:#f92672">=&lt;/span>[
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;system&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: system_prompt},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> {&lt;span style="color:#e6db74">&amp;#34;role&amp;#34;&lt;/span>: &lt;span style="color:#e6db74">&amp;#34;user&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;content&amp;#34;&lt;/span>: user_prompt},
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> ],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> temperature&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#ae81ff">0&lt;/span>,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> )
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get the review content&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> review &lt;span style="color:#f92672">=&lt;/span> response&lt;span style="color:#f92672">.&lt;/span>choices[&lt;span style="color:#ae81ff">0&lt;/span>]&lt;span style="color:#f92672">.&lt;/span>message&lt;span style="color:#f92672">.&lt;/span>content
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Format as a GitHub comment&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> comment &lt;span style="color:#f92672">=&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;# Senior Data Engineer Review for `&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">`&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>review&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">---&lt;/span>&lt;span style="color:#ae81ff">\n\n&lt;/span>&lt;span style="color:#e6db74">*This automated review was generated using AI and repository knowledge. If you have feedback on improving these reviews, please let the team know!*&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> comment
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error generating review: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error generating review for `&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file_path&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">`: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">post_github_comment&lt;/span>(pr, comment):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Post a comment to the GitHub PR&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pr&lt;span style="color:#f92672">.&lt;/span>create_issue_comment(comment)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;✅ Posted review comment to PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr&lt;span style="color:#f92672">.&lt;/span>number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#66d9ef">True&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error posting comment: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> &lt;span style="color:#66d9ef">False&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">main&lt;/span>(pr_number):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Main function to process PR and generate reviews&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Starting review process for PR #&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>pr_number&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get PR information&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pr_info &lt;span style="color:#f92672">=&lt;/span> get_pr_info(pr_number)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#f92672">not&lt;/span> pr_info[&lt;span style="color:#e6db74">&amp;#34;data_files&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">&amp;#34;No SQL, YML, or DBT files found in this PR. Skipping review.&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Extract knowledge from repository to build or update vector database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># This could be done less frequently in production&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> extract_repository_knowledge(pr_info[&lt;span style="color:#e6db74">&amp;#34;repo&amp;#34;&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Process each data file in the PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> file &lt;span style="color:#f92672">in&lt;/span> pr_info[&lt;span style="color:#e6db74">&amp;#34;data_files&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>info(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Reviewing file: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Get file content&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> file_content &lt;span style="color:#f92672">=&lt;/span> requests&lt;span style="color:#f92672">.&lt;/span>get(file&lt;span style="color:#f92672">.&lt;/span>raw_url)&lt;span style="color:#f92672">.&lt;/span>text
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Generate AI review feedback&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> review_comment &lt;span style="color:#f92672">=&lt;/span> get_feedback(file&lt;span style="color:#f92672">.&lt;/span>filename, file_content)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Post comment to PR&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> post_github_comment(pr_info[&lt;span style="color:#e6db74">&amp;#34;pr&amp;#34;&lt;/span>], review_comment)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error processing &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>file&lt;span style="color:#f92672">.&lt;/span>filename&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">if&lt;/span> __name__ &lt;span style="color:#f92672">==&lt;/span> &lt;span style="color:#e6db74">&amp;#34;__main__&amp;#34;&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> len(sys&lt;span style="color:#f92672">.&lt;/span>argv) &lt;span style="color:#f92672">&amp;lt;&lt;/span> &lt;span style="color:#ae81ff">2&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">&amp;#34;PR number is required as an argument&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> sys&lt;span style="color:#f92672">.&lt;/span>exit(&lt;span style="color:#ae81ff">1&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> pr_number &lt;span style="color:#f92672">=&lt;/span> int(sys&lt;span style="color:#f92672">.&lt;/span>argv[&lt;span style="color:#ae81ff">1&lt;/span>])
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> main(pr_number)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Next, let&amp;rsquo;s update the GitHub Actions workflow file (autofeedback.yml) to include the Pinecone API key and install the necessary dependencies:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Senior Data Engineer AI Review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">on&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pull_request&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">types&lt;/span>: [&lt;span style="color:#ae81ff">opened, synchronize]&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">paths&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**.sql&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**/models/**.yml&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#39;**/dbt_project.yml&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">jobs&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">review&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">runs-on&lt;/span>: &lt;span style="color:#ae81ff">ubuntu-latest&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">permissions&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">contents&lt;/span>: &lt;span style="color:#ae81ff">read&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">pull-requests&lt;/span>: &lt;span style="color:#ae81ff">write&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">steps&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Checkout code&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">uses&lt;/span>: &lt;span style="color:#ae81ff">actions/checkout@v2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">with&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">fetch-depth&lt;/span>: &lt;span style="color:#ae81ff">0&lt;/span> &lt;span style="color:#75715e"># Fetch all history to analyze codebase&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Set up Python&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">uses&lt;/span>: &lt;span style="color:#ae81ff">actions/setup-python@v2&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">with&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">python-version&lt;/span>: &lt;span style="color:#e6db74">&amp;#39;3.10&amp;#39;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Install dependencies&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> python -m pip install --upgrade pip
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> pip install PyGithub requests openai pinecone-client sentence-transformers&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#f92672">name&lt;/span>: &lt;span style="color:#ae81ff">Run Data Engineer AI Review&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">env&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">GIT_TOKEN&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.GIT_TOKEN }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">GITHUB_REPOSITORY&lt;/span>: &lt;span style="color:#ae81ff">${{ github.repository }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">OPENAI_API_KEY&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.OPENAI_API_KEY }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">PINECONE_API_KEY&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.PINECONE_API_KEY }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">PINECONE_ENVIRONMENT&lt;/span>: &lt;span style="color:#ae81ff">${{ secrets.PINECONE_ENVIRONMENT }}&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">run&lt;/span>: |&lt;span style="color:#e6db74">
&lt;/span>&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#e6db74"> python src/generate_comment.py ${{ github.event.pull_request.number }}&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>Finally, create a requirements.txt file to ensure all dependencies are installed:&lt;/p>



&lt;div class="goat svg-container ">
 
 &lt;svg
 xmlns="http://www.w3.org/2000/svg"
 font-family="Menlo,Lucida Console,monospace"
 
 viewBox="0 0 232 89"
 >
 &lt;g transform='translate(8,16)'>
&lt;text text-anchor='middle' x='0' y='4' fill='currentColor' style='font-size:1em'>P&lt;/text>
&lt;text text-anchor='middle' x='0' y='20' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='0' y='36' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='0' y='52' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='0' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='8' y='4' fill='currentColor' style='font-size:1em'>y&lt;/text>
&lt;text text-anchor='middle' x='8' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='8' y='36' fill='currentColor' style='font-size:1em'>p&lt;/text>
&lt;text text-anchor='middle' x='8' y='52' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='8' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='16' y='4' fill='currentColor' style='font-size:1em'>G&lt;/text>
&lt;text text-anchor='middle' x='16' y='20' fill='currentColor' style='font-size:1em'>q&lt;/text>
&lt;text text-anchor='middle' x='16' y='36' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='16' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='16' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='24' y='4' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='24' y='20' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='24' y='36' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='24' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='24' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='32' y='4' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='32' y='20' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='32' y='36' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='32' y='52' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='32' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='40' y='4' fill='currentColor' style='font-size:1em'>h&lt;/text>
&lt;text text-anchor='middle' x='40' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='40' y='36' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='40' y='52' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='40' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='48' y='4' fill='currentColor' style='font-size:1em'>u&lt;/text>
&lt;text text-anchor='middle' x='48' y='20' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='48' y='36' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='48' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='48' y='68' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='56' y='4' fill='currentColor' style='font-size:1em'>b&lt;/text>
&lt;text text-anchor='middle' x='56' y='20' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='56' y='36' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='56' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='56' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='64' y='4' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='64' y='20' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='64' y='36' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='64' y='52' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='64' y='68' fill='currentColor' style='font-size:1em'>-&lt;/text>
&lt;text text-anchor='middle' x='72' y='4' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='72' y='20' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='72' y='36' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='72' y='52' fill='currentColor' style='font-size:1em'>c&lt;/text>
&lt;text text-anchor='middle' x='72' y='68' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='80' y='4' fill='currentColor' style='font-size:1em'>1&lt;/text>
&lt;text text-anchor='middle' x='80' y='20' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='80' y='36' fill='currentColor' style='font-size:1em'>3&lt;/text>
&lt;text text-anchor='middle' x='80' y='52' fill='currentColor' style='font-size:1em'>l&lt;/text>
&lt;text text-anchor='middle' x='80' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='88' y='4' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='88' y='20' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='88' y='36' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='88' y='52' fill='currentColor' style='font-size:1em'>i&lt;/text>
&lt;text text-anchor='middle' x='88' y='68' fill='currentColor' style='font-size:1em'>a&lt;/text>
&lt;text text-anchor='middle' x='96' y='4' fill='currentColor' style='font-size:1em'>5&lt;/text>
&lt;text text-anchor='middle' x='96' y='20' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='96' y='36' fill='currentColor' style='font-size:1em'>0&lt;/text>
&lt;text text-anchor='middle' x='96' y='52' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='96' y='68' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='4' fill='currentColor' style='font-size:1em'>8&lt;/text>
&lt;text text-anchor='middle' x='104' y='20' fill='currentColor' style='font-size:1em'>8&lt;/text>
&lt;text text-anchor='middle' x='104' y='52' fill='currentColor' style='font-size:1em'>n&lt;/text>
&lt;text text-anchor='middle' x='104' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='112' y='4' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='112' y='20' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='112' y='52' fill='currentColor' style='font-size:1em'>t&lt;/text>
&lt;text text-anchor='middle' x='112' y='68' fill='currentColor' style='font-size:1em'>f&lt;/text>
&lt;text text-anchor='middle' x='120' y='4' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='120' y='20' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='120' y='52' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='120' y='68' fill='currentColor' style='font-size:1em'>o&lt;/text>
&lt;text text-anchor='middle' x='128' y='52' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='128' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='136' y='52' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='136' y='68' fill='currentColor' style='font-size:1em'>m&lt;/text>
&lt;text text-anchor='middle' x='144' y='52' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='144' y='68' fill='currentColor' style='font-size:1em'>e&lt;/text>
&lt;text text-anchor='middle' x='152' y='52' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='152' y='68' fill='currentColor' style='font-size:1em'>r&lt;/text>
&lt;text text-anchor='middle' x='160' y='52' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='160' y='68' fill='currentColor' style='font-size:1em'>s&lt;/text>
&lt;text text-anchor='middle' x='168' y='52' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='168' y='68' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='176' y='68' fill='currentColor' style='font-size:1em'>=&lt;/text>
&lt;text text-anchor='middle' x='184' y='68' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='192' y='68' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='200' y='68' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;text text-anchor='middle' x='208' y='68' fill='currentColor' style='font-size:1em'>.&lt;/text>
&lt;text text-anchor='middle' x='216' y='68' fill='currentColor' style='font-size:1em'>2&lt;/text>
&lt;/g>

 &lt;/svg>
 
&lt;/div>
&lt;h3 id="setting-up-the-secrets">Setting Up the Secrets&lt;/h3>
&lt;p>For this implementation to work, you&amp;rsquo;ll need to add these secrets to your GitHub repository:&lt;/p>
&lt;ol>
&lt;li>Navigate to your repository on GitHub&lt;/li>
&lt;li>Go to &lt;strong>Settings&lt;/strong> → &lt;strong>Secrets and variables&lt;/strong> → &lt;strong>Actions&lt;/strong>&lt;/li>
&lt;li>Add the following secrets:
&lt;ul>
&lt;li>&lt;code>GIT_TOKEN&lt;/code>: Your GitHub Personal Access Token&lt;/li>
&lt;li>&lt;code>OPENAI_API_KEY&lt;/code>: Your OpenAI API key&lt;/li>
&lt;li>&lt;code>PINECONE_API_KEY&lt;/code>: Your Pinecone API key&lt;/li>
&lt;li>&lt;code>PINECONE_ENVIRONMENT&lt;/code>: Your Pinecone environment (e.g., &amp;ldquo;gcp-starter&amp;rdquo;)&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ol>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="how-it-works-the-end-to-end-process">How It Works: The End-to-End Process&lt;/h2>
&lt;/br>
&lt;p>Let&amp;rsquo;s walk through the complete workflow of our AI Data Engineer review system:&lt;/p>
&lt;h3 id="1-initial-knowledge-extraction">1. Initial Knowledge Extraction&lt;/h3>
&lt;p>When the GitHub Action first runs on a repository, it performs these steps:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Extracts SQL Standards&lt;/strong>: It looks for a standards document in the repository or uses default standards.&lt;/li>
&lt;li>&lt;strong>Analyses Existing Code&lt;/strong>: It processes a sample of SQL and DBT files to learn naming conventions and data types.&lt;/li>
&lt;li>&lt;strong>Encodes Knowledge into Vectors&lt;/strong>: It converts all discovered patterns and standards into embeddings.&lt;/li>
&lt;li>&lt;strong>Populates the Vector Database&lt;/strong>: It creates namespaces in Pinecone and uploads the vectors with metadata.&lt;/li>
&lt;/ol>
&lt;p>This creates a semantic search index that understands your codebase&amp;rsquo;s specific conventions.&lt;/p>
&lt;h3 id="2-pull-request-triggered-review">2. Pull Request Triggered Review&lt;/h3>
&lt;p>When a developer creates or updates a Pull Request:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>GitHub Action Activates&lt;/strong>: The workflow triggers when SQL, DBT, or YAML files are modified.&lt;/li>
&lt;li>&lt;strong>Files are Extracted&lt;/strong>: The system gets the content of each changed file.&lt;/li>
&lt;li>&lt;strong>For Each File&lt;/strong>:
&lt;ul>
&lt;li>&lt;strong>RAG Context Building&lt;/strong>: It queries Pinecone for relevant standards and patterns based on the file content.&lt;/li>
&lt;li>&lt;strong>LLM Review Generation&lt;/strong>: It sends the file and RAG context to the LLM for detailed analysis.&lt;/li>
&lt;li>&lt;strong>Comment Posting&lt;/strong>: It formats the review and posts it to the PR.&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ol>
&lt;h3 id="3-the-developer-experience">3. The Developer Experience&lt;/h3>
&lt;p>From the developer&amp;rsquo;s perspective, the process feels seamless:&lt;/p>
&lt;ol>
&lt;li>They create a Pull Request with data engineering code changes.&lt;/li>
&lt;li>Within minutes, they receive a detailed, contextual review comment on their PR.&lt;/li>
&lt;li>The review highlights issues and suggestions specific to their organisation&amp;rsquo;s standards.&lt;/li>
&lt;li>They can address the feedback before human reviewers look at the code.&lt;/li>
&lt;/ol>
&lt;h3 id="4-the-review-content">4. The Review Content&lt;/h3>
&lt;p>The AI-generated review typically includes:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Summary&lt;/strong>: A brief overview of the code quality&lt;/li>
&lt;li>&lt;strong>Issues&lt;/strong>: Specific problems with line numbers and references&lt;/li>
&lt;li>&lt;strong>Suggestions&lt;/strong>: Constructive improvements&lt;/li>
&lt;li>&lt;strong>Proposed Tests&lt;/strong>: Data quality checks that should be added&lt;/li>
&lt;/ul>
&lt;figure>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-08-leveraging-llm-for-business-impact-part-2/images/ai-review-flow.png" width="800x">
&lt;/figure>

&lt;p>The most powerful aspect is that these reviews reflect your organisation&amp;rsquo;s actual practices, not generic rules. For example, if your team consistently uses &lt;code>account_number&lt;/code> as an INT in all tables, the AI will flag inconsistencies like &lt;code>acct_no&lt;/code> or VARCHAR types.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="benefits-and-business-impact">Benefits and Business Impact&lt;/h2>
&lt;/br>
&lt;p>Implementing an AI Data Engineer for code reviews delivers significant business value:&lt;/p>
&lt;h3 id="immediate-impact">Immediate Impact&lt;/h3>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>Improved Code Quality&lt;/strong>: Automatically enforces naming conventions, data type consistency, and SQL best practices.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: Reduced data pipeline errors by 42% after implementing AI-powered DBT model reviews.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Faster Code Reviews&lt;/strong>: Human reviewers can focus on business logic and architecture instead of style and conventions.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: Reduced PR review time from an average of 2 days to 4 hours by having the AI handle initial reviews.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Knowledge Transfer&lt;/strong>: New team members learn standards and patterns through specific, contextual feedback.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: New hires reach productivity milestones 3 weeks faster with AI-assisted code reviews.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;h3 id="long-term-benefits">Long-term Benefits&lt;/h3>
&lt;ol>
&lt;li>
&lt;p>&lt;strong>Consistency Across Teams&lt;/strong>: Enforces the same standards across distributed engineering teams and projects.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: Teams in two continents used AI reviews to maintain consistent data schemas across regions.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Technical Debt Prevention&lt;/strong>: Catches potential issues before they enter the codebase.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: Identified and fixed 85% of inefficient SQL patterns before deployment, reducing future refactoring costs.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Quality Metrics Tracking&lt;/strong>: By analysing reviews over time, organisations can monitor trends in data quality and engineering practices.&lt;/p>
&lt;p>&lt;em>Example&lt;/em>: Using AI review metrics to identify that 78% of quality issues stemmed from a specific class of SQL anti-patterns, leading to targeted training.&lt;/p>
&lt;/li>
&lt;/ol>
&lt;h3 id="roi-calculation">ROI Calculation&lt;/h3>
&lt;p>The investment in an AI Data Engineer review system can be justified by the significant returns:&lt;/p>
&lt;ul>
&lt;li>
&lt;p>&lt;strong>Time Savings&lt;/strong>: A team of 5 data engineers spending 1 hour each day on code reviews could save 1,250 hours annually (assuming 250 working days).&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Error Reduction&lt;/strong>: Catching data quality and performance issues early prevents costly production issues. A single critical bug in a data pipeline can cost $10,000+ in engineering time and business impact.&lt;/p>
&lt;/li>
&lt;li>
&lt;p>&lt;strong>Onboarding Efficiency&lt;/strong>: Faster knowledge transfer to new team members can save 2-3 weeks of ramp-up time per hire.&lt;/p>
&lt;/li>
&lt;/ul>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="advanced-configurations-and-extensions">Advanced Configurations and Extensions&lt;/h2>
&lt;/br>
&lt;p>The base AI Data Engineer review system can be extended in several ways to make it even more powerful:&lt;/p>
&lt;h3 id="customising-the-review-focus">Customising the Review Focus&lt;/h3>
&lt;p>You can create a configuration file in your repository to customise the AI reviewer&amp;rsquo;s behaviour:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-yaml" data-lang="yaml">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#75715e"># .github/ai-reviewer-config.yml&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">focus_areas&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">naming_conventions&lt;/span>: &lt;span style="color:#ae81ff">high&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">data_types&lt;/span>: &lt;span style="color:#ae81ff">high&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">performance&lt;/span>: &lt;span style="color:#ae81ff">medium&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">tests&lt;/span>: &lt;span style="color:#ae81ff">high&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">documentation&lt;/span>: &lt;span style="color:#ae81ff">medium&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">severity_levels&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">critical&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;SELECT *&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;DELETE without WHERE&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Missing partition key&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">warning&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;Inconsistent naming&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;No description&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Inefficient join&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#f92672">info&lt;/span>: [&lt;span style="color:#e6db74">&amp;#34;Consider alternative&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Minor formatting&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;Style suggestion&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span>&lt;span style="color:#f92672">ignored_patterns&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#34;legacy/deprecated/**/*.sql&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> - &lt;span style="color:#e6db74">&amp;#34;**/*_test.sql&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This allows team leaders to emphasise the aspects of code quality that matter most to their organisation.&lt;/p>
&lt;h3 id="integrating-with-data-documentation">Integrating with Data Documentation&lt;/h3>
&lt;p>Connect the AI reviewer with your data documentation tools:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">extract_documentation_context&lt;/span>(repo):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Extract context from data dictionaries and documentation&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Look for common documentation files&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">for&lt;/span> doc_path &lt;span style="color:#f92672">in&lt;/span> [&lt;span style="color:#e6db74">&amp;#34;docs/data_dictionary.md&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;definitions/tables.yml&amp;#34;&lt;/span>, &lt;span style="color:#e6db74">&amp;#34;catalog.json&amp;#34;&lt;/span>]:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> doc_file &lt;span style="color:#f92672">=&lt;/span> repo&lt;span style="color:#f92672">.&lt;/span>get_contents(doc_path)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> doc_content &lt;span style="color:#f92672">=&lt;/span> base64&lt;span style="color:#f92672">.&lt;/span>b64decode(doc_file&lt;span style="color:#f92672">.&lt;/span>content)&lt;span style="color:#f92672">.&lt;/span>decode(&lt;span style="color:#e6db74">&amp;#34;utf-8&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Process and embed documentation content&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># ...&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">continue&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>warning(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Could not process documentation: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This integration ensures the AI recommendations align with your documented data models and definitions.&lt;/p>
&lt;h3 id="adding-data-quality-test-suggestions">Adding Data Quality Test Suggestions&lt;/h3>
&lt;p>Enhance the reviewer to propose specific DBT tests:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">suggest_dbt_tests&lt;/span>(column_name, data_type, is_pk, is_fk, referenced_table&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">None&lt;/span>):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Generate appropriate DBT test suggestions based on column characteristics&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests &lt;span style="color:#f92672">=&lt;/span> []
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Always suggest not_null for primary keys&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_pk:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- not_null&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- unique&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Suggest relationship tests for foreign keys&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> is_fk &lt;span style="color:#f92672">and&lt;/span> referenced_table:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- relationships:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74"> to: ref(&amp;#39;&lt;/span>&lt;span style="color:#e6db74">{&lt;/span>referenced_table&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;)&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74"> field: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>column_name&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Type-specific tests&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;date&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> data_type&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- dbt_utils.date_in_valid_range:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74"> max_date: &amp;#39;&lt;/span>&lt;span style="color:#ae81ff">{{&lt;/span>&lt;span style="color:#e6db74"> current_date() &lt;/span>&lt;span style="color:#ae81ff">}}&lt;/span>&lt;span style="color:#e6db74">&amp;#39;&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">if&lt;/span> &lt;span style="color:#e6db74">&amp;#34;amount&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> column_name&lt;span style="color:#f92672">.&lt;/span>lower() &lt;span style="color:#f92672">or&lt;/span> &lt;span style="color:#e6db74">&amp;#34;price&amp;#34;&lt;/span> &lt;span style="color:#f92672">in&lt;/span> column_name&lt;span style="color:#f92672">.&lt;/span>lower():
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> tests&lt;span style="color:#f92672">.&lt;/span>append(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;- dbt_expectations.expect_column_values_to_be_between:&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74"> min_value: 0&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">return&lt;/span> tests
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;h3 id="creating-a-dashboard-of-code-quality-metrics">Creating a Dashboard of Code Quality Metrics&lt;/h3>
&lt;p>Build a simple tracking system for review metrics:&lt;/p>
&lt;div class="highlight">&lt;pre tabindex="0" style="color:#f8f8f2;background-color:#272822;-moz-tab-size:4;-o-tab-size:4;tab-size:4;">&lt;code class="language-python" data-lang="python">&lt;span style="display:flex;">&lt;span>&lt;span style="color:#66d9ef">def&lt;/span> &lt;span style="color:#a6e22e">track_review_metrics&lt;/span>(pr_number, repo_name, metrics):
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;&amp;#34;&amp;#34;Store review metrics for trend analysis&amp;#34;&amp;#34;&amp;#34;&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> timestamp &lt;span style="color:#f92672">=&lt;/span> datetime&lt;span style="color:#f92672">.&lt;/span>now()&lt;span style="color:#f92672">.&lt;/span>isoformat()
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> metrics_record &lt;span style="color:#f92672">=&lt;/span> {
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;timestamp&amp;#34;&lt;/span>: timestamp,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;pr_number&amp;#34;&lt;/span>: pr_number,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;repo_name&amp;#34;&lt;/span>: repo_name,
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>: metrics[&lt;span style="color:#e6db74">&amp;#34;files_analyzed&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>: metrics[&lt;span style="color:#e6db74">&amp;#34;issues_found&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;naming_consistency_issues&amp;#34;&lt;/span>: metrics[&lt;span style="color:#e6db74">&amp;#34;naming_consistency_issues&amp;#34;&lt;/span>],
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#e6db74">&amp;#34;dbt_best_practice_issues&amp;#34;&lt;/span>: metrics[&lt;span style="color:#e6db74">&amp;#34;dbt_best_practice_issues&amp;#34;&lt;/span>]
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> }
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># In a production environment, this would write to a database&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Here we&amp;#39;re just appending to a JSON file&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">try&lt;/span>:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#75715e"># Create metrics directory if it doesn&amp;#39;t exist&lt;/span>
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> os&lt;span style="color:#f92672">.&lt;/span>makedirs(&lt;span style="color:#e6db74">&amp;#34;.metrics&amp;#34;&lt;/span>, exist_ok&lt;span style="color:#f92672">=&lt;/span>&lt;span style="color:#66d9ef">True&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> 
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">with&lt;/span> open(&lt;span style="color:#e6db74">&amp;#39;.metrics/review_metrics.jsonl&amp;#39;&lt;/span>, &lt;span style="color:#e6db74">&amp;#39;a&amp;#39;&lt;/span>) &lt;span style="color:#66d9ef">as&lt;/span> f:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> f&lt;span style="color:#f92672">.&lt;/span>write(json&lt;span style="color:#f92672">.&lt;/span>dumps(metrics_record) &lt;span style="color:#f92672">+&lt;/span> &lt;span style="color:#e6db74">&amp;#34;&lt;/span>&lt;span style="color:#ae81ff">\n&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> &lt;span style="color:#66d9ef">except&lt;/span> &lt;span style="color:#a6e22e">Exception&lt;/span> &lt;span style="color:#66d9ef">as&lt;/span> e:
&lt;/span>&lt;/span>&lt;span style="display:flex;">&lt;span> logger&lt;span style="color:#f92672">.&lt;/span>error(&lt;span style="color:#e6db74">f&lt;/span>&lt;span style="color:#e6db74">&amp;#34;Error storing metrics: &lt;/span>&lt;span style="color:#e6db74">{&lt;/span>str(e)&lt;span style="color:#e6db74">}&lt;/span>&lt;span style="color:#e6db74">&amp;#34;&lt;/span>)
&lt;/span>&lt;/span>&lt;/code>&lt;/pre>&lt;/div>&lt;p>This data can be visualised in a simple dashboard to track code quality trends over time.&lt;/p>
&lt;/br>
&lt;/br>
&lt;hr>
&lt;h2 id="conclusion-and-next-steps">Conclusion and Next Steps&lt;/h2>
&lt;/br>
&lt;p>In this article, we&amp;rsquo;ve taken the theoretical foundations of LLMs, RAG, and vector databases from Part 1 and implemented them in a practical, high-value system that can transform your data engineering workflow. By creating an AI Data Engineer for code reviews, we&amp;rsquo;re addressing the common challenges of maintaining code quality, consistency, and best practices across data engineering teams.&lt;/p>
&lt;p>The power of this approach comes from combining:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Contextual Intelligence&lt;/strong>: The AI understands your specific codebase, not just generic standards&lt;/li>
&lt;li>&lt;strong>Semantic Understanding&lt;/strong>: Vector embeddings enable the system to find similar patterns even when exact matches don&amp;rsquo;t exist&lt;/li>
&lt;li>&lt;strong>RAG Architecture&lt;/strong>: The LLM&amp;rsquo;s reasoning capabilities are grounded in your organisation&amp;rsquo;s actual practices&lt;/li>
&lt;li>&lt;strong>GitHub Integration&lt;/strong>: The system fits seamlessly into existing development workflows&lt;/li>
&lt;/ol>
&lt;h3 id="getting-started">Getting Started&lt;/h3>
&lt;p>To implement this in your organisation:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Start Small&lt;/strong>: Begin with a single repository and a focused set of standards&lt;/li>
&lt;li>&lt;strong>Collect Feedback&lt;/strong>: Ask your team which aspects of the reviews are most helpful&lt;/li>
&lt;li>&lt;strong>Iterate&lt;/strong>: Refine the prompts and vector database to improve relevance&lt;/li>
&lt;li>&lt;strong>Measure Impact&lt;/strong>: Track metrics like PR cycle time and bug reduction&lt;/li>
&lt;/ol>
&lt;h3 id="where-to-go-next">Where to Go Next&lt;/h3>
&lt;p>As you become comfortable with the base implementation, consider these extensions:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Multi-Agent Reviews&lt;/strong>: Add specialised agents for security, performance, and business logic&lt;/li>
&lt;li>&lt;strong>Interactive Suggestions&lt;/strong>: Enable the AI to propose specific fixes that can be applied with one click&lt;/li>
&lt;li>&lt;strong>Natural Language Interfaces&lt;/strong>: Allow engineers to ask the AI questions about data models and transformations&lt;/li>
&lt;li>&lt;strong>Knowledge Management&lt;/strong>: Use the vector database as a foundation for a broader data engineering knowledge system&lt;/li>
&lt;/ol>
&lt;p>The possibilities are limited only by your imagination. The core technology stack we&amp;rsquo;ve implemented—LLMs with RAG capabilities and vector databases—can be applied to numerous data engineering challenges beyond code reviews.&lt;/p>
&lt;p>By combining these advanced AI capabilities with your team&amp;rsquo;s domain expertise, you can create a powerful system that accelerates development, improves quality, and makes your data engineering practice more robust and consistent.&lt;/p>
&lt;p>Remember: The goal isn&amp;rsquo;t to replace human judgment but to augment it—freeing your team to focus on the creative and strategic aspects of data engineering while the AI handles the routine checks and reinforces your best practices.&lt;/p></content:encoded><category>AI</category><category>Data Engineering</category><category>GitHub Actions</category><category>CI/CD</category><category>AI Agents</category><category>Code Review</category><category>Data Quality</category><category>dbt</category><category>SQL Standards</category></item><item><title>Leveraging LLMs for Business Impact: Part 1 - Theory and Foundations</title><link>https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/</link><pubDate>Fri, 07 Mar 2025 09:00:00 +1100</pubDate><guid>https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/</guid><author>Chris Hillman</author><description>A comprehensive exploration of Large Language Models, Retrieval Augmented Generation, and Vector Databases - the foundational technologies driving AI business transformation.</description><content:encoded>&lt;h2 id="introduction">Introduction&lt;/h2>
&lt;p>In today&amp;rsquo;s rapidly evolving technological landscape, Large Language Models (LLMs) have emerged as transformative tools with the potential to revolutionize business operations across industries. While the hype around these technologies is intense, understanding their practical applications and underlying mechanisms is crucial for organisations seeking to leverage them effectively.&lt;/p>
&lt;p>This two-part series aims to demystify LLMs and their associated technologies, starting with the theoretical foundations in Part 1, followed by a hands-on implementation guide using AWS services in Part 2.&lt;/p>
&lt;/br> 
&lt;/br> 
&lt;hr>
&lt;h2 id="understanding-large-language-models-llms">Understanding Large Language Models (LLMs)&lt;/h2>
&lt;/br> 
&lt;h3 id="what-are-llms">What Are LLMs?&lt;/h3>
&lt;p>Large Language Models are sophisticated AI systems trained on vast amounts of text data to understand and generate human-like language. Unlike traditional rule-based systems, LLMs learn patterns and relationships within language, enabling them to perform a wide range of tasks - from answering questions and summarising content to translating languages and generating creative text.&lt;/p>
&lt;p>Modern LLMs like GPT-4, Claude, and Llama 2 can reason, infer, and adapt to different contexts with remarkable flexibility. These models have billions or even trillions of parameters (adjustable values that the model learns during training), allowing them to capture nuanced linguistic patterns and knowledge.&lt;/p>
&lt;/br> 
&lt;h3 id="llm-capabilities-and-limitations">LLM Capabilities and Limitations&lt;/h3>
&lt;p>LLMs excel at:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Language understanding and generation&lt;/strong>: Comprehending complex queries and producing coherent, contextually appropriate responses&lt;/li>
&lt;li>&lt;strong>Knowledge retrieval&lt;/strong>: Accessing information they&amp;rsquo;ve been trained on&lt;/li>
&lt;li>&lt;strong>Task adaptation&lt;/strong>: Applying language skills to various domains with minimal explicit instruction&lt;/li>
&lt;li>&lt;strong>Pattern recognition&lt;/strong>: Identifying trends and relationships within text&lt;/li>
&lt;/ul>
&lt;p>However, they face key limitations:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Hallucination&lt;/strong>: Sometimes generating plausible-sounding but factually incorrect information&lt;/li>
&lt;li>&lt;strong>Knowledge cutoff&lt;/strong>: Limited to information available up to their training cutoff date&lt;/li>
&lt;li>&lt;strong>Context window constraints&lt;/strong>: Only able to &amp;ldquo;see&amp;rdquo; a finite amount of text at once&lt;/li>
&lt;li>&lt;strong>Reasoning limitations&lt;/strong>: Struggling with complex logical or mathematical reasoning&lt;/li>
&lt;li>&lt;strong>Lack of domain-specific knowledge&lt;/strong>: Generic models often lack deep expertise in specialised fields&lt;/li>
&lt;/ul>
&lt;p>These limitations are particularly relevant in business contexts where accuracy and reliability are paramount. This is where Retrieval Augmented Generation (RAG) and vector databases come in.&lt;/p>
&lt;hr>
&lt;/br> 
&lt;/br> 
&lt;h2 id="retrieval-augmented-generation-rag-enhancing-llms-with-external-knowledge">Retrieval Augmented Generation (RAG): Enhancing LLMs with External Knowledge&lt;/h2>
&lt;/br> 
&lt;h3 id="the-rag-framework">The RAG Framework&lt;/h3>
&lt;p>RAG represents a paradigm shift in how we interact with LLMs by combining retrieval systems with generative AI. At its core, RAG works by:&lt;/p>
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/images/rag.png" alt="RAG daframeworktabase">&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Retrieving&lt;/strong> relevant information from a knowledge base&lt;/li>
&lt;li>&lt;strong>Augmenting&lt;/strong> the LLM&amp;rsquo;s prompt with this contextual information&lt;/li>
&lt;li>&lt;strong>Generating&lt;/strong> responses informed by both the model&amp;rsquo;s parametric knowledge and the retrieved information&lt;/li>
&lt;/ol>
&lt;p>This approach addresses several key LLM limitations:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Improves factual accuracy&lt;/strong> by grounding responses in verified information&lt;/li>
&lt;li>&lt;strong>Extends knowledge beyond training cutoff&lt;/strong> by incorporating up-to-date information&lt;/li>
&lt;li>&lt;strong>Enables domain-specific expertise&lt;/strong> through specialised knowledge bases&lt;/li>
&lt;li>&lt;strong>Creates auditability&lt;/strong> by tracking sources of information&lt;/li>
&lt;/ul>
&lt;p>LLMs alone might lack knowledge of specific company data architectures, but RAG can bridge this gap. By integrating company documentation, data dictionaries, and past incident reports into a vector database, engineers can query a system that retrieves relevant knowledge before generating responses.&lt;/p>
&lt;p>Example:
A data engineer is troubleshooting a failing data pipeline in an AWS Glue job. Instead of searching through Confluence pages or old Slack threads, they ask an LLM-powered assistant:
&amp;ldquo;Why is my Glue job failing with an &amp;lsquo;out of memory&amp;rsquo; error?&amp;rdquo;&lt;/p>
&lt;h4 id="the-rag-powered-assistant">The RAG-powered assistant:&lt;/h4>
&lt;p>Retrieves past incident reports where engineers faced similar issues.
Finds a company-specific best practices document on Glue memory tuning.
Generates a recommendation based on both the retrieved knowledge and general LLM expertise.
This saves hours of manual searching and accelerates issue resolution.&lt;/p>
&lt;/br> 
&lt;h3 id="the-rag-pipeline">The RAG Pipeline&lt;/h3>
&lt;p>A typical RAG implementation involves several key steps:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Document ingestion&lt;/strong>: Processing and chunking documents into manageable pieces&lt;/li>
&lt;li>&lt;strong>Embedding generation&lt;/strong>: Converting text chunks into vector embeddings&lt;/li>
&lt;li>&lt;strong>Vector storage&lt;/strong>: Storing these embeddings in a vector database&lt;/li>
&lt;li>&lt;strong>Query processing&lt;/strong>: Converting user queries into the same vector space&lt;/li>
&lt;li>&lt;strong>Similarity search&lt;/strong>: Finding the most relevant documents based on vector similarity&lt;/li>
&lt;li>&lt;strong>Context augmentation&lt;/strong>: Adding retrieved information to the LLM prompt&lt;/li>
&lt;li>&lt;strong>Response generation&lt;/strong>: Using the enhanced prompt to generate an informed response&lt;/li>
&lt;/ol>
&lt;p>This architecture provides businesses with the flexibility to incorporate proprietary knowledge, specialised datasets, and real-time information into their AI systems.&lt;/p>
&lt;hr>
&lt;/br> 
&lt;/br> 
&lt;h2 id="vector-databases-the-engine-behind-semantic-search">Vector Databases: The Engine Behind Semantic Search&lt;/h2>
&lt;/br> 
&lt;h3 id="what-are-vector-databases">What Are Vector Databases?&lt;/h3>
&lt;p>Vector databases are specialised storage systems designed to efficiently store, index, and query high-dimensional vectors (mathematical representations of data). Unlike traditional databases that excel at exact matches, vector databases are optimised for similarity searches - finding items that are conceptually similar rather than identical.&lt;/p>
&lt;p>In the context of LLMs and RAG, vector databases store embeddings - numerical representations of text that capture semantic meaning. These embeddings allow us to find information based on conceptual similarity rather than keyword matching.&lt;/p>
&lt;/br> 
&lt;h3 id="key-components-and-characteristics">Key Components and Characteristics&lt;/h3>
&lt;p>Vector databases have several distinctive features:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Vector embeddings&lt;/strong>: Numerical representations (typically 768-4096 dimensions) that encode semantic meaning&lt;/li>
&lt;li>&lt;strong>Similarity metrics&lt;/strong>: Methods for measuring the &amp;ldquo;closeness&amp;rdquo; of vectors (cosine similarity, Euclidean distance)&lt;/li>
&lt;li>&lt;strong>Indexing structures&lt;/strong>: Specialised algorithms (HNSW, IVF, etc.) that enable efficient similarity search&lt;/li>
&lt;li>&lt;strong>Metadata storage&lt;/strong>: Additional information about each vector for filtering and retrieval&lt;/li>
&lt;li>&lt;strong>Scalability features&lt;/strong>: Capabilities for handling billions of vectors across distributed systems&lt;/li>
&lt;/ul>
&lt;p>Popular vector database implementations include:&lt;/p>
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/images/vector.png" alt="Vector database">&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Pinecone&lt;/strong>: Fully managed vector database service&lt;/li>
&lt;li>&lt;strong>Weaviate&lt;/strong>: Open-source vector search engine&lt;/li>
&lt;li>&lt;strong>Chroma&lt;/strong>: Lightweight embedding database for RAG applications&lt;/li>
&lt;li>&lt;strong>Milvus&lt;/strong>: Open-source vector database built for enterprise-scale deployments&lt;/li>
&lt;li>&lt;strong>FAISS (Facebook AI Similarity Search)&lt;/strong>: Library for efficient similarity search&lt;/li>
&lt;li>&lt;strong>Amazon OpenSearch&lt;/strong>: Service with vector search capabilities&lt;/li>
&lt;/ul>
&lt;/br> 
&lt;h3 id="how-vector-databases-work">How Vector Databases Work&lt;/h3>
&lt;p>At a fundamental level, vector databases solve the nearest neighbour search problem - given a query vector, find the most similar vectors in the database. However, as the number of dimensions and vectors grows, this becomes computationally expensive.&lt;/p>
&lt;p>Vector databases employ approximate nearest neighbour (ANN) algorithms that trade perfect accuracy for dramatic performance improvements. These approaches include:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Hierarchical Navigable Small World (HNSW)&lt;/strong>: Creates a multi-layered graph structure for efficient navigation&lt;/li>
&lt;li>&lt;strong>Inverted File Index (IVF)&lt;/strong>: Partitions the vector space into clusters for faster search&lt;/li>
&lt;li>&lt;strong>Product Quantization (PQ)&lt;/strong>: Compresses vectors to reduce memory usage while preserving similarity characteristics&lt;/li>
&lt;/ol>
&lt;p>These technical optimisations enable systems to search millions or billions of vectors in milliseconds, making real-time RAG applications possible.&lt;/p>
&lt;hr>
&lt;/br> 
&lt;/br> 
&lt;h2 id="the-synergy-how-these-technologies-work-together">The Synergy: How These Technologies Work Together&lt;/h2>
&lt;p>The real power emerges when LLMs, RAG, and vector databases operate as an integrated system:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Knowledge Base Creation&lt;/strong>:
&lt;ul>
&lt;li>Documents are processed, chunked, and converted to embeddings&lt;/li>
&lt;li>These embeddings, along with metadata, are stored in a vector database&lt;/li>
&lt;li>The system builds efficient indexes for fast retrieval
&lt;/br>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;strong>Query Processing&lt;/strong>:
&lt;ul>
&lt;li>A user submits a query&lt;/li>
&lt;li>The query is embedded into the same vector space&lt;/li>
&lt;li>The vector database finds semantically similar content&lt;/li>
&lt;li>Relevant information is retrieved and formatted
&lt;/br>&lt;/li>
&lt;/ul>
&lt;/li>
&lt;li>&lt;strong>Enhanced Response Generation&lt;/strong>:
&lt;ul>
&lt;li>The LLM receives both the original query and retrieved information&lt;/li>
&lt;li>The model generates a response that incorporates this external knowledge&lt;/li>
&lt;li>The system can cite sources, explain reasoning, and provide verifiable information&lt;/li>
&lt;/ul>
&lt;/li>
&lt;/ol>
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/images/ai.png" alt="AI database">&lt;/p>
&lt;p>This architecture represents a significant advancement over standalone LLMs, enabling businesses to leverage their proprietary knowledge while benefiting from the reasoning capabilities of foundation models.&lt;/p>
&lt;hr>
&lt;/br> 
&lt;/br> 
&lt;h2 id="business-implications-and-applications">Business Implications and Applications&lt;/h2>
&lt;/br> 
&lt;h3 id="transform-business-processes">Transform Business Processes&lt;/h3>
&lt;p>LLMs with RAG can transform numerous business functions:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Customer Support&lt;/strong>: Creating AI assistants that can access product documentation, support tickets, and knowledge bases to resolve customer issues&lt;/li>
&lt;li>&lt;strong>Content Creation&lt;/strong>: Generating marketing materials, reports, and documentation informed by brand guidelines and product details&lt;/li>
&lt;li>&lt;strong>Legal and Compliance&lt;/strong>: Answering questions based on regulatory documents, contracts, and policies&lt;/li>
&lt;li>&lt;strong>Research and Development&lt;/strong>: Summarising research papers and identifying patterns across technical literature&lt;/li>
&lt;li>&lt;strong>Knowledge Management&lt;/strong>: Making organisational knowledge accessible and actionable&lt;/li>
&lt;/ul>
&lt;/br> 
&lt;h3 id="training-and-upskilling-new-data-engineers">Training and Upskilling New Data Engineers&lt;/h3>
&lt;p>LLMs can be used as interactive mentors for new data engineers by providing:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Code Walkthroughs&lt;/strong>: Junior engineers can input a piece of SQL, Python, or Spark code, and an LLM can break it down, explaining best practices, inefficiencies, or potential pitfalls.&lt;/li>
&lt;li>&lt;strong>Data Pipeline Debugging&lt;/strong>: An LLM-based assistant can suggest fixes when encountering issues in ETL processes.&lt;/li>
&lt;li>&lt;strong>Documentation Generation&lt;/strong>: Automatically generate documentation for a given data pipeline based on code and metadata, reducing the manual effort required.&lt;/li>
&lt;/ul>
&lt;p>Example:
A new data engineer at a financial services company is tasked with optimising a slow SQL query running against a large Postgres database. They submit their query to an LLM-powered assistant, which:&lt;/p>
&lt;p>Identifies missing indexes and suggests improvements.
Recognises inefficient JOIN operations and proposes alternatives.
Provides a step-by-step execution plan analysis, teaching the engineer how to optimise queries themselves in the future.&lt;/p>
&lt;/br> 
&lt;h3 id="peer-reviewing-initial-code-submissions">Peer Reviewing Initial Code Submissions&lt;/h3>
&lt;p>LLMs can assist in the code review process by:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Checking for errors&lt;/strong>: Identifying syntax issues, anti-patterns, or security risks in Python, SQL, or Scala.&lt;/li>
&lt;li>&lt;strong>Ensuring adherence to style guides&lt;/strong>: Comparing code against company-specific best practices and formatting guidelines.&lt;/li>
&lt;li>&lt;strong>Providing alternative solutions&lt;/strong>: Suggesting more efficient ways to write transformation logic or queries.&lt;/li>
&lt;/ul>
&lt;p>Example:
A data engineering team at an e-commerce company uses an LLM-integrated CI/CD tool that automatically reviews every pull request. When a junior engineer submits a PySpark script for processing customer orders, the LLM:&lt;/p>
&lt;p>Flags an inefficient groupBy() operation that causes excessive shuffling.
Suggests an alternative reduceByKey() approach for better performance.
Explains why the change is beneficial, reinforcing best practices.&lt;/p>
&lt;/br> 
&lt;h3 id="multi-agent-systems-for-enhanced-reliability">Multi-Agent Systems for Enhanced Reliability&lt;/h3>
&lt;p>One of the most promising approaches to mitigating hallucinations and improving LLM reliability is the implementation of multi-agent systems. These systems leverage multiple LLM instances working together to verify outputs and catch errors.&lt;/p>
&lt;h4 id="how-multi-agent-systems-work">How Multi-Agent Systems Work&lt;/h4>
&lt;p>Multi-agent systems typically involve several specialised roles:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Primary Agent&lt;/strong>: Generates initial responses based on user queries and retrieved context&lt;/li>
&lt;li>&lt;strong>Critic Agent&lt;/strong>: Reviews the primary agent&amp;rsquo;s output for factual accuracy, hallucinations, and logical inconsistencies&lt;/li>
&lt;li>&lt;strong>Research Agent&lt;/strong>: Actively searches for additional relevant information when knowledge gaps are identified&lt;/li>
&lt;li>&lt;strong>Consensus Agent&lt;/strong>: Reconciles potentially conflicting information from multiple sources&lt;/li>
&lt;li>&lt;strong>Explainer Agent&lt;/strong>: Provides reasoning transparency and confidence levels for final outputs&lt;/li>
&lt;/ol>
&lt;p>&lt;img src="https://ghostinthedata.info/posts/2025/2025-03-07-leveraging-llm-for-business-impact-part-1/images/multiagent.png" alt="Multi Agent">&lt;/p>
&lt;h4 id="hallucination-mitigation-through-peer-review">Hallucination Mitigation Through Peer Review&lt;/h4>
&lt;p>Multi-agent systems reduce hallucinations through several mechanisms:&lt;/p>
&lt;ol>
&lt;li>&lt;strong>Distributed Verification&lt;/strong>: Each agent independently verifies information, creating a system of checks and balances&lt;/li>
&lt;li>&lt;strong>Specialised Expertise&lt;/strong>: Agents can be fine-tuned for specific domains or tasks, improving accuracy in their areas of focus&lt;/li>
&lt;li>&lt;strong>Explicit Reasoning Chains&lt;/strong>: Agents are required to explain their reasoning process, making erroneous logic easier to detect&lt;/li>
&lt;li>&lt;strong>Confidence Scoring&lt;/strong>: Systems can implement confidence metrics, flagging uncertain statements for human review&lt;/li>
&lt;li>&lt;strong>Adversarial Testing&lt;/strong>: Specifically designed agents can challenge assertions made by other agents, testing their validity&lt;/li>
&lt;/ol>
&lt;/br> 
&lt;h3 id="implementation-considerations">Implementation Considerations&lt;/h3>
&lt;p>Organisations considering RAG implementations should evaluate:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Data suitability&lt;/strong>: Whether existing documents are structured appropriately for RAG&lt;/li>
&lt;li>&lt;strong>Embedding strategy&lt;/strong>: Which embedding models and chunking approaches to use&lt;/li>
&lt;li>&lt;strong>Security and privacy&lt;/strong>: How to handle sensitive information&lt;/li>
&lt;li>&lt;strong>Infrastructure requirements&lt;/strong>: Computing resources needed for embeddings and inference&lt;/li>
&lt;li>&lt;strong>Evaluation frameworks&lt;/strong>: Methods for measuring system accuracy and effectiveness&lt;/li>
&lt;/ul>
&lt;/br> 
&lt;h3 id="the-business-impact-opportunity">The Business Impact Opportunity&lt;/h3>
&lt;p>When properly implemented, these systems can deliver significant business value:&lt;/p>
&lt;ul>
&lt;li>&lt;strong>Reduced operational costs&lt;/strong> through automation of information-intensive tasks&lt;/li>
&lt;li>&lt;strong>Enhanced decision-making&lt;/strong> by making relevant information readily accessible&lt;/li>
&lt;li>&lt;strong>Improved customer experiences&lt;/strong> through more accurate and helpful AI interactions&lt;/li>
&lt;li>&lt;strong>Accelerated innovation&lt;/strong> by helping employees leverage institutional knowledge&lt;/li>
&lt;li>&lt;strong>Competitive differentiation&lt;/strong> through personalized and contextually aware services&lt;/li>
&lt;/ul>
&lt;hr>
&lt;/br> 
&lt;/br> 
&lt;h2 id="conclusion-preparing-for-implementation">Conclusion: Preparing for Implementation&lt;/h2>
&lt;p>The theoretical understanding of LLMs, RAG, and vector databases provides the foundation for practical implementation. Organisations that grasp these concepts can make informed decisions about how to leverage these technologies effectively.&lt;/p>
&lt;p>As with any transformative technology, the organisations that will benefit most are those that approach implementation thoughtfully, with clear business objectives and a solid understanding of the underlying mechanisms. By combining the generative capabilities of LLMs with the factual grounding of RAG and the efficient retrieval of vector databases, businesses can create AI systems that are not just impressive demonstrations but valuable tools that drive tangible business impact.&lt;/p></content:encoded><category>AI</category><category>Data Engineering</category><category>LLM</category><category>RAG</category><category>Vector Databases</category><category>AI Business Applications</category><category>Data Architecture</category></item></channel></rss>