Ghost in the data
  • Home
  • About
  • Posts
  • Start Here
  • Topics
  • Resources
  • RSS
  • Categories
  • AI
  • Career
  • Data Engineering
  • Data Modelling
  • Data Quality & Governance
  • Leadership
  • Life
  • Platforms & Cloud
Hero Image
Very Boring but Very Responsible

In 1983 Lisanne Bainbridge wrote a short paper called Ironies of Automation. The irony she cared most about was that the more capable the automation, the less practice the human operator gets, and the more important it becomes that the human can take over when the automation fails. A job that’s mostly monitoring, she wrote, is “very boring but very responsible”. She was writing about chemical plants and cockpits. Annie Vella has spent this year arguing that she was also writing about us.

  • AI Agents
  • Engineering Leadership
  • Code Review
  • Team Culture
  • Claude Code
  • dbt
  • Supervisory Control
Sunday, November 15, 2026 Read
Hero Image
World Building: Your World Still Needs a Why

On the 17th of October, people sat down in the Great Hall of Stirling Castle to play Call of Cthulhu. I was in Scotland at the time. I didn’t make it. That still annoys me. What caught my attention wasn’t the castle (well fine, it kind of was the castle). It was what sat underneath the show. The game came out of A Viking in the Sun, a University of Edinburgh research project on Harald Hardrada, and its sister show in Istanbul earlier this year was built from real archaeological objects, historical sources and Icelandic sagas. Historians did months of careful work so that four players could walk into a room and go wherever they liked.

  • Obsidian
  • Knowledge Management
  • Vector Search
  • Claude
  • Tabletop RPG
  • Data Modelling
  • Semantic Layer
Friday, October 30, 2026 Read
Hero Image
Ghost Skills: Teaching AI Agents to Think Like Data Engineers

Another week, another skills repo on the GitHub trending page. I know. There are roughly seventeen of them now, all promising to turn your AI coding agent from a confident intern into a slightly-less-confident intern. Most of them are great. Most of them are also built by solo devs, for solo devs, on solo-dev codebases that fit comfortably in a context window. Which is fine, if that’s your world. Less fine if your world involves a Snowflake warehouse with four tables that could be the source of truth for “customer”, an SCD2 someone half-built in 2021 and quietly walked away from, and a dbt project where stg_users_final_v3_actually_use_this is, somehow, the one you’re meant to use. (Don’t laugh. You’ve seen worse.)

  • AI Agents
  • Claude Code
  • dbt
  • Data Modelling
  • Dimensional Modelling
  • Data Quality
  • Open Source
  • Snowflake
Sunday, June 14, 2026 Read
Hero Image
When AI Stops Treating Us Like People: The Twin Crises Nobody Wants to Discuss

The moment it stops being theoretical A friend was telling me about their recent job interviews. Not the questions they struggled with or the technical challenge they bombed—that’s normal interview anxiety. No, they were frustrated because they’d spent forty-five minutes talking to an AI chatbot that analysed their “enthusiasm for remote work” and tracked their eye movements while asking whether they’d “misrepresent themselves 3.8x more than average candidates.” The speed with which I would exit an AI interview would rival even the most seasoned teenager’s alt-tab skills.

  • AI Bubble
  • Financial Crisis
  • Trust
  • Data Teams
  • AI Ethics
  • Career Development
Friday, December 12, 2025 Read
Hero Image
Building AI Agents with Claude Code

Introduction Imagine you’re reviewing a pull request with dozens of SQL files, each containing complex queries for your data pipeline. You spot inconsistent formatting, or syntax which doesn’t work with your infrastructure. Sound familiar? It’s common for data professionals to struggle with maintaining consistent SQL standards across their projects, especially when working with specialised platforms and it can be time consuming to review these elements within a peer review. It would be better use of time to focus on the hard thinking elements, like logic etc. However these small syntax or style issues, can be distracting. Well at least they are for me.

  • Claude Code
  • AI Agents
  • Starburst
  • Delta Lake
  • Trino
  • Data Validation
  • dbt
  • Data Engineering
  • AI Tools
  • VSCode
Saturday, September 13, 2025 Read
Hero Image
AI Simplified: Understanding LLMs, Workflows, and Agents

AI Buzzwords Demystified If you’ve been following AI developments lately, you’ve probably encountered terms like LLMs, RAG, ReAct, and AI Agents. While these technologies are transforming how we interact with AI, the terminology can be overwhelming. In this post, I’ll break down these concepts into digestible explanations with practical examples. Let’s start with the foundation and progressively build up to more complex systems. Large Language Models (LLMs): The Foundation At the core of today’s AI revolution are Large Language Models (LLMs). Popular applications such as ChatGPT and Claude are built on top of these powerful models. They excel at generating and manipulating text based on the prompts we provide.

  • AI Concepts
  • ChatGPT
  • Claude
  • LLM
  • AI Workflows
  • RAG
  • AI Agents
Saturday, May 24, 2025 Read
Hero Image
The One Simple Secret to Effective AI Prompting

Forget Everything You’ve Learned About AI Prompting There’s a sea of articles out there about “how to talk to AI” or “the perfect prompt structure.” Frameworks, formulas, special keywords—it can get overwhelming. But what if I told you that you could forget all of it? That’s right. All those complicated prompting techniques might be unnecessary, because there’s one fundamental principle that works better than anything else: AI excels at roleplaying.

  • AI Prompting
  • ChatGPT
  • Claude
  • LLM
  • AI Communication
  • Roleplaying
  • AI Productivity
Sunday, May 11, 2025 Read
Hero Image
Leveraging LLMs for Business Impact: Part 2 - Building an AI Data Engineer Agent

Introduction In Part 1 of this series, we explored the theoretical foundations of Large Language Models (LLMs), Retrieval Augmented Generation (RAG), and vector databases. Now, it’s time to put theory into practice. This is going to be a long read, so grab some coffee, and one (couple) of your favourite biscuits. One use case for leveraging LLM’s, is creating of a Agent - a Senior Data Engineer AI that automatically reviews Pull Requests in your data engineering projects. This agent will be that nit picky Data Engineer that enforces SQL formatting standards, ensure naming and data type consistency, validate data quality checks, and suggest improvements based on best practices. By integrating this into your GitHub workflow, you can maintain higher code quality, accelerate onboarding for new team members, and reduce the burden of manual code reviews.

  • GitHub Actions
  • CI/CD
  • AI Agents
  • Code Review
  • Data Quality
  • dbt
  • SQL Standards
Saturday, March 8, 2025 Read
Hero Image
Leveraging LLMs for Business Impact: Part 1 - Theory and Foundations

Introduction In today’s rapidly evolving technological landscape, Large Language Models (LLMs) have emerged as transformative tools with the potential to revolutionize business operations across industries. While the hype around these technologies is intense, understanding their practical applications and underlying mechanisms is crucial for organisations seeking to leverage them effectively. This two-part series aims to demystify LLMs and their associated technologies, starting with the theoretical foundations in Part 1, followed by a hands-on implementation guide using AWS services in Part 2.

  • LLM
  • RAG
  • Vector Databases
  • AI Business Applications
  • Data Architecture
Friday, March 7, 2025 Read