2026

The Style Isn't the Skill

I was a kid when I first watched Bruce Lee, soon after and my friend signed up for kung fu lessons. What hooked me wasn’t the fighting. It was the speed at which Bruce moved. The more you watched, though, the more you noticed something sitting behind the speed. He was obviously smart, fast, cool, and clearly thinking several moves ahead of everyone else in the room. Even as a kid you could tell the fighting was the visible bit of something much bigger.

Monday, August 31, 2026 Read

AWS MWAA: The Practitioner's Unvarnished Guide

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built. But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head. The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

Saturday, July 18, 2026 Read

Data Engineers: Just Do Code Reviews

In Code Complete, Steve McConnell writes: Software testing alone has limited effectiveness — the average defect detection rate is only 25 percent for unit testing, 35 percent for function testing, and 45 percent for integration testing. In contrast, the average effectiveness of design and code inspections are 55 and 60 percent. That was published in 1993. The evidence has only accumulated since. Peer code review is the single most effective defect-removal technique available to software teams. Nothing else — not tests, not CI pipelines, not observability tooling — comes close on a per-hour basis.

Saturday, July 11, 2026 Read

You Can't Incentivise a Pipeline That Doesn't Break

I worked alongside a data engineer who was, by every formal measure, the best performer on the team. He was also quietly destroying the platform. Not maliciously. He was optimising for the thing being measured. His work shipped fast because he skipped the edge case analysis. He closed tickets at first resolution without ever checking whether the underlying pattern would recur. He didn’t review anyone else’s PRs (not his KPIs, so why would he?).

Saturday, July 4, 2026 Read

Your Team Already Has Patterns. They Just Don't Know It.

When I started a new role, one of the first things I did was try to understand how data moved through the system. Not the dashboards, not the data models — the pipes. Where did things come from? How did they get in? What happened to them along the way? There were somewhere between twenty and thirty source systems feeding the platform. Not a massive number, but enough to tell a story when you looked at the ingestion layer all at once. What I found was that all the pipelines had originated from two base templates. A sensible starting point. The kind of thing a small team puts in place early to stop complete chaos.

Saturday, June 27, 2026 Read

Keep Moving

Some days I open my laptop and by 5pm I genuinely cannot tell you what I did. Not because it was complicated. Not because there were emergencies. The stand-up happened. A few Teams messages were sent. A ticket was groomed. A document was “reviewed”. A meeting was attended where everyone agreed something was important and then the meeting ended and nothing changed. And then somehow it was evening and the pipeline I meant to fix was exactly as broken as it was in the morning.

Tuesday, June 23, 2026 Read

Ghost Skills: Teaching AI Agents to Think Like Data Engineers

Another week, another skills repo on the GitHub trending page. I know. There are roughly seventeen of them now, all promising to turn your AI coding agent from a confident intern into a slightly-less-confident intern. Most of them are great. Most of them are also built by solo devs, for solo devs, on solo-dev codebases that fit comfortably in a context window. Which is fine, if that’s your world. Less fine if your world involves a Snowflake warehouse with four tables that could be the source of truth for “customer”, an SCD2 someone half-built in 2021 and quietly walked away from, and a dbt project where stg_users_final_v3_actually_use_this is, somehow, the one you’re meant to use. (Don’t laugh. You’ve seen worse.)

Sunday, June 14, 2026 Read

The Competitive Moat That AI Can't Replicate

The Restaurant That Refused to Take Bookings Online Let me tell you a story about a restaurant owner who became obsessed with human connection. He didn’t want people booking online. He wanted them to call. He wanted the ritual of a human voice, the small exchange about an anniversary or a first date, the warmth of being recognised. His team thought he was losing his mind. Online bookings were standard. Everyone did it. Why make customers work harder?

Saturday, June 13, 2026 Read

SQL Tells You What. Comments Tell You Why.

The best SQL doesn’t need comments. Write meaningful CTE names, descriptive aliases, clear column labels — and a skilled reader will follow your logic without a single annotation. That’s the right instinct. It’s also only half right. SQL is a declarative language. You’re not writing how the database retrieves your data; you’re writing what you want. That’s a useful distinction, because “what” and “why” are very different questions, and SQL can answer exactly one of them.

Saturday, June 6, 2026 Read

Don't Go Dark: Visibility Is a Data Engineering Skill

There’s a specific kind of silence in data engineering that I’ve learned to fear. Not the silence of a system that’s working well. Not the comfortable quiet of a team in flow. I mean the silence of a project that’s been running for three weeks and you still can’t point to a single visible thing it has produced. The kind of silence where, if your manager stopped you in the hallway and asked “how’s that migration going?”, you’d say “fine” because saying anything more accurate would require explaining things you haven’t fully articulated yet — even to yourself.

Saturday, May 23, 2026 Read

The Broken Window in Your Data Pipeline

There’s a particular kind of data problem that doesn’t announce itself. It accumulates. We were receiving Salesforce data through delta extraction — sensible in theory, because full snapshots can run to hundreds of terabytes and less than 1% of records change on any given day. The problem is that deltas require someone to know what “changed” means. In Salesforce, that’s less obvious than it sounds. Watch a last_modified column and you’ll miss objects that get updated when a related object changes, without their own timestamp reflecting it.

Saturday, May 9, 2026 Read

Five Worlds of Data Engineering

You watch a conference talk about implementing data contracts, and nobody mentions that the advice assumes you have multiple teams producing data — which you don’t. You read a post declaring “if you’re still using stored procedures in 2026, you’re doing it wrong,” and the comments erupt. Half the people are nodding along. Half are furious. Both sides are right. They’re just living in different worlds and don’t realise it.

Saturday, May 2, 2026 Read