AWS

AWS MWAA: The Practitioner's Unvarnished Guide

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built. But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head. The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

Saturday, July 18, 2026 Read

Your Data Platform Costs More Than It Should

Let me tell you about the moment I stopped treating cloud costs as someone else’s problem. We were three months into a Snowflake migration. Everything was humming. Pipelines were green, dashboards were fast, the analytics team was happier than I’d seen them before. I felt good about the work we’d done. Then finance forwarded me the invoice. The number wasn’t catastrophic. But it was significantly higher than what we’d budgeted, and when I started digging, I couldn’t explain where most of it was going. I knew we had warehouses running. I knew we had pipelines executing. But I couldn’t tell you which warehouse was responsible for what cost, which pipelines were the expensive ones, or whether the money was well spent. I had built a platform I was proud of — and I had no idea what it actually cost to operate.

Saturday, April 25, 2026 Read

Why Your Pipeline Finishes Later Every Month

Let me tell you about a graph that changed how I think about data engineering. A junior engineer on my team — let’s call her Priya — had been tracking something nobody asked her to track. Every morning for two months, she’d noted the timestamp when our main analytics pipeline completed. She wasn’t trying to make a point. She was just curious, because the finance team kept mentioning their dashboards weren’t ready when they arrived at 8 AM anymore.

Saturday, April 18, 2026 Read

12 Steps to Better Data Engineering

Let me tell you about the moment I stopped trusting architecture diagrams. I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you’d see in a data conference. Then I asked a question that changed everything: “Can you rebuild your finance table from scratch right now?”

Saturday, March 7, 2026 Read

Building Your First AWS Data Pipeline: A Guide for Data Professionals Who've Never Touched Cloud Infrastructure

The spreadsheet that changed everything Here’s a story that might sound familiar. You’re pulling data from an API—maybe daily sales numbers, maybe customer interactions, maybe something else entirely. Every morning, you open your laptop, run a Python script, save the CSV somewhere, and get on with your actual work. It takes maybe five minutes, but it’s five minutes you can’t forget about. Miss a day and you’ve got a gap in your data. Go on vacation? Better hope someone remembers to run your script.

Wednesday, November 26, 2025 Read