Ghost in the data
  • Home
  • About
  • Posts
  • Start Here
  • Topics
  • Resources
  • RSS
  • Categories
  • AI
  • Career
  • Data Engineering
  • Data Modelling
  • Data Quality & Governance
  • Leadership
  • Life
  • Platforms & Cloud
Hero Image
AWS MWAA: The Practitioner's Unvarnished Guide

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built. But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head. The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

  • AWS
  • MWAA
  • Apache Airflow
  • Astronomer
  • Dagster
  • Prefect
  • Step Functions
  • Orchestration
  • Data Pipelines
Saturday, July 18, 2026 Read
Hero Image
Your Data Platform Costs More Than It Should

Let me tell you about the moment I stopped treating cloud costs as someone else’s problem. We were three months into a Snowflake migration. Everything was humming. Pipelines were green, dashboards were fast, the analytics team was happier than I’d seen them before. I felt good about the work we’d done. Then finance forwarded me the invoice. The number wasn’t catastrophic. But it was significantly higher than what we’d budgeted, and when I started digging, I couldn’t explain where most of it was going. I knew we had warehouses running. I knew we had pipelines executing. But I couldn’t tell you which warehouse was responsible for what cost, which pipelines were the expensive ones, or whether the money was well spent. I had built a platform I was proud of — and I had no idea what it actually cost to operate.

  • Snowflake
  • AWS
  • Cost Optimisation
  • FinOps
  • dbt
  • Data Platform
Saturday, April 25, 2026 Read
Hero Image
Why Your Pipeline Finishes Later Every Month

Let me tell you about a graph that changed how I think about data engineering. A junior engineer on my team — let’s call her Priya — had been tracking something nobody asked her to track. Every morning for two months, she’d noted the timestamp when our main analytics pipeline completed. She wasn’t trying to make a point. She was just curious, because the finance team kept mentioning their dashboards weren’t ready when they arrived at 8 AM anymore.

  • Snowflake
  • AWS
  • Apache Airflow
  • Performance Optimisation
  • dbt
  • Data Freshness
Saturday, April 18, 2026 Read
Hero Image
Stop Building Salesforce Integrations From Scratch

Let me tell you about Marcus. Marcus was on a team I led a few years back. Sharp, motivated, the kind of engineer who actually read documentation before writing code. When the business asked us to get Salesforce data into our warehouse, Marcus volunteered. He’d done API work before. He figured a few weeks, tops. He scoped it carefully. Built a Python service that authenticated via OAuth, pulled Account, Contact, and Opportunity objects through the Bulk API, flattened the nested JSON into relational tables, handled pagination, managed rate limits. Wrote solid tests. Documented everything. The kind of work you’d point to in a code review and say this is how it’s done.

  • Data Engineering
  • Snowflake
  • OpenFlow
  • Salesforce
  • API Integration
  • Schema Evolution
  • Fivetran
  • Data Pipelines
Saturday, April 4, 2026 Read
Hero Image
That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries

The Ten-Minute Query I’m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says ‘Query running… 4 minutes, 37 seconds.’ I lean back in my chair and let out this long sigh that probably says more than I intended. My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I’m about to tell her. I didn’t need to explain.

  • AWS Glue
  • Dimensional Modelling
  • Kimball
  • Data Quality
  • ETL
  • Write-Audit-Publish
  • Apache Iceberg
  • Step Functions
  • SCD
Friday, January 16, 2026 Read
Hero Image
Building Your First AWS Data Pipeline: A Guide for Data Professionals Who've Never Touched Cloud Infrastructure

The spreadsheet that changed everything Here’s a story that might sound familiar. You’re pulling data from an API—maybe daily sales numbers, maybe customer interactions, maybe something else entirely. Every morning, you open your laptop, run a Python script, save the CSV somewhere, and get on with your actual work. It takes maybe five minutes, but it’s five minutes you can’t forget about. Miss a day and you’ve got a gap in your data. Go on vacation? Better hope someone remembers to run your script.

  • AWS
  • Data Pipelines
  • Lambda
  • S3
  • Athena
  • Cloud Computing
  • Data Ingestion
Wednesday, November 26, 2025 Read
Hero Image
Delta-lake - Z-Ordering, Z-Cube, Liquid Clustering and Partitions

Introduction Ever feel like your data lake is more of a data swamp, swallowing queries whole and spitting out eternity? You’re not alone. Managing massive datasets can be a Herculean task, especially when it comes to squeezing out those precious milliseconds of query performance. But fear not, data warriors, for Delta Lake has hidden treasures waiting to be unearthed: Z-ordering, Z-cube, and liquid clustering. Partition Pruning: The OG Hero Before we dive into these exotic beasts, let’s pay homage to the OG hero of data organisation: partition pruning. Imagine your data lake as a meticulously organised library, with each book (partition) shelved by a specific topic (partition column). When a query saunters in, it doesn’t have to wander through every aisle. It simply heads straight for the relevant section, drastically reducing the time it takes to find what it needs. That’s the magic of partition pruning!

  • Delta Lake
Sunday, January 14, 2024 Read
Hero Image
Create a Minecraft (PE)Bedrock Server in GCP

Introduction My eldest daughter, plays with her friends a lot with the Minecraft Pocket Edition (PE). The PE Edition, really limits it to WiiU, xbox or ipad. This is the most connivent gaming version for them as they all have iPads. Occasionally the world they are all playing on gets corrupted and she looses all the work they have done. There is ways to recover it, but its rather time consuming.

  • GCP
  • Minecraft
  • Cloud Gaming
  • Server Setup
  • Family Gaming
  • Bedrock Edition
Sunday, January 15, 2023 Read