Ghost in the data
  • Home
  • About
  • Posts
  • Topics
  • Resources
  • RSS
  • Categories
  • AI Development
  • AI Tools
  • Analytics Engineering
  • Artificial Intelligence
  • AWS
  • Banking
  • Best Practices
  • Big Data
  • Business Technology
  • Career Development
  • Career Growth
  • Cloud Architecture
  • Cloud Computing
  • Cloud Infrastructure
  • Communication
  • Conflict Resolution
  • Data Architecture
  • Data Culture
  • Data Engineering
  • Data Ethics
  • Data Governance
  • Data Modeling
  • Data Modelling
  • Data Pipelines
  • Data Privacy
  • Data Quality
  • Data Storage
  • Data Warehousing
  • Database Design
  • Dbt
  • Delta-Lake
  • Development
  • Development Tools
  • DevOps
  • Employee Engagement
  • Gaming Servers
  • Google Cloud Platform
  • Hiring
  • Industry Analysis
  • Interviews
  • IT Management
  • Leadership
  • Life Hacks
  • Mindfulness
  • Minecraft
  • Personal
  • Personal Development
  • Personal Finance
  • Pipeline
  • Pipeline Design
  • Productivity
  • Professional Development
  • Professional Growth
  • Promotion
  • Psychology
  • Python
  • Python Tools
  • Setup Guide
  • SQL
  • Stakeholder Management
  • Team
  • Team Building
  • Team Culture
  • Team Management
  • Technical Architecture
  • Technology Trends
  • Tools
  • Tutorial
  • User Experience
  • Version Control
  • Workplace Dynamics
Hero Image
AWS MWAA: The Practitioner's Unvarnished Guide

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built. But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head. The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

  • AWS
  • MWAA
  • Apache Airflow
  • Astronomer
  • Dagster
  • Prefect
  • Step Functions
  • Orchestration
  • Data Pipelines
Saturday, July 18, 2026 Read
Hero Image
That Tuesday Morning When I Finally Fixed Our Ten-Minute Queries

The Ten-Minute Query I’m sitting at my laptop on a Tuesday morning, waiting. The progress bar on my screen says ‘Query running… 4 minutes, 37 seconds.’ I lean back in my chair and let out this long sigh that probably says more than I intended. My manager walks past my desk. She glances at my screen, and I can see that look—the one that says she already knows what I’m about to tell her. I didn’t need to explain.

  • AWS Glue
  • Dimensional Modeling
  • Kimball Methodology
  • Data Quality
  • ETL
  • Write-Audit-Publish
  • Apache Iceberg
  • Step Functions
  • SCD Type 2
Friday, January 16, 2026 Read
Hero Image
Building Your First AWS Data Pipeline: A Guide for Data Professionals Who've Never Touched Cloud Infrastructure

The spreadsheet that changed everything Here’s a story that might sound familiar. You’re pulling data from an API—maybe daily sales numbers, maybe customer interactions, maybe something else entirely. Every morning, you open your laptop, run a Python script, save the CSV somewhere, and get on with your actual work. It takes maybe five minutes, but it’s five minutes you can’t forget about. Miss a day and you’ve got a gap in your data. Go on vacation? Better hope someone remembers to run your script.

  • AWS
  • Data Pipelines
  • Lambda
  • S3
  • Athena
  • Cloud Computing
  • Data Ingestion
Wednesday, November 26, 2025 Read