Ghost in the data
  • Home
  • About
  • Posts
  • Topics
  • Resources
  • RSS
  • Categories
  • AI Development
  • AI Tools
  • Analytics Engineering
  • Artificial Intelligence
  • AWS
  • Banking
  • Best Practices
  • Big Data
  • Business Technology
  • Career Development
  • Career Growth
  • Cloud Architecture
  • Cloud Computing
  • Cloud Infrastructure
  • Communication
  • Conflict Resolution
  • Data Architecture
  • Data Culture
  • Data Engineering
  • Data Ethics
  • Data Governance
  • Data Modeling
  • Data Modelling
  • Data Pipelines
  • Data Privacy
  • Data Quality
  • Data Storage
  • Data Warehousing
  • Database Design
  • Dbt
  • Delta-Lake
  • Development
  • Development Tools
  • DevOps
  • Employee Engagement
  • Gaming Servers
  • Google Cloud Platform
  • Hiring
  • Industry Analysis
  • Interviews
  • IT Management
  • Leadership
  • Life Hacks
  • Mindfulness
  • Minecraft
  • Personal
  • Personal Development
  • Personal Finance
  • Pipeline
  • Pipeline Design
  • Productivity
  • Professional Development
  • Professional Growth
  • Promotion
  • Psychology
  • Python
  • Python Tools
  • Setup Guide
  • SQL
  • Stakeholder Management
  • Team
  • Team Building
  • Team Culture
  • Team Management
  • Technical Architecture
  • Technology Trends
  • Tools
  • Tutorial
  • User Experience
  • Version Control
  • Workplace Dynamics
Hero Image
The Style Isn't the Skill

I was a kid when I first watched Bruce Lee, soon after and my friend signed up for kung fu lessons. What hooked me wasn’t the fighting. It was the speed at which Bruce moved. The more you watched, though, the more you noticed something sitting behind the speed. He was obviously smart, fast, cool, and clearly thinking several moves ahead of everyone else in the room. Even as a kid you could tell the fighting was the visible bit of something much bigger.

  • Leadership
  • Career Development
  • Performance Reviews
  • Self Evaluation
  • Data Modelling
  • Streaming
  • Skills
  • Philosophy
Monday, August 31, 2026 Read
Hero Image
You Can't Incentivise a Pipeline That Doesn't Break

I worked alongside a data engineer who was, by every formal measure, the best performer on the team. He was also quietly destroying the platform. Not maliciously. He was optimising for the thing being measured. His work shipped fast because he skipped the edge case analysis. He closed tickets at first resolution without ever checking whether the underlying pattern would recur. He didn’t review anyone else’s PRs (not his KPIs, so why would he?).

  • Leadership
  • Team Culture
  • Performance Reviews
  • Data Engineering
  • Management
  • Incentives
  • Motivation
Saturday, July 4, 2026 Read
Hero Image
Your Team Already Has Patterns. They Just Don't Know It.

When I started a new role, one of the first things I did was try to understand how data moved through the system. Not the dashboards, not the data models — the pipes. Where did things come from? How did they get in? What happened to them along the way? There were somewhere between twenty and thirty source systems feeding the platform. Not a massive number, but enough to tell a story when you looked at the ingestion layer all at once. What I found was that all the pipelines had originated from two base templates. A sensible starting point. The kind of thing a small team puts in place early to stop complete chaos.

  • Data Engineering
  • Pattern Bank
  • Team Culture
  • Project Management
  • Solution Design
  • Engineering Leadership
  • Estimation
Saturday, June 27, 2026 Read
Hero Image
Keep Moving

Some days I open my laptop and by 5pm I genuinely cannot tell you what I did. Not because it was complicated. Not because there were emergencies. The stand-up happened. A few Teams messages were sent. A ticket was groomed. A document was “reviewed”. A meeting was attended where everyone agreed something was important and then the meeting ended and nothing changed. And then somehow it was evening and the pipeline I meant to fix was exactly as broken as it was in the morning.

  • Productivity
  • Data Engineering
  • Platform Strategy
  • Team Leadership
  • Technical Strategy
Tuesday, June 23, 2026 Read
Hero Image
The Competitive Moat That AI Can't Replicate

The Restaurant That Refused to Take Bookings Online Let me tell you a story about a restaurant owner who became obsessed with human connection. He didn’t want people booking online. He wanted them to call. He wanted the ritual of a human voice, the small exchange about an anniversary or a first date, the warmth of being recognised. His team thought he was losing his mind. Online bookings were standard. Everyone did it. Why make customers work harder?

  • Human Connection
  • Trust
  • Customer Experience
  • Leadership
  • AI
  • Employee Experience
  • Organisational Culture
Saturday, June 13, 2026 Read
Hero Image
Don't Go Dark: Visibility Is a Data Engineering Skill

There’s a specific kind of silence in data engineering that I’ve learned to fear. Not the silence of a system that’s working well. Not the comfortable quiet of a team in flow. I mean the silence of a project that’s been running for three weeks and you still can’t point to a single visible thing it has produced. The kind of silence where, if your manager stopped you in the hallway and asked “how’s that migration going?”, you’d say “fine” because saying anything more accurate would require explaining things you haven’t fully articulated yet — even to yourself.

  • Communication
  • Data Quality
  • GitHub
  • dbt
  • Remote Work
  • Career Development
  • Engineering Culture
  • Psychological Safety
Saturday, May 23, 2026 Read
Hero Image
Your Friends Will Be There for You. Your Work Won't.

There’s a table in a rented house in Queenscliff — nothing fancy, just whatever furniture came with the place — with a hand-drawn map spread across it, dice scattered at the edges, and snacks slowly migrating toward the centre of the board. Once a year, Steve, one of our friends, organises for us to make a trip down to Queenscliff to play tabletop RPGs for a weekend. More recently, we’ve started playing monthly or fortnightly at home too. Nothing glamorous about any of it. We’re not Glass Cannon Network or Critical Role — no cameras, no production values, no audience watching us roleplay with solemn gravity.

  • Leadership
  • Career Development
  • Mental Health
  • Burnout
  • Wellbeing
  • Friendship
  • Work-Life Balance
Wednesday, March 18, 2026 Read
Hero Image
12 Steps to Better Data Engineering

Let me tell you about the moment I stopped trusting architecture diagrams. I was three days into a new role, getting up to speed with the data team. Smart people. Modern stack. On paper, everything looked right. They walked me through a beautiful data platform diagram: clean lines, labelled layers, colour-coded domains. It looked like something you’d see in a data conference. Then I asked a question that changed everything: “Can you rebuild your finance table from scratch right now?”

  • Data Engineering
  • dbt
  • Snowflake
  • GitHub Actions
  • AWS
  • Data Quality
  • CI/CD
  • Data Contracts
Saturday, March 7, 2026 Read
Hero Image
What an NBA Coach Can Teach Data Leaders About Building Teams That Actually Work

I was three hours into a retrospective that had devolved into blame-shifting when the most senior engineer on the team finally spoke up. “Look,” he said, “we can keep pointing fingers at the data model, or we can admit we don’t actually trust each other enough to have an honest conversation about what went wrong.” The room went quiet. He was right. That moment stuck with me because it exposed something I’ve seen destroy more data teams than bad architecture ever could: the absence of genuine connection between people who spend forty-plus hours a week depending on each other.

  • Leadership
  • Team Building
  • Culture
  • Management
  • Data Teams
  • Remote Work
  • Psychological Safety
Monday, February 2, 2026 Read
Hero Image
The Duct Tape Data Engineer

The Engineer Who Ships I want to tell you about a data engineer I worked with. Let’s call her Sarah. Sarah had a reputation. When business stakeholders had an urgent question—the kind that arrives at 4 PM on a Friday with the CEO’s name in the subject line—they went to Sarah. Not to the senior architect with the impeccable data model. Not to the platform team with their carefully orchestrated Airflow DAGs. They went to Sarah.

  • Data Engineering
  • DuckDB
  • Architecture
  • Pragmatism
  • Career Development
  • Technical Strategy
  • Data Platforms
  • Kimball
  • Data Modeling
Saturday, January 24, 2026 Read
Hero Image
Why Your Ideas Die in Planning Meetings

The silence that kills good ideas One morning, I sat in yet another meeting where we just spent two weeks backfilling a table then we found it was riddled with issues with the data. Even if we resolve the issue, it would then be another 2 weeks to backfill the data, there has to be a better way. “So, what do we think? Give me your best ideas for tackling this.”

  • Team Culture
  • Collaboration
  • Psychological Safety
  • Innovation
  • Change Management
  • Technical Leadership
  • Data Teams
Wednesday, January 7, 2026 Read
Hero Image
The Science of Conversation for people who hate small talk

One morning, I watched a data engineer struggle with using AI for thirty minutes, trying to debug a DBT job. The problem wasn’t the LLM’s capabilities—it was how the engineer framed the question. No context about what they’d already tried. No explanation of the expected versus actual output. Just “fix this code” followed by a massive code dump. This same engineer had similar struggles with stakeholders. Presentations that assumed too much context. Emails that buried the ask. Meetings where they answered questions nobody asked.

  • Communication
  • Soft Skills
  • Leadership
  • Career Growth
  • Team Building
  • Stakeholder Management
  • Professional Development
Sunday, January 4, 2026 Read
  • ««
  • «
  • 1
  • 2
  • »
  • »»