Ghost in the data
  • Home
  • About
  • Posts
  • Topics
  • Resources
  • RSS
  • Categories
  • AI Development
  • Analytics Engineering
  • Artificial Intelligence
  • AWS
  • Banking
  • Best Practices
  • Big Data
  • Business Technology
  • Career Development
  • Career Growth
  • Cloud Architecture
  • Cloud Computing
  • Cloud Infrastructure
  • Communication
  • Conflict Resolution
  • Data Architecture
  • Data Culture
  • Data Engineering
  • Data Ethics
  • Data Governance
  • Data Modeling
  • Data Modelling
  • Data Pipelines
  • Data Privacy
  • Data Quality
  • Data Storage
  • Data Warehousing
  • Database Design
  • Dbt
  • Delta-Lake
  • Development
  • Development Tools
  • DevOps
  • Employee Engagement
  • Gaming Servers
  • Google Cloud Platform
  • Hiring
  • Industry Analysis
  • Interviews
  • IT Management
  • Leadership
  • Life Hacks
  • Mindfulness
  • Minecraft
  • Personal
  • Personal Development
  • Personal Finance
  • Pipeline
  • Pipeline Design
  • Productivity
  • Professional Development
  • Professional Growth
  • Promotion
  • Psychology
  • Python
  • Python Tools
  • Setup Guide
  • SQL
  • Stakeholder Management
  • Team Building
  • Team Culture
  • Team Management
  • Technical Architecture
  • Technology Trends
  • Tutorial
  • User Experience
  • Version Control
  • Workplace Dynamics
Hero Image
What an NBA Coach Can Teach Data Leaders About Building Teams That Actually Work

I was three hours into a retrospective that had devolved into blame-shifting when the most senior engineer on the team finally spoke up. “Look,” he said, “we can keep pointing fingers at the data model, or we can admit we don’t actually trust each other enough to have an honest conversation about what went wrong.” The room went quiet. He was right. That moment stuck with me because it exposed something I’ve seen destroy more data teams than bad architecture ever could: the absence of genuine connection between people who spend forty-plus hours a week depending on each other.

  • Leadership
  • Team Building
  • Culture
  • Management
  • Data Teams
  • Remote Work
  • Psychological Safety
Monday, February 2, 2026 Read
Hero Image
Context Engineering: The New Must-Have Skill for Data Engineers

Last year I watched a colleague ask AI to help write a dbt model. The AI spit out perfectly functional SQL—clean syntax, proper CTEs, the works. Looked great. Then I noticed the table would eventually hold 800 million rows. No partitioning. No clustering. Just a raw, unoptimised heap waiting to turn into a query performance nightmare (that would likely become my nightmare to fix). The engineer wasn’t at fault. The AI wasn’t at fault either, really. The AI simply didn’t know that our environment clusters large tables by date. It didn’t know our team’s conventions around incremental models. It couldn’t know, because nobody had told it.

  • AI
  • dbt
  • Data Quality
  • SQL
  • Productivity
  • VSCode
  • Claude
Saturday, January 31, 2026 Read
Hero Image
The Duct Tape Data Engineer

The Engineer Who Ships I want to tell you about a data engineer I worked with. Let’s call her Sarah. Sarah had a reputation. When business stakeholders had an urgent question—the kind that arrives at 4 PM on a Friday with the CEO’s name in the subject line—they went to Sarah. Not to the senior architect with the impeccable data model. Not to the platform team with their carefully orchestrated Airflow DAGs. They went to Sarah.

  • Data Engineering
  • DuckDB
  • Architecture
  • Pragmatism
  • Career Development
  • Technical Strategy
  • Data Platforms
  • Kimball
  • Data Modeling
Saturday, January 24, 2026 Read
Hero Image
The 2026 Data Engineering Strategy Nobody's Writing (But Everyone Needs)

What if I told you the biggest threat to your data platform isn’t technology—it’s that we’ve stopped building the next generation of engineers who’ll run it? Not the latest database that promises to solve everything. Not whether you picked the right orchestrator. The real crisis is that we’ve systematically broken our talent pipeline. And in 2026, that decision is going to start costing us in ways that no amount of tooling can fix.

  • Strategy
  • Team Building
  • Cost Optimization
  • DuckDB
  • AI Tools
  • Career Planning
  • 2026 Trends
  • Future of Work
Thursday, January 15, 2026 Read
Hero Image
The Guerrilla Guide to Data Engineering Interviews

The Scenario That Changes Everything Picture this: You’re sitting in an interview room—or more likely these days, staring at a Zoom window with your carefully curated bookshelf background—and the interviewer asks you about data quality. “Tell me about your experience with data quality,” they say. You have two choices. Choice A: “Data quality is really important in data engineering. It involves ensuring data is accurate, complete, consistent, and timely. I believe strongly in implementing data quality checks throughout the pipeline.”

  • Interviews
  • Career Growth
  • Technical Assessment
  • SQL
  • Data Modeling
  • Problem Solving
  • Delta Lake
  • dbt
  • Data Quality
Sunday, January 11, 2026 Read
Hero Image
Why Your Ideas Die in Planning Meetings

The silence that kills good ideas One morning, I sat in yet another meeting where we just spent two weeks backfilling a table then we found it was riddled with issues with the data. Even if we resolve the issue, it would then be another 2 weeks to backfill the data, there has to be a better way. “So, what do we think? Give me your best ideas for tackling this.”

  • Team Culture
  • Collaboration
  • Psychological Safety
  • Innovation
  • Change Management
  • Technical Leadership
  • Data Teams
Wednesday, January 7, 2026 Read
Hero Image
The Science of Conversation for people who hate small talk

One morning, I watched a data engineer struggle with using AI for thirty minutes, trying to debug a DBT job. The problem wasn’t the LLM’s capabilities—it was how the engineer framed the question. No context about what they’d already tried. No explanation of the expected versus actual output. Just “fix this code” followed by a massive code dump. This same engineer had similar struggles with stakeholders. Presentations that assumed too much context. Emails that buried the ask. Meetings where they answered questions nobody asked.

  • Communication
  • Soft Skills
  • Leadership
  • Career Growth
  • Team Building
  • Stakeholder Management
  • Professional Development
Sunday, January 4, 2026 Read
Hero Image
Financial Independence: Your Shield Against Job Loss Fear

The Fear That Follows You Home One evening, after pushing another commit past midnight, I couldn’t bring myself to sit up. Not because I was tired—though I was. Not because the commit had issues—it went smoothly, and tested all fine. I couldn’t get up because I’d spent the entire day with a knot in my stomach, wondering if our team would survive the next round of “organizational restructuring.” Here’s what made it worse: I had no idea if my fear was rational. Were we really at risk? Or was I just catastrophizing? The uncertainty was eating me alive.

  • financial independence
  • job security
  • emergency fund
  • career development
  • mental health
  • workplace stress
  • budgeting
  • redundancy
Sunday, November 2, 2025 Read
Hero Image
Mastering One-on-One Meetings: Building Trust and Driving Growth

Introduction Have you ever felt that slight relief when your manager cancels your 1:1 meeting? Early in my career as a data professional, I viewed 1:1s as just another checkbox on my calendar—often treating them like mini-standups where I’d rattle off project updates before awkwardly waiting for the meeting to end. Looking back, I realize how much potential growth I left on the table. As I progressed from an individual contributor to leading a team, I’ve learned that 1:1 meetings aren’t administrative burdens—they’re golden opportunities for trust-building, relationship development, and strategic alignment that many of us simply don’t know how to leverage.

  • One-on-One Meetings
  • Management
  • Emotional Intelligence
  • Trust Building
  • Workplace Communication
  • Professional Development
  • Feedback
  • Mentorship
Saturday, March 22, 2025 Read
Hero Image
From Senior to Staff: Navigating the Data Engineering Leadership Path

Introduction: The Critical Inflection Point The transition from Senior to Staff Engineer represents a pivotal moment in any technical career path. It’s the point where your impact extends beyond your code and transforms into something much more profound – true technical leadership. While this shift can feel daunting, it also opens doors to some of the most rewarding work of your career. The beautiful thing about the engineering career ladder is that it uniquely allows for advancement without stepping away from the technical work that many of us love.

  • Staff Engineer
  • Career Growth
  • Technical Leadership
  • Chapter Lead
  • Data Leadership
  • Engineering Career
  • Promotion
Sunday, March 2, 2025 Read
Hero Image
Your First 90 Days as a Data Engineer: A Strategic Guide

Introduction Landing your first data engineering role—or starting at a new company—is both exhilarating and daunting. After navigating multiple interviews and accepting an offer, you’ve finally arrived at your desk with a new laptop and company swag (if your lucky). Even now, after solving countless problems ranging from minor bugs to enterprise-scale data challenges, I still occasionally feel that flutter of uncertainty in my stomach, when starting a new role. What if I don’t know what I’m doing? What if I make a mistake?

  • Onboarding
  • Professional Growth
  • Team Collaboration
  • Career Advice
  • Data Culture
Sunday, February 23, 2025 Read
Hero Image
Mastering Data Interviews: A Comprehensive Guide

Introduction After nearly two decades in the data engineering field, I’ve sat on both sides of the interview table countless times. Whether you’re a seasoned professional looking to change roles or a newcomer trying to break into the field, the interview process for data engineering positions can be both challenging and mysterious. There’s often uncertainty about what questions you’ll face, what skills you need to demonstrate, and what interviewers are really looking for beneath the surface.

  • Interviews
  • Technical Assessment
  • Career Growth
  • SQL
  • Data Modeling
  • Problem Solving
Saturday, February 22, 2025 Read
  • ««
  • «
  • 1
  • 2
  • »
  • »»