Ghost in the data
  • Home
  • About
  • Posts
  • Topics
  • Resources
  • Posts
  • 2025
    • UV Tools
    • Zsh Virtual Environments
    • Piracy Service Problem
    • 2025 Data Trends
    • Data Modeling Approaches
    • MacOS Dev Setup
    • Windows Dev Setup
    • Business Context Guide
    • Data Impact
    • Data Engineering Interviews
    • First 90 Days as Data Engineer
    • Senior to Staff Engineer
    • LLMs for Business Part 1
    • LLMs for Business Part 2
    • Mastering 1:1 Meetings
    • AI Prompting Secret
    • Conceptual Data Modeling
    • WAP Pattern for Data Pipelines
    • AI Simplified
    • dbt Fusion: The Engine Upgrade
    • Continuous Integration for Data Teams
    • Claude Code AI Agents
    • Clear Communication Superpower
    • Compliance vs Commitment
    • D&D Leadership
    • Reflective Best Self
    • Financial Independence
    • Dimensional Modeling Lives
    • Balancing Data Accessibility & Privacy
    • Data Quality Crisis
    • Data Quality Framework
    • AWS Data Pipeline
  • 2024
    • Delta-lake
    • Data Normalisation
    • Data Profiling
    • Defensive Engineering
    • CI/CD
    • Setup Docker and Airflow
    • Find and Attract Data Engineers
    • 17 Years of Insights
    • Relationship Building
    • Individual Contributor
  • 2023
    • GitBash with SSH
    • Journalling
    • Minecraft Server in GCP
    • Onboarding a data team
    • File Format for Big Data
    • Incident Management
    • Data Vault
    • Books that are worth you time?
Hero Image
Building Your First AWS Data Pipeline: A Guide for Data Professionals Who've Never Touched Cloud Infrastructure

The spreadsheet that changed everything Here’s a story that might sound familiar. You’re pulling data from an API—maybe daily sales numbers, maybe customer interactions, maybe something else entirely. Every morning, you open your laptop, run a Python script, save the CSV somewhere, and get on with your actual work. It takes maybe five minutes, but it’s five minutes you can’t forget about. Miss a day and you’ve got a gap in your data. Go on vacation? Better hope someone remembers to run your script.

  • AWS
  • Data Pipelines
  • Lambda
  • S3
  • Athena
  • Cloud Computing
  • Data Ingestion
Wednesday, November 26, 2025 Read
Hero Image
The Four Stages of Data Quality: From Hidden Costs to Measurable Value

This is the fundamental problem with data quality. You know it matters. Everyone knows it matters. But until you can quantify the impact, connect it to business outcomes, and build a credible business case, it remains this abstract thing that’s important but never urgent enough to properly fund. I wrote a practical guide to data quality last week that walks through hands-on implementation—the SQL queries, the profiling techniques, the actual mechanics of finding and fixing data issues. Think of that as the “how to use the tools” guide. This article is different. This is the “why these tools matter and how to convince your organization to actually use them” guide.

  • Data Quality
  • ROI
  • Business Case
  • Data Governance
  • Strategy
  • Frameworks
Monday, November 24, 2025 Read
Hero Image
When Your Data Quality Fails at 9 PM on a Friday

When everything goes wrong at once It’s 9 PM on a Friday. You’re halfway through your second beer, finally relaxing after a brutal week. Your phone buzzes. Then it buzzes again. And again. The support team’s in full panic mode, your manager’s calling, and somewhere in Melbourne, two very angry guests are standing outside the same Airbnb property—both holding confirmation emails that say the place is theirs for the weekend.

  • Data Quality
  • SQL
  • Database Design
  • Data Validation
  • Testing
  • Data Engineering
  • Production Issues
Saturday, November 22, 2025 Read
Hero Image
Balancing Data Accessibility and Privacy in Financial Services

The Data Tightrope: Where Accessibility Meets Privacy Let’s face it—in today’s data landscape, data is simultaneously your most valuable asset and your biggest potential liability. Finding that sweet spot where data remains accessible enough to drive business decisions while being locked down enough to satisfy privacy regulations. It’s not just about ticking compliance boxes—it’s about maintaining customer trust while still extracting every bit of analytical value from your data assets.

  • DataPrivacy
  • Anonymization
  • RetentionPolicies
  • BankingData
  • DataMinimization
  • GDPR
  • DataGovernance
Friday, November 21, 2025 Read
Hero Image
When Pirates Offered Better Service

The Day Music Changed Forever On June 1, 1999, an eighteen-year-old kid in a Northeastern University dorm room launched something that would bring the music industry to its knees. Shawn Fanning called it Napster, and within two years, 80 million people were using it to download 14,000 songs every minute.1 The technology was simple: a central server indexed which songs each user had, then let computers talk directly to each other. No complicated setup. No technical expertise required. Just type in “Metallica” and boom—there it was.

  • DataGovernance
  • UserExperience
  • ShadowIT
  • DataDemocratization
  • Leadership
  • ServiceDesign
Sunday, November 16, 2025 Read
Hero Image
Why Dimensional Modeling Isn't Dead—It's Just Getting Started

The Great Data Modeling Debate Nobody Asked For Another meeting where someone confidently declared, “We don’t need data modeling anymore—just dump everything in the data lake and let analysts figure it out.” I’ve heard variations of this statement for years now, in meetings or at conferences. The pitch is always the same: traditional data warehousing is dead, dimensional modeling is a relic from the 90s, and modern big data tools have made structured modeling obsolete. Schema-on-read is the future. Agility over architecture.

  • DimensionalModeling
  • DataWarehouse
  • DataModeling
  • DataQuality
  • Analytics
  • Kimball
  • BigData
Friday, November 7, 2025 Read
Hero Image
Financial Independence: Your Shield Against Job Loss Fear

The Fear That Follows You Home One evening, after pushing another commit past midnight, I couldn’t bring myself to sit up. Not because I was tired—though I was. Not because the commit had issues—it went smoothly, and tested all fine. I couldn’t get up because I’d spent the entire day with a knot in my stomach, wondering if our team would survive the next round of “organizational restructuring.” Here’s what made it worse: I had no idea if my fear was rational. Were we really at risk? Or was I just catastrophizing? The uncertainty was eating me alive.

  • financial independence
  • job security
  • emergency fund
  • career development
  • mental health
  • workplace stress
  • budgeting
  • redundancy
Sunday, November 2, 2025 Read
Hero Image
Discover our best selves

Introduction Here’s what nobody tells you about: your biggest blind spot isn’t your technical weaknesses—it’s your strengths. That sounds weird doesn’t it. Most of us can recite our shortcomings on command, yet we struggle to articulate what we’re genuinely good at. This isn’t just modesty. It’s a fundamental quirk of human psychology that keeps us from reaching our full potential. The Reflected Best Self Portrait exercise, developed by researchers at the University of Michigan’s Ross School of Business, flips this script entirely. Instead of the usual deficit-focused approach (“here’s what you need to fix”), it asks a radical question: what if we built our careers around who we are when we’re at our absolute best? For data professionals navigating an industry that’s constantly evolving—where yesterday’s cutting-edge tool becomes today’s legacy system—this approach isn’t just refreshing. It’s essential.

  • Strengths
  • Self-Awareness
  • Career Development
  • Data Leadership
  • Professional Growth
Saturday, November 1, 2025 Read
Hero Image
Rolling for Initiative: How Dungeons & Dragons Taught Me Everything About Team Leadership

The Unexpected Training Ground I never thought a game about pretending to be elves and wizards would teach me more about leadership than any management training I’ve ever attended. But here we are. Growing up, Dungeons & Dragons was this weird thing you did if you had friends—which, honestly, I didn’t have a lot of. My best friend and I played these sort of solo-person adventures, just the two of us hunched over character sheets and dice on pumped up inflatable air beds in the living room. It wasn’t exactly the epic party campaigns you see on Critical Role, but it was still magic. Years passed. We got older. Eventually, we managed to rope in some other friends, and now? Now we’ve got a whole group that gets together yearly, and those weekends have become something we all look forward to more than just about anything else.

  • Team Building
  • Leadership
  • Communication
  • Problem Solving
  • Culture
Sunday, October 26, 2025 Read
Hero Image
From Compliance to Commitment: What Decades of Research Reveals About Moral Courage

The Question That Launched a Six-Year Study A twelve-year-old boy watched from a rooftop as soldiers spent 18 hours eliminating a thousand people. He was the only survivor from his entire family. When he finally escaped and stumbled barefoot across the countryside, a peasant woman opened her door, took one look at him, and without hesitation pulled him inside—despite knowing she’d face execution if discovered. Four decades later, that boy—now sociology professor Samuel Oliner—launched a six-year study interviewing 700 Europeans to answer the question that haunted him: Why did she risk everything when so many others didn’t?

  • Moral Development
  • Leadership
  • Organizational Culture
  • Data Ethics
  • Decision Making
  • Psychological Safety
Saturday, October 25, 2025 Read
Hero Image
The Superpower of Clear Communication: Mastering Volume, Melody, Tonality, and Pause

The Meeting That Changed Everything Picture this: It’s the quarterly business review, and your team’s project is on the chopping block. Budget cuts are looming, and you have fifteen minutes to convince the leadership team that your data initiative deserves continued funding. You’ve prepared extensively. Your slides are perfect. Your data is compelling. But as you begin presenting, you notice glazed expressions around the table. One person is checking her phone. Another is drumming his fingers impatiently.

  • communication
  • leadership
  • public-speaking
  • professional-skills
  • presentation-skills
  • team-management
  • vocal-techniques
  • business-communication
Saturday, September 20, 2025 Read
Hero Image
Building AI Agents with Claude Code

Introduction Imagine you’re reviewing a pull request with dozens of SQL files, each containing complex queries for your data pipeline. You spot inconsistent formatting, or syntax which doesn’t work with your infrastructure. Sound familiar? It’s common for data professionals to struggle with maintaining consistent SQL standards across their projects, especially when working with specialized platforms and it can be time consuming to review these elements within a peer review. It would be better use of time to focus on the hard thinking elements, like logic etc. However these small syntax or style issues, can be distracting. Well at least they are for me.

  • claude-code
  • sql-agents
  • starburst
  • delta-lake
  • trino
  • sql-validation
  • dbt
  • data-engineering
  • ai-tools
  • vscode
Saturday, September 13, 2025 Read
  • ««
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • »»