Ghost in the data
  • Home
  • About
  • Posts
  • Topics
  • Resources
  • RSS
  • Posts
  • 2026
    • Talk
    • Brainstorming
    • Guerrilla Interview Guide
    • 2026 Strategy
    • Dimensional Modeling AWS
    • Duct Tape Data Engineer
    • AI Peer Reviewer
    • NBA Coach Lessons for Data Leaders
    • For Sooty
    • Healing Tables SCD2
    • WAP Iceberg Snowflake
    • The CSV Test Suite Nobody Writes
    • 12 Steps to Better Data Engineering
    • Your Data Model Isn't Broken Pt I
    • Your Friends Will Be There
    • Fix Your Data Without Permission
    • Your Data Model Isn't Broken Pt II
    • Stop Building Salesforce Integrations
    • Why Your Pipeline Finishes Later Every Month
  • 2025
    • UV Tools
    • Zsh Virtual Environments
    • Piracy Service Problem
    • 2025 Data Trends
    • Data Modeling Approaches
    • MacOS Dev Setup
    • Windows Dev Setup
    • Business Context Guide
    • Data Impact
    • Data Engineering Interviews
    • First 90 Days as Data Engineer
    • Senior to Staff Engineer
    • LLMs for Business Part 1
    • LLMs for Business Part 2
    • Mastering 1:1 Meetings
    • Data Quality Test
    • AI Prompting Secret
    • Conceptual Data Modeling
    • WAP Pattern for Data Pipelines
    • AI Simplified
    • dbt Fusion: The Engine Upgrade
    • Continuous Integration for Data Teams
    • Claude Code AI Agents
    • Clear Communication Superpower
    • Compliance vs Commitment
    • D&D Leadership
    • Reflective Best Self
    • Financial Independence
    • Dimensional Modeling Lives
    • Balancing Data Accessibility & Privacy
    • Data Quality Crisis
    • Data Quality Framework
    • AWS Data Pipeline
    • Invisible PR
    • AI's Twin Crises
  • 2024
    • Delta-lake
    • Data Normalisation
    • Data Profiling
    • Defensive Engineering
    • CI/CD
    • Setup Docker and Airflow
    • Find and Attract Data Engineers
    • 17 Years of Insights
    • Relationship Building
    • Individual Contributor
  • 2023
    • GitBash with SSH
    • Journalling
    • Minecraft Server in GCP
    • Onboarding a data team
    • File Format for Big Data
    • Incident Management
    • Data Vault
    • Books that are worth you time?
Hero Image
Balancing Data Accessibility and Privacy in Financial Services

The Data Tightrope: Where Accessibility Meets Privacy Let’s face it—in today’s data landscape, data is simultaneously your most valuable asset and your biggest potential liability. Finding that sweet spot where data remains accessible enough to drive business decisions while being locked down enough to satisfy privacy regulations. It’s not just about ticking compliance boxes—it’s about maintaining customer trust while still extracting every bit of analytical value from your data assets.

  • DataPrivacy
  • Anonymization
  • RetentionPolicies
  • BankingData
  • DataMinimization
  • GDPR
  • DataGovernance
Friday, November 21, 2025 Read
Hero Image
When Pirates Offered Better Service

The Day Music Changed Forever On June 1, 1999, an eighteen-year-old kid in a Northeastern University dorm room launched something that would bring the music industry to its knees. Shawn Fanning called it Napster, and within two years, 80 million people were using it to download 14,000 songs every minute.1 The technology was simple: a central server indexed which songs each user had, then let computers talk directly to each other. No complicated setup. No technical expertise required. Just type in “Metallica” and boom—there it was.

  • DataGovernance
  • UserExperience
  • ShadowIT
  • DataDemocratization
  • Leadership
  • ServiceDesign
Sunday, November 16, 2025 Read
Hero Image
Why Dimensional Modeling Isn't Dead—It's Just Getting Started

The Great Data Modeling Debate Nobody Asked For Another meeting where someone confidently declared, “We don’t need data modeling anymore—just dump everything in the data lake and let analysts figure it out.” I’ve heard variations of this statement for years now, in meetings or at conferences. The pitch is always the same: traditional data warehousing is dead, dimensional modeling is a relic from the 90s, and modern big data tools have made structured modeling obsolete. Schema-on-read is the future. Agility over architecture.

  • DimensionalModeling
  • DataWarehouse
  • DataModeling
  • DataQuality
  • Analytics
  • Kimball
  • BigData
Friday, November 7, 2025 Read
Hero Image
Financial Independence: Your Shield Against Job Loss Fear

The Fear That Follows You Home One evening, after pushing another commit past midnight, I couldn’t bring myself to sit up. Not because I was tired—though I was. Not because the commit had issues—it went smoothly, and tested all fine. I couldn’t get up because I’d spent the entire day with a knot in my stomach, wondering if our team would survive the next round of “organizational restructuring.” Here’s what made it worse: I had no idea if my fear was rational. Were we really at risk? Or was I just catastrophizing? The uncertainty was eating me alive.

  • financial independence
  • job security
  • emergency fund
  • career development
  • mental health
  • workplace stress
  • budgeting
  • redundancy
Sunday, November 2, 2025 Read
Hero Image
Discover our best selves

Introduction Here’s what nobody tells you about: your biggest blind spot isn’t your technical weaknesses—it’s your strengths. That sounds weird doesn’t it. Most of us can recite our shortcomings on command, yet we struggle to articulate what we’re genuinely good at. This isn’t just modesty. It’s a fundamental quirk of human psychology that keeps us from reaching our full potential. The Reflected Best Self Portrait exercise, developed by researchers at the University of Michigan’s Ross School of Business, flips this script entirely. Instead of the usual deficit-focused approach (“here’s what you need to fix”), it asks a radical question: what if we built our careers around who we are when we’re at our absolute best? For data professionals navigating an industry that’s constantly evolving—where yesterday’s cutting-edge tool becomes today’s legacy system—this approach isn’t just refreshing. It’s essential.

  • Strengths
  • Self-Awareness
  • Career Development
  • Data Leadership
  • Professional Growth
Saturday, November 1, 2025 Read
Hero Image
Rolling for Initiative: How Dungeons & Dragons Taught Me Everything About Team Leadership

The Unexpected Training Ground I never thought a game about pretending to be elves and wizards would teach me more about leadership than any management training I’ve ever attended. But here we are. Growing up, Dungeons & Dragons was this weird thing you did if you had friends—which, honestly, I didn’t have a lot of. My best friend and I played these sort of solo-person adventures, just the two of us hunched over character sheets and dice on pumped up inflatable air beds in the living room. It wasn’t exactly the epic party campaigns you see on Critical Role, but it was still magic. Years passed. We got older. Eventually, we managed to rope in some other friends, and now? Now we’ve got a whole group that gets together yearly, and those weekends have become something we all look forward to more than just about anything else.

  • Team Building
  • Leadership
  • Communication
  • Problem Solving
  • Culture
Sunday, October 26, 2025 Read
Hero Image
From Compliance to Commitment: What Decades of Research Reveals About Moral Courage

The Question That Launched a Six-Year Study A twelve-year-old boy watched from a rooftop as soldiers spent 18 hours eliminating a thousand people. He was the only survivor from his entire family. When he finally escaped and stumbled barefoot across the countryside, a peasant woman opened her door, took one look at him, and without hesitation pulled him inside—despite knowing she’d face execution if discovered. Four decades later, that boy—now sociology professor Samuel Oliner—launched a six-year study interviewing 700 Europeans to answer the question that haunted him: Why did she risk everything when so many others didn’t?

  • Moral Development
  • Leadership
  • Organizational Culture
  • Data Ethics
  • Decision Making
  • Psychological Safety
Saturday, October 25, 2025 Read
Hero Image
The Superpower of Clear Communication: Mastering Volume, Melody, Tonality, and Pause

The Meeting That Changed Everything Picture this: It’s the quarterly business review, and your team’s project is on the chopping block. Budget cuts are looming, and you have fifteen minutes to convince the leadership team that your data initiative deserves continued funding. You’ve prepared extensively. Your slides are perfect. Your data is compelling. But as you begin presenting, you notice glazed expressions around the table. One person is checking her phone. Another is drumming his fingers impatiently.

  • communication
  • leadership
  • public-speaking
  • professional-skills
  • presentation-skills
  • team-management
  • vocal-techniques
  • business-communication
Saturday, September 20, 2025 Read
Hero Image
Building AI Agents with Claude Code

Introduction Imagine you’re reviewing a pull request with dozens of SQL files, each containing complex queries for your data pipeline. You spot inconsistent formatting, or syntax which doesn’t work with your infrastructure. Sound familiar? It’s common for data professionals to struggle with maintaining consistent SQL standards across their projects, especially when working with specialized platforms and it can be time consuming to review these elements within a peer review. It would be better use of time to focus on the hard thinking elements, like logic etc. However these small syntax or style issues, can be distracting. Well at least they are for me.

  • claude-code
  • sql-agents
  • starburst
  • delta-lake
  • trino
  • sql-validation
  • dbt
  • data-engineering
  • ai-tools
  • vscode
Saturday, September 13, 2025 Read
Hero Image
Continuous Integration for Data Teams: Beyond the Buzzwords

The Day Everything Broke (And How CI Could Have Saved Us) Picture this: It’s 9 AM on a Monday, and your Slack is exploding. The executive dashboard is showing impossible numbers. Customer support is fielding complaints about incorrect billing amounts. The marketing team is questioning why their conversion metrics suddenly dropped to zero. You trace it back to a seemingly innocent change you merged Friday afternoon—a simple column rename that seemed harmless enough. But that “harmless” change cascaded through your entire data pipeline, breaking downstream models, dashboards, and automated reports.

  • ContinuousIntegration
  • DataQuality
  • dbt
  • DevOps
  • DataEngineering
  • GitHub
  • Datafold
  • DataValidation
Saturday, June 28, 2025 Read
Hero Image
dbt Fusion: The Engine Upgrade That's Got Everyone Talking

When Your Favorite Tool Gets a Makeover You know that feeling when your favorite app suddenly changes its interface? That mix of excitement and anxiety about whether the changes will actually improve your workflow or just mess with muscle memory you’ve spent years building. That’s exactly what happened when dbt Labs dropped dbt Fusion on the analytics engineering community. The reactions were… let’s call them passionate. Some folks were celebrating like they’d just discovered fire, while others were questioning whether this marked the beginning of the end for open-source dbt.

  • dbt
  • DataEngineering
  • AnalyticsEngineering
  • OpenSource
  • DataTools
  • SQL
  • DataModeling
Saturday, June 21, 2025 Read
Hero Image
AI Simplified: Understanding LLMs, Workflows, and Agents

AI Buzzwords Demystified If you’ve been following AI developments lately, you’ve probably encountered terms like LLMs, RAG, ReAct, and AI Agents. While these technologies are transforming how we interact with AI, the terminology can be overwhelming. In this post, I’ll break down these concepts into digestible explanations with practical examples. Let’s start with the foundation and progressively build up to more complex systems. Large Language Models (LLMs): The Foundation At the core of today’s AI revolution are Large Language Models (LLMs). Popular applications such as ChatGPT and Claude are built on top of these powerful models. They excel at generating and manipulating text based on the prompts we provide.

  • AI Concepts
  • ChatGPT
  • Claude
  • LLM Interaction
  • AI Workflows
  • Language Models
  • RAG
  • AI Agents
Saturday, May 24, 2025 Read
  • ««
  • «
  • 1
  • 2
  • 3
  • 4
  • 5
  • »
  • »»