Ghost in the data
  • Home
  • About
  • Posts
  • Start Here
  • Topics
  • Resources
  • RSS
  • Tags
  • 2026 Trends
  • AI
  • AI Agents
  • AI Bubble
  • AI Business Applications
  • AI Communication
  • AI Concepts
  • AI Ethics
  • AI Productivity
  • AI Prompting
  • AI Tools
  • AI Workflows
  • Analytics
  • Analytics Engineering
  • Anonymisation
  • Apache Airflow
  • Apache Iceberg
  • API Integration
  • Architecture
  • Astronomer
  • Athena
  • Automation
  • AVRO
  • AWS
  • AWS Glue
  • Banking
  • Bedrock Edition
  • Best Practices
  • Big Data
  • Blue-Green Deployment
  • Business Case
  • Business Value
  • Career Development
  • Change Management
  • Chapter Lead
  • ChatGPT
  • CI/CD
  • Claude
  • Claude Code
  • Cloud Computing
  • Cloud Gaming
  • Code Comments
  • Code Review
  • Communication
  • Conceptual Data Modelling
  • Continuous Learning
  • Cost Optimisation
  • CSV
  • Customer Experience
  • Dagster
  • Data Architecture
  • Data Contracts
  • Data Democratisation
  • Data Engineering
  • Data Ethics
  • Data Freshness
  • Data Governance
  • Data Impact
  • Data Ingestion
  • Data Minimisation
  • Data Modelling
  • Data Observability
  • Data Ownership
  • Data Pipelines
  • Data Platform
  • Data Privacy
  • Data Quality
  • Data Reliability
  • Data Solutions
  • Data System Resilience
  • Data Teams
  • Data Testing
  • Data Transformation
  • Data Validation
  • Data Vault
  • Data Warehouse
  • Database Design
  • Datafold
  • Dbt
  • Decision Making
  • Delta Lake
  • Development
  • Development Tools
  • DevOps
  • Dimensional Modelling
  • Documentation
  • DuckDB
  • Emotional Intelligence
  • Empathetic Design
  • Employee Engagement
  • Employee Experience
  • Employee Productivity
  • Enterprise
  • Estimation
  • ETL
  • Family Gaming
  • Feedback
  • File Formats
  • Financial Crisis
  • FinOps
  • Fivetran
  • Frameworks
  • Friendship
  • Future of Work
  • GCP
  • GDPR
  • Git
  • GitBash
  • GitHub
  • GitHub Actions
  • Grief
  • Hiring
  • Historical Load
  • Human Connection
  • Idempotency
  • Incentives
  • Incident Response
  • Industry Trends
  • Innovation
  • Inspirational Quote
  • Intergroup Conflict
  • Interviews
  • Journal
  • Journaling Techniques
  • JSON
  • Junior Engineer
  • Kimball
  • Knowledge Management
  • Lakehouse
  • Lambda
  • Leadership
  • Legacy Systems
  • Life
  • LLM
  • Loss
  • MacOS
  • Mental Health
  • Mentorship
  • Mindfulness Practices
  • Minecraft
  • Modern Data Stack
  • Moral Development
  • Motivation
  • MWAA
  • Obsidian
  • Onboarding
  • One-on-One Meetings
  • Open Source
  • OpenFlow
  • ORC
  • Orchestration
  • Organizational Culture
  • Parquet
  • Pattern Bank
  • Performance Optimisation
  • Performance Reviews
  • Personal
  • Personal Finance
  • Personal Growth
  • Philosophy
  • Pipeline
  • Pipeline Architecture
  • Pipeline Design
  • Platform Strategy
  • PostgreSQL
  • Pragmatism
  • Prefect
  • Problem Solving
  • Production Issues
  • Productivity
  • Professional Relationships
  • Project Management
  • Promotion
  • Psychological Safety
  • Public Speaking
  • Python
  • RAG
  • Refactoring
  • Remote Work
  • Reputation
  • Requirements Gathering
  • Retention Policies
  • RFC 4180
  • Risk Management
  • Robbers Cave Experiment
  • ROI
  • Roleplaying
  • S3
  • Salesforce
  • SCD
  • Schema Drift
  • Schema Evolution
  • Self Evaluation
  • Self-Awareness
  • Self-Reflection
  • Semantic Layer
  • Server Setup
  • Service Design
  • Shadow IT
  • Skills
  • Snowflake
  • Soft Skills
  • Solution Design
  • SQL
  • SQL Standards
  • SSH
  • SSH Keys
  • Staff Engineer
  • Stakeholder Management
  • Star Schema
  • Starburst
  • Step Functions
  • Strangler Fig
  • Strategy
  • Streaming
  • Strengths
  • Success Habits
  • Tabletop RPG
  • Team Building
  • Team Collaboration
  • Team Culture
  • Team Enablement
  • Technical Assessment
  • Technical Debt
  • Technical Strategy
  • Testing
  • Tools and Access
  • Trino
  • Trust
  • User Experience
  • UV
  • Value Creation
  • Vector Databases
  • Vector Search
  • Virtual Environments
  • Visualisation
  • VSCode
  • Wellbeing
  • Windows
  • Workplace Communication
  • Write-Audit-Publish
  • Zsh
Hero Image
Why Your Ideas Die in Planning Meetings

The silence that kills good ideas One morning, I sat in yet another meeting where we just spent two weeks backfilling a table then we found it was riddled with issues with the data. Even if we resolve the issue, it would then be another 2 weeks to backfill the data, there has to be a better way. “So, what do we think? Give me your best ideas for tackling this.”

  • Team Culture
  • Team Collaboration
  • Psychological Safety
  • Innovation
  • Change Management
  • Leadership
  • Data Teams
Wednesday, January 7, 2026 Read
Hero Image
Your First 90 Days as a Data Engineer: A Strategic Guide

Introduction Landing your first data engineering role—or starting at a new company—is both exhilarating and daunting. After navigating multiple interviews and accepting an offer, you’ve finally arrived at your desk with a new laptop and company swag (if your lucky). Even now, after solving countless problems ranging from minor bugs to enterprise-scale data challenges, I still occasionally feel that flutter of uncertainty in my stomach, when starting a new role. What if I don’t know what I’m doing? What if I make a mistake?

  • Onboarding
  • Career Development
  • Team Collaboration
  • Team Culture
Sunday, February 23, 2025 Read
Hero Image
Enhance Workplace Relationships

Introduction: A Tale of Two Tribes and the Modern Workplace Imagine a serene summer camp in the rugged heart of Robbers Cave State Park, Oklahoma, 1954. Two groups of boys, unaware of each other’s existence, are about to embark on an adventure that mirrors the timeless tale of rivalry and reconciliation—a story that still resonates in the corridors of contemporary workplaces. The Robbers Cave Experiment, conducted by social psychologist Muzafer Sherif, is not just a fascinating study on group dynamics; it’s a blueprint for understanding and enhancing cooperation in any setting where diverse minds meet. This experiment beautifully illustrates how perceived differences can dissolve into unity, given the right conditions and shared objectives.

  • Robbers Cave Experiment
  • Professional Relationships
  • Team Collaboration
  • Intergroup Conflict
  • Employee Productivity
Saturday, April 6, 2024 Read
Hero Image
Mastering Data Engineering: Insights and Best Practices

Introduction I have been working with Data for a bit over 17 years now, I have seen it evolve from its nascent stages to a cornerstone of the tech industry. The journey has been nothing short of revolutionary, impacting businesses and society at large. The evolution and the role of a data engineer have expanded, requiring not just technical skills, but a deep understanding of business, security, and the human element within technology.

  • Team Culture
  • Continuous Learning
  • Data Quality
  • Career Development
  • Data Pipelines
  • Data System Resilience
  • Team Collaboration
Saturday, March 30, 2024 Read