Ghost in the data
  • Home
  • About
  • Posts
  • Start Here
  • Topics
  • Resources
  • RSS
  • Tags
  • 2026 Trends
  • AI
  • AI Agents
  • AI Bubble
  • AI Business Applications
  • AI Communication
  • AI Concepts
  • AI Ethics
  • AI Productivity
  • AI Prompting
  • AI Tools
  • AI Workflows
  • Analytics
  • Analytics Engineering
  • Anonymisation
  • Apache Airflow
  • Apache Iceberg
  • API Integration
  • Architecture
  • Astronomer
  • Athena
  • Automation
  • AVRO
  • AWS
  • AWS Glue
  • Banking
  • Bedrock Edition
  • Best Practices
  • Big Data
  • Blue-Green Deployment
  • Business Case
  • Business Value
  • Career Development
  • Change Management
  • Chapter Lead
  • ChatGPT
  • CI/CD
  • Claude
  • Claude Code
  • Cloud Computing
  • Cloud Gaming
  • Code Comments
  • Code Review
  • Communication
  • Conceptual Data Modelling
  • Continuous Learning
  • Cost Optimisation
  • CSV
  • Customer Experience
  • Dagster
  • Data Architecture
  • Data Contracts
  • Data Democratisation
  • Data Engineering
  • Data Ethics
  • Data Freshness
  • Data Governance
  • Data Impact
  • Data Ingestion
  • Data Minimisation
  • Data Modelling
  • Data Observability
  • Data Ownership
  • Data Pipelines
  • Data Platform
  • Data Privacy
  • Data Quality
  • Data Reliability
  • Data Solutions
  • Data System Resilience
  • Data Teams
  • Data Testing
  • Data Transformation
  • Data Validation
  • Data Vault
  • Data Warehouse
  • Database Design
  • Datafold
  • Dbt
  • Decision Making
  • Delta Lake
  • Development
  • Development Tools
  • DevOps
  • Dimensional Modelling
  • Documentation
  • DuckDB
  • Emotional Intelligence
  • Empathetic Design
  • Employee Engagement
  • Employee Experience
  • Employee Productivity
  • Engineering Leadership
  • Enterprise
  • Estimation
  • ETL
  • Family Gaming
  • Feedback
  • File Formats
  • Financial Crisis
  • FinOps
  • Fivetran
  • Frameworks
  • Friendship
  • Future of Work
  • GCP
  • GDPR
  • Git
  • GitBash
  • GitHub
  • GitHub Actions
  • Grief
  • Hiring
  • Historical Load
  • Human Connection
  • Idempotency
  • Incentives
  • Incident Response
  • Industry Trends
  • Innovation
  • Inspirational Quote
  • Intergroup Conflict
  • Interviews
  • Journal
  • Journaling Techniques
  • JSON
  • Junior Engineer
  • Kimball
  • Knowledge Management
  • Lakehouse
  • Lambda
  • Leadership
  • Legacy Systems
  • Life
  • LLM
  • Loss
  • MacOS
  • Mental Health
  • Mentorship
  • Mindfulness Practices
  • Minecraft
  • Modern Data Stack
  • Moral Development
  • Motivation
  • MWAA
  • Obsidian
  • Onboarding
  • One-on-One Meetings
  • Open Source
  • OpenFlow
  • ORC
  • Orchestration
  • Organizational Culture
  • Parquet
  • Pattern Bank
  • Performance Optimisation
  • Performance Reviews
  • Personal
  • Personal Finance
  • Personal Growth
  • Philosophy
  • Pipeline
  • Pipeline Architecture
  • Pipeline Design
  • Platform Strategy
  • PostgreSQL
  • Pragmatism
  • Prefect
  • Problem Solving
  • Production Issues
  • Productivity
  • Professional Relationships
  • Project Management
  • Promotion
  • Psychological Safety
  • Public Speaking
  • Python
  • RAG
  • Refactoring
  • Remote Work
  • Reputation
  • Requirements Gathering
  • Retention Policies
  • RFC 4180
  • Risk Management
  • Robbers Cave Experiment
  • ROI
  • Roleplaying
  • S3
  • Salesforce
  • SCD
  • Schema Drift
  • Schema Evolution
  • Self Evaluation
  • Self-Awareness
  • Self-Reflection
  • Semantic Layer
  • Server Setup
  • Service Design
  • Shadow IT
  • Skills
  • Snowflake
  • Soft Skills
  • Solution Design
  • SQL
  • SQL Standards
  • SSH
  • SSH Keys
  • Staff Engineer
  • Stakeholder Management
  • Star Schema
  • Starburst
  • Step Functions
  • Strangler Fig
  • Strategy
  • Streaming
  • Strengths
  • Success Habits
  • Supervisory Control
  • Tabletop RPG
  • Team Building
  • Team Collaboration
  • Team Culture
  • Team Enablement
  • Technical Assessment
  • Technical Debt
  • Technical Strategy
  • Testing
  • Tools and Access
  • Trino
  • Trust
  • User Experience
  • UV
  • Value Creation
  • Vector Databases
  • Vector Search
  • Virtual Environments
  • Visualisation
  • VSCode
  • Wellbeing
  • Windows
  • Workplace Communication
  • Write-Audit-Publish
  • Zsh
Hero Image
Discover our best selves

Introduction Here’s what nobody tells you about: your biggest blind spot isn’t your technical weaknesses—it’s your strengths. That sounds weird doesn’t it. Most of us can recite our shortcomings on command, yet we struggle to articulate what we’re genuinely good at. This isn’t just modesty. It’s a fundamental quirk of human psychology that keeps us from reaching our full potential. The Reflected Best Self Portrait exercise, developed by researchers at the University of Michigan’s Ross School of Business, flips this script entirely. Instead of the usual deficit-focused approach (“here’s what you need to fix”), it asks a radical question: what if we built our careers around who we are when we’re at our absolute best? For data professionals navigating an industry that’s constantly evolving—where yesterday’s cutting-edge tool becomes today’s legacy system—this approach isn’t just refreshing. It’s essential.

  • Strengths
  • Self-Awareness
  • Career Development
  • Leadership
Saturday, November 1, 2025 Read
Hero Image
Mastering One-on-One Meetings: Building Trust and Driving Growth

Introduction Have you ever felt that slight relief when your manager cancels your 1:1 meeting? Early in my career as a data professional, I viewed 1:1s as just another checkbox on my calendar—often treating them like mini-standups where I’d rattle off project updates before awkwardly waiting for the meeting to end. Looking back, I realise how much potential growth I left on the table. As I progressed from an individual contributor to leading a team, I’ve learned that 1:1 meetings aren’t administrative burdens—they’re golden opportunities for trust-building, relationship development, and strategic alignment that many of us simply don’t know how to leverage.

  • One-on-One Meetings
  • Leadership
  • Emotional Intelligence
  • Trust
  • Workplace Communication
  • Career Development
  • Feedback
  • Mentorship
Saturday, March 22, 2025 Read
Hero Image
From Senior to Staff: Navigating the Data Engineering Leadership Path

Introduction: The Critical Inflection Point The transition from Senior to Staff Engineer represents a pivotal moment in any technical career path. It’s the point where your impact extends beyond your code and transforms into something much more profound – true technical leadership. While this shift can feel daunting, it also opens doors to some of the most rewarding work of your career. The beautiful thing about the engineering career ladder is that it uniquely allows for advancement without stepping away from the technical work that many of us love.

  • Staff Engineer
  • Career Development
  • Leadership
  • Chapter Lead
  • Promotion
Sunday, March 2, 2025 Read
Hero Image
Your First 90 Days as a Data Engineer: A Strategic Guide

Introduction Landing your first data engineering role—or starting at a new company—is both exhilarating and daunting. After navigating multiple interviews and accepting an offer, you’ve finally arrived at your desk with a new laptop and company swag (if your lucky). Even now, after solving countless problems ranging from minor bugs to enterprise-scale data challenges, I still occasionally feel that flutter of uncertainty in my stomach, when starting a new role. What if I don’t know what I’m doing? What if I make a mistake?

  • Onboarding
  • Career Development
  • Team Collaboration
  • Team Culture
Sunday, February 23, 2025 Read
Hero Image
Mastering Data Interviews: A Comprehensive Guide

Introduction After nearly two decades in the data engineering field, I’ve sat on both sides of the interview table countless times. Whether you’re a seasoned professional looking to change roles or a newcomer trying to break into the field, the interview process for data engineering positions can be both challenging and mysterious. There’s often uncertainty about what questions you’ll face, what skills you need to demonstrate, and what interviewers are really looking for beneath the surface.

  • Interviews
  • Technical Assessment
  • Career Development
  • SQL
  • Data Modelling
  • Problem Solving
Saturday, February 22, 2025 Read
Hero Image
Mastering Data Engineering: Insights and Best Practices

Introduction I have been working with Data for a bit over 17 years now, I have seen it evolve from its nascent stages to a cornerstone of the tech industry. The journey has been nothing short of revolutionary, impacting businesses and society at large. The evolution and the role of a data engineer have expanded, requiring not just technical skills, but a deep understanding of business, security, and the human element within technology.

  • Team Culture
  • Continuous Learning
  • Data Quality
  • Career Development
  • Data Pipelines
  • Data System Resilience
  • Team Collaboration
Saturday, March 30, 2024 Read
Hero Image
2023 - Books that are worth you time?

Introduction As a Data Engineer, it’s crucial to constantly improve your skills and knowledge to stay ahead of the curve. Whether it’s working with large data sets, building efficient data pipelines, or collaborating with a team, there are many different aspects to consider. To help you succeed, I’ve put together a list of books that cover a range of topics, from culture and team building to Python and SQL. Each of the books I’ve selected offers valuable insights and practical advice to help you become a better Data Engineer. Whether you’re looking to strengthen your coding skills, learn how to effectively communicate with your team, or improve your organisation’s data processes, there’s something here for everyone. So, without further ado, let’s dive into the books that can help you take your skills to the next level.

  • Career Development
Sunday, March 5, 2023 Read
Hero Image
Onboarding a data team

Introduction Onboarding is so important to give a great impression, but also setting up the scaffolding of what a new employee would expect the culture to be like at a company. Needless to say, it’s not normally a great experience, it normally starts with spending the first two weeks getting access to systems and tools. Then once you have access, especially in the remote working environment - sometimes you get only a brief introduction with your manager, and get a small glimpse as to what you are to be working on. This common scenario, makes people feel less connected to the team, and no excitement or passion for the work - they feel undervalued, which isn’t a great start from day one or two.

  • Career Development
  • Onboarding
  • Employee Engagement
  • Team Culture
  • Tools and Access
  • Remote Work
Sunday, January 29, 2023 Read
  • ««
  • «
  • 1
  • 2
  • »
  • »»