Ghost in the data
  • Home
  • About
  • Posts
  • Start Here
  • Topics
  • Resources
  • RSS
  • Tags
  • 2026 Trends
  • AI
  • AI Agents
  • AI Bubble
  • AI Business Applications
  • AI Communication
  • AI Concepts
  • AI Ethics
  • AI Productivity
  • AI Prompting
  • AI Tools
  • AI Workflows
  • Analytics
  • Analytics Engineering
  • Anonymisation
  • Apache Airflow
  • Apache Iceberg
  • API Integration
  • Architecture
  • Astronomer
  • Athena
  • Automation
  • AVRO
  • AWS
  • AWS Glue
  • Banking
  • Bedrock Edition
  • Best Practices
  • Big Data
  • Blue-Green Deployment
  • Business Case
  • Business Value
  • Career Development
  • Change Management
  • Chapter Lead
  • ChatGPT
  • CI/CD
  • Claude
  • Claude Code
  • Cloud Computing
  • Cloud Gaming
  • Code Comments
  • Code Review
  • Communication
  • Conceptual Data Modelling
  • Continuous Learning
  • Cost Optimisation
  • CSV
  • Customer Experience
  • Dagster
  • Data Architecture
  • Data Contracts
  • Data Democratisation
  • Data Engineering
  • Data Ethics
  • Data Freshness
  • Data Governance
  • Data Impact
  • Data Ingestion
  • Data Minimisation
  • Data Modelling
  • Data Observability
  • Data Ownership
  • Data Pipelines
  • Data Platform
  • Data Privacy
  • Data Quality
  • Data Reliability
  • Data Solutions
  • Data System Resilience
  • Data Teams
  • Data Testing
  • Data Transformation
  • Data Validation
  • Data Vault
  • Data Warehouse
  • Database Design
  • Datafold
  • Dbt
  • Decision Making
  • Delta Lake
  • Development
  • Development Tools
  • DevOps
  • Dimensional Modelling
  • Documentation
  • DuckDB
  • Emotional Intelligence
  • Empathetic Design
  • Employee Engagement
  • Employee Experience
  • Employee Productivity
  • Enterprise
  • Estimation
  • ETL
  • Family Gaming
  • Feedback
  • File Formats
  • Financial Crisis
  • FinOps
  • Fivetran
  • Frameworks
  • Friendship
  • Future of Work
  • GCP
  • GDPR
  • Git
  • GitBash
  • GitHub
  • GitHub Actions
  • Grief
  • Hiring
  • Historical Load
  • Human Connection
  • Idempotency
  • Incentives
  • Incident Response
  • Industry Trends
  • Innovation
  • Inspirational Quote
  • Intergroup Conflict
  • Interviews
  • Journal
  • Journaling Techniques
  • JSON
  • Junior Engineer
  • Kimball
  • Knowledge Management
  • Lakehouse
  • Lambda
  • Leadership
  • Legacy Systems
  • Life
  • LLM
  • Loss
  • MacOS
  • Mental Health
  • Mentorship
  • Mindfulness Practices
  • Minecraft
  • Modern Data Stack
  • Moral Development
  • Motivation
  • MWAA
  • Obsidian
  • Onboarding
  • One-on-One Meetings
  • Open Source
  • OpenFlow
  • ORC
  • Orchestration
  • Organizational Culture
  • Parquet
  • Pattern Bank
  • Performance Optimisation
  • Performance Reviews
  • Personal
  • Personal Finance
  • Personal Growth
  • Philosophy
  • Pipeline
  • Pipeline Architecture
  • Pipeline Design
  • Platform Strategy
  • PostgreSQL
  • Pragmatism
  • Prefect
  • Problem Solving
  • Production Issues
  • Productivity
  • Professional Relationships
  • Project Management
  • Promotion
  • Psychological Safety
  • Public Speaking
  • Python
  • RAG
  • Refactoring
  • Remote Work
  • Reputation
  • Requirements Gathering
  • Retention Policies
  • RFC 4180
  • Risk Management
  • Robbers Cave Experiment
  • ROI
  • Roleplaying
  • S3
  • Salesforce
  • SCD
  • Schema Drift
  • Schema Evolution
  • Self Evaluation
  • Self-Awareness
  • Self-Reflection
  • Semantic Layer
  • Server Setup
  • Service Design
  • Shadow IT
  • Skills
  • Snowflake
  • Soft Skills
  • Solution Design
  • SQL
  • SQL Standards
  • SSH
  • SSH Keys
  • Staff Engineer
  • Stakeholder Management
  • Star Schema
  • Starburst
  • Step Functions
  • Strangler Fig
  • Strategy
  • Streaming
  • Strengths
  • Success Habits
  • Tabletop RPG
  • Team Building
  • Team Collaboration
  • Team Culture
  • Team Enablement
  • Technical Assessment
  • Technical Debt
  • Technical Strategy
  • Testing
  • Tools and Access
  • Trino
  • Trust
  • User Experience
  • UV
  • Value Creation
  • Vector Databases
  • Vector Search
  • Virtual Environments
  • Visualisation
  • VSCode
  • Wellbeing
  • Windows
  • Workplace Communication
  • Write-Audit-Publish
  • Zsh
Hero Image
Five Worlds of Data Engineering

You watch a conference talk about implementing data contracts, and nobody mentions that the advice assumes you have multiple teams producing data — which you don’t. You read a post declaring “if you’re still using stored procedures in 2026, you’re doing it wrong,” and the comments erupt. Half the people are nodding along. Half are furious. Both sides are right. They’re just living in different worlds and don’t realise it.

  • Data Engineering
  • Modern Data Stack
  • Enterprise
  • Data Architecture
  • Career Development
  • Leadership
Saturday, May 2, 2026 Read
Hero Image
Leveraging LLMs for Business Impact: Part 1 - Theory and Foundations

Introduction In today’s rapidly evolving technological landscape, Large Language Models (LLMs) have emerged as transformative tools with the potential to revolutionize business operations across industries. While the hype around these technologies is intense, understanding their practical applications and underlying mechanisms is crucial for organisations seeking to leverage them effectively. This two-part series aims to demystify LLMs and their associated technologies, starting with the theoretical foundations in Part 1, followed by a hands-on implementation guide using AWS services in Part 2.

  • LLM
  • RAG
  • Vector Databases
  • AI Business Applications
  • Data Architecture
Friday, March 7, 2025 Read