Ghost in the data
  • Home
  • About
  • Posts
  • Tags
  • AI
  • AI Agents
  • AI Business Applications
  • AI Communication
  • AI Concepts
  • AI Productivity
  • AI Prompting
  • AI Workflows
  • Airflow
  • Apache Airflow
  • Apache Iceberg
  • Automation
  • AVRO
  • Bedrock Edition
  • Blue-Green Deployment
  • Business Value
  • Career Advice
  • Career Growth
  • Chapter Lead
  • ChatGPT
  • CI/CD
  • Claude
  • Cloud Gaming
  • Code Review
  • Communication
  • ConceptualDataModeling
  • Continuous Learning
  • CSV
  • Culture
  • Data Architecture
  • Data Culture
  • Data Engineering
  • Data Governance
  • Data Impact
  • Data Leadership
  • Data Modeling
  • Data Modelling
  • Data Pipeline
  • Data Quality
  • Data Reliability
  • Data Solutions
  • Data System Resilience
  • Data Testing
  • Data Transformation
  • Data Vault
  • Data Warehouse
  • Data Warehouse Architecture
  • Database Design
  • DataEngineering
  • DataPipelines
  • DBT
  • Delta-Lake
  • Development
  • Development Tools
  • Emotional Intelligence
  • EmpatheticDesign
  • Employee Engagement
  • Employee Productivity
  • Engineering Career
  • ETL
  • ETL Pipeline
  • Family Gaming
  • Feedback
  • File Formats
  • GCP
  • Git
  • GitBash
  • Github
  • GitHub Actions
  • Hiring Strategies
  • Incident Response
  • Industry Trends
  • Inspirational Quote
  • Intergroup Conflict
  • Interviews
  • Journal
  • Journaling Techniques
  • JSON
  • Language Models
  • LLM
  • LLM Interaction
  • MacOS
  • Management
  • Mentorship
  • Mindfulness Practices
  • Minecraft
  • Onboarding
  • One-on-One Meetings
  • ORC
  • Parquet
  • Performance Optimization
  • Personal Growth
  • Pipeline
  • PostegreSQL
  • Problem Solving
  • Professional Development
  • Professional Growth
  • Promotion
  • Python
  • RAG
  • Recruitment
  • Remote Work
  • RequirementGathering
  • Risk Management
  • Robbers Cave Experiment
  • Roleplaying
  • Schema Evolution
  • Self-Reflection
  • Server Setup
  • SQL
  • SQL Standards
  • SSH
  • SSH Keys
  • Staff Engineer
  • Stakeholder Engagement
  • Stakeholder Management
  • StakeholderManagement
  • Star Schema
  • Success Habits
  • Talent Acquisition
  • Team Collaboration
  • Team Enablement
  • Technical Assessment
  • Technical Leadership
  • Tools and Access
  • Trust Building
  • UV
  • UV Package Manager
  • Value Creation
  • Vector Databases
  • Virtual Environments
  • Visualization
  • VSCode
  • WAP Pattern
  • Windows
  • Workplace Communication
  • Workplace Relationships
  • Write-Audit-Publish
  • Zsh
Hero Image
Setting Up Your Data Engineering Environment on Windows

Introduction Setting up a development environment for data engineering on Windows requires some specific considerations that differ from Unix-based systems. This guide will walk you through creating a robust Python development environment on Windows, with detailed explanations of each component and why it’s important. Clean Slate: Removing Existing Python Installations Before starting, it’s important to remove any existing Python installations to avoid conflicts: Open Windows Settings > Apps > Apps & Features Search for “Python” Uninstall any Python versions listed Also check and remove Python from these locations:

  • Python
  • DBT
  • Windows
  • UV Package Manager
  • VSCode
Monday, February 3, 2025 Read
Hero Image
Setting Up Your Data Engineering Environment on MacOS

Introduction Setting up a development environment for data engineering on MacOS requires careful consideration of package management, Python version control, and tool configuration. This guide will walk you through the process, explaining not just how to set up these tools, but why each component is important. Clean Slate: Removing Existing Python Installations Before we begin, it’s important to ensure we’re starting with a clean slate. Multiple Python installations can cause confusion and conflicts. Let’s remove any existing Python installations:

  • Python
  • DBT
  • MacOS
  • UV Package Manager
  • VSCode
Sunday, February 2, 2025 Read