AWS MWAA: The Practitioner's Unvarnished Guide

I remember the first time I sat down properly with the Step Functions architecture our team inherited. Someone had designed it thoughtfully — real engineering thought had gone into it when it was built.

But what I was looking at in the console was a map of state machines and Lambda functions that had grown well past the point where anyone could hold it in their head.

The first production issue we had to debug on that setup took most of a working day. Not because the problem was complicated — it wasn’t — but because finding it meant jumping between CloudWatch log groups, correlating timestamps, building a mental picture of which Lambda had received what input and where the chain had broken. Every time I thought I had it, another log group. Another timestamp comparison. Another dead end.

That’s when I suggested to the team that we test out Airflow. I remember the first time something failed on the new setup. I clicked into the task, read the log attached to that specific run, saw the error, fixed it, and restarted the DAG. Four minutes. The CDK stacks for the old architecture ran to thousands of lines. The equivalent DAG was fifty to two hundred. Same logic. Made legible.

The difference wasn’t performance — Lambda and Airflow are comparable where it counts. It was visibility. One place to look. That’s what orchestration is actually about: not execution speed, not infrastructure elegance, but whether you can understand what’s happening in your data platform — at 9am in a post-mortem, or at 2am when something breaks and you need to know why.

AWS MWAA — Managed Workflows for Apache Airflow — is Amazon’s answer to this problem. And it’s genuinely good at some things. But the gap between what the marketing page promises and what you’ll actually experience in production is wider than most managed services I’ve worked with. The $365/month sticker price? That’s just the beginning. The “managed” part? It manages less than you’d think.

This guide covers everything I wish someone had told me before committing to MWAA — the real architecture, the actual costs (including the ones AWS conveniently leaves off the pricing page), the operational gotchas that’ll bite you in production, and honest head-to-head comparisons with every serious alternative: Astronomer Astro, Dagster Cloud, Prefect Cloud, Step Functions, and the self-hosted approach that might be smarter than you’d expect.

How MWAA Actually Works Under the Hood

MWAA runs on a dual-VPC architecture that’s important to understand because it explains most of the service’s constraints.

AWS provisions Fargate containers for your schedulers, workers, and web server inside an AWS-managed ECS cluster. These containers connect to your VPC through Elastic Network Interfaces injected into your private subnets. A single-tenant Aurora PostgreSQL instance stores the Airflow metadata — fully managed by AWS, but completely inaccessible to you. No connection string. No direct SQL queries. No custom dashboards built against the metadata database. If you’ve ever relied on querying task_instance or dag_run tables directly for custom monitoring, MWAA takes that away.

The networking requirements are specific and unforgiving. You need two private subnets in different Availability Zones, a NAT gateway (or VPC endpoints) for internet access, and a self-referencing security group. The VPC you choose at creation is permanent — change your mind later and you’re building a new environment from scratch. Choose private webserver mode (which most production teams do for security), and accessing the Airflow UI requires a VPN or bastion host setup.

DAGs live in an S3 bucket with versioning enabled. MWAA syncs files to the workers every 30 seconds, though new DAG files take roughly five minutes to be recognised (controlled by dag_dir_list_interval). Updates to requirements.txt or plugins.zip trigger a full environment update — and that means 20 to 40 minutes of waiting while MWAA reprovisions everything. More on that particular joy later.

A few setup details that’ll save you time: the S3 bucket must be in the same region as your MWAA environment and must have block public access enabled. Your dags/ folder sits at the root of the bucket, with requirements.txt alongside it (not inside the dags/ folder). The plugins.zip file follows a strict structure — if your custom operator lives at plugins/operators/my_operator.py, the zip must mirror that path exactly. Get the structure wrong and the import will silently fail with no useful error message until you dig into the DAGProcessing logs.

For IAM, MWAA needs an execution role with access to your S3 bucket, CloudWatch Logs, SQS (for the Celery broker), and whatever AWS services your DAGs interact with — Glue, EMR, Redshift, Lambda, whatever. The principle of least privilege matters here, but AWS’s own documentation starts with fairly broad permissions and leaves tightening to you. In practice, most teams start broad and restrict after stabilising.

Environment classes span six tiers since the 2024 additions, from micro through 2xlarge:

The micro class (November 2024) is a genuine game-changer for dev/test — at roughly $37/month, you can finally have a development MWAA environment without the $365+ price floor. The catch: it combines the scheduler and worker into a single container with no autoscaling. Fine for testing DAGs, not for production.

Worker autoscaling follows a straightforward formula: (running + queued tasks) / (tasks per worker) = required workers. Scale-down triggers after running and queued tasks hit zero for more than two minutes, and takes two to five minutes to complete. A significant autoscaling bug — where tasks could be assigned to workers being decommissioned — persisted until May 2025. If you’re running a version from before that fix, you’ll want to upgrade.

The True Cost of MWAA (It’s Not $365 a Month)

The AWS pricing page says mw1.small costs $0.49/hr, which works out to $364.56/month. That number is misleading. Every production MWAA deployment carries infrastructure costs that the headline figure conveniently excludes.

NAT gateways are the number one hidden cost. The standard architecture requires two NAT gateways (one per AZ), costing $0.045/hr each — that’s $65.70/month before a single byte of data moves through them. Add $0.045/GB for data processing. One team documented a $12K monthly bill where 87% was unused NAT gateway costs from misconfigured routing. This is not an MWAA-specific problem, but MWAA forces you into it because of the VPC requirement.

CloudWatch metrics are the second surprise. MWAA auto-publishes per-DAG and per-task metrics, and one practitioner documented $272/month for custom metrics alone — 907 metrics at $0.30 per metric. Unless you actively restrict what gets published using metrics.statsd_allow_list, you’ll see this line item growing steadily as you add more DAGs.

CloudWatch Logs at INFO level with active workflows adds $50–300/month in ingestion and storage charges, depending on how chatty your DAGs are.

Here’s what a small team running a production mw1.small environment actually pays:

Scale up to mw1.large with additional workers running six hours a day, extra schedulers, and full logging, and you’re looking at $2,150–2,300/month all-in. AWS’s own pricing example for a comparable setup quotes $1,047/month — but conveniently excludes NAT, CloudWatch, and networking overhead.

Here’s that breakdown for a heavier workload — say a mid-size data team running 200+ DAGs with burst periods:

The range depends heavily on how well you control CloudWatch metric sprawl and how long your burst periods actually last. The point is: you need to model the real infrastructure cost, not just the MWAA line item.

Cost optimisation levers do exist. Replace NAT gateways with VPC endpoints — the S3 Gateway Endpoint is free, and interface endpoints process data at $0.01/GB versus NAT’s $0.045/GB. Restrict CloudWatch metrics with metrics.statsd_allow_list = scheduler,executor or disable custom metrics entirely. Set log retention to 30–90 days instead of indefinite. For dev/test environments, implement pause/resume schedules — one team reported 70% savings by running MWAA only during business hours.

The November 2025 launch of MWAA Serverless fundamentally changes the cost equation for intermittent workloads. At $0.08/hr per task (billed per-second, minimum one minute), 2,000 tasks averaging one to two minutes costs roughly $4/month — versus $365+ for the always-on equivalent. The trade-off is substantial though: no Airflow UI, YAML-based workflow definitions, and only 80-odd AWS operators supported (no PythonOperator, no BashOperator). It’s better understood as a different product targeting a different use case than as a cheaper MWAA.

The Gotchas

Practitioners consistently flag the same pain points, and understanding them before committing is worth more than any architecture diagram.

Dependency management is the highest-risk operational area. A bad requirements.txt can crash-loop your entire environment for hours. Not minutes — hours. The constraint statement is mandatory since Airflow 2.7.2, and omitting it risks catastrophic dependency conflicts. If pip install exceeds ten minutes, the Fargate task times out and rolls back — leaving your environment stuck in an update cycle. Certain packages can break MWAA’s CloudWatch logging by overriding the watchtower library. The only safe approach: always test with the aws-mwaa-local-runner Docker tool before deploying dependencies to production.

Environment update times frustrate everyone. Creating an environment takes 25–30 minutes. Updating requirements or plugins: 20–40 minutes. Version upgrades: up to two hours with the environment unavailable during the process. The May 2025 “graceful updates” feature helps — it replaces components without interrupting running tasks — but the wall-clock time for the update itself hasn’t changed meaningfully.

Observability is CloudWatch-only, and navigating it is painful. MWAA creates five separate CloudWatch log groups per environment: DAGProcessing, Scheduler, Task, WebServer, and Worker. Correlating a single pipeline failure requires jumping between log groups, and CloudWatch Logs Insights queries aren’t intuitive for Airflow-style troubleshooting. You can’t redirect StatsD metrics to external tools like Datadog or Prometheus — MWAA overrides the statsd_host configuration. External monitoring integration requires CloudWatch Logs subscriptions via Kinesis Data Firehose, adding both complexity and cost.

If you’ve spent time in the trenches of AWS logs — and maybe this is just me — it can be a genuine nightmare to piece together what happened across services. MWAA should improve this, and in some ways it does (the Airflow UI’s task logs are excellent). But the moment you need infrastructure-level debugging, you’re back in CloudWatch hell.

The 12-hour task execution limit is a hard wall. Tasks exceeding 12 hours are killed and get stuck in the SQS queue for another 12 hours due to SQS timeout settings. The CeleryExecutor-only constraint means no per-task Docker images and no pod-level resource isolation. If you need tasks that run longer than 12 hours, you’ll need to offload compute to ECS Fargate or Lambda, using MWAA purely for orchestration.

Secrets Manager integration works but generates surprising API volume. Without configuring connections_lookup_pattern, Airflow attempts to look up every connection in Secrets Manager — including non-existent ones. One practitioner reported 100,000+ error API calls per day per MWAA instance. Secrets stored in Secrets Manager don’t appear in the Airflow UI either, which only shows connections from its own backend.

Other gotchas worth noting: you can’t remove plugins.zip or requirements.txt once added (you can only point to empty files), the VPC can’t be changed after creation, frequent DAG updates to the S3 folder can break the MWAA installation, and if VPC endpoints are accidentally deleted the environment is broken and must be entirely recreated.

One more that catches people off guard: Airflow configuration overrides. MWAA lets you set most Airflow configuration options through the console or API, but several are explicitly blocked — statsd_host, statsd_port, broker_url, result_backend, and anything related to the metadata database connection. These are locked down because MWAA manages those components. If your Airflow experience includes tuning Celery broker settings or metadata database connection pools, you’ll need to adjust your expectations about what “managed” means here. It means AWS decides those values, and some of them (particularly the default Celery broker configuration) aren’t optimal for all workload patterns.

The practical impact: if you’ve tuned self-hosted Airflow for specific performance characteristics — aggressive task polling, custom broker acknowledgement settings, metadata database query timeouts — MWAA won’t let you replicate that tuning. For most teams this doesn’t matter. For teams at scale with specific performance requirements, it can be a dealbreaker.

Head-to-Head: Astronomer Astro

Astronomer’s Astro platform runs Apache Airflow underneath, so your DAGs, operators, and muscle memory all transfer. The differences are in the operational layer — and they’re significant.

Workers scale to zero. When no tasks are running, you pay nothing for compute. MWAA charges for the environment 24/7 regardless. Astro’s base pricing starts at $0.35/hr for the control plane, with worker costs at $0.13/hr that only accrue when tasks actually execute.

The developer experience gap is substantial. Astro CLI provides a full local Airflow environment with astro dev start that mirrors production exactly — same Airflow version, same Python version, same provider packages. MWAA has no equivalent. The aws-mwaa-local-runner Docker tool is helpful but doesn’t perfectly replicate the MWAA environment. DAG deployment via astro deploy takes 5–10 minutes through a proper CI/CD pipeline; MWAA’s S3 upload plus sync cycle is slower and less integrated into developer workflows.

Astro supports the KubernetesExecutor natively, meaning you get per-task Docker images, custom resource limits, and the thin orchestrator pattern described below — without managing Kubernetes yourself. MWAA is locked to CeleryExecutor with no workaround.

Campspot, which migrated from MWAA to Astro, reported that a critical nightly job went from over two hours to two to three minutes. Another company, Black Crow AI, recouped roughly 20% more engineering time after leaving MWAA, citing zombie task issues and the absence of Airflow-specific support.

Astro’s Observe feature provides built-in pipeline lineage, SLA tracking, and data quality monitoring — capabilities absent from MWAA. Support comes from actual Airflow core committers, versus MWAA’s generic AWS support channels.

The migration story from MWAA to Astro is worth paying attention to because it reveals what teams actually struggle with. Campspot completed their migration in a two-week sprint — their DAGs transferred with minimal modification because both platforms run Airflow. The performance improvement came not from different code, but from better infrastructure: KubernetesExecutor allowing parallel task execution in isolated pods versus CeleryExecutor bottlenecking through shared workers. Black Crow AI’s experience was similar — the DAGs were the same, but the operational layer (scaling, monitoring, deployment speed) was categorically better.

That said, Astronomer is a startup, not AWS. If your organisation’s procurement process requires a vendor that’s been around for 20+ years, has SOC 2 Type II (Astronomer does have this, to be fair), and won’t disappear if funding dries up — that’s a valid concern. Astronomer raised significant funding and is growing, but the risk calculus is different from buying from AWS.

The trade-offs: Astro’s dedicated clusters cost $2.40/hr ($1,782/month at the cluster level), and networking costs (NAT, PrivateLink) are passed through from your cloud provider. For teams deeply embedded in AWS wanting consolidated billing and native IAM integration without multi-cloud requirements, MWAA’s tight ecosystem coupling has genuine value. But if developer experience and operational flexibility are priorities, Astro is meaningfully ahead.

Head-to-Head: Dagster Cloud

Dagster represents the most fundamental philosophical departure from the Airflow model — and by extension, from MWAA.

Where Airflow (and MWAA) asks “which tasks ran successfully?”, Dagster asks “which data assets are fresh, and why?” The difference sounds academic until you’re debugging a pipeline at 7am and you want to know not just which task failed, but which downstream datasets are now stale and which business reports can’t be trusted.

Software-Defined Assets are Dagster’s core abstraction. Each asset declares its dependencies, its materialisation logic, and its freshness expectations. The framework automatically builds a dependency graph, tracks lineage at the column level, and can tell you in real time which assets are stale, which are being materialised, and which are healthy. This is observability that MWAA simply can’t match — you’d need to bolt on separate lineage tools, metadata platforms, and custom monitoring to approximate what Dagster provides natively.

The developer experience is excellent. dagster dev starts a local instance immediately without Docker. Branch deployments create isolated staging environments from pull requests — push a feature branch, get a dedicated Dagster instance for testing. dbt integration maps models directly into the asset graph, so your dbt transformations are first-class citizens alongside your Python pipelines.

Dagster+ pricing uses a credit-based model: one credit equals one asset materialisation. The Solo tier starts at $10/month, Starter at $100/month. For infrequent batch workloads under 30,000 materialisations per month, Dagster is dramatically cheaper than MWAA. But credits can scale unpredictably — one practitioner calculated that an 8-operation job running every five minutes consumed 69,120 credits per month, costing $2,464 at Solo rates. For high-frequency, multi-step jobs, MWAA’s predictable hourly pricing may actually be more economical.

The ecosystem trade-off is real. Airflow has 1,600+ operators and a community of 80,000+ organisations built over a decade. Dagster’s integration library is growing rapidly but isn’t as broad. If you need a provider for an obscure SaaS API or legacy system, Airflow almost certainly has one; Dagster might not. For teams starting fresh without Airflow baggage, Dagster’s asset-centric model is compelling. For teams with hundreds of existing DAGs, the migration cost is substantial.

There’s also a philosophical difference worth naming. Airflow thinks in terms of schedules and tasks — “run this DAG at 6am, execute these tasks in order.” Dagster thinks in terms of data freshness and assets — “these datasets should be no more than 2 hours stale, materialise them as needed.” Both models work, but they lead to fundamentally different approaches to monitoring, alerting, and debugging. If your team’s primary question is “did the 6am job run?” then Airflow’s model fits naturally. If your question is “is the executive dashboard showing current data, and if not, what’s stale?” then Dagster’s model is more direct.

The deployment model also differs meaningfully. Dagster+ offers both Serverless (fully managed, AWS-hosted) and Hybrid (Dagster manages the control plane, your infrastructure runs the compute). The Hybrid model gives you the thin orchestrator pattern with professional management of the scheduling layer — a compelling middle ground between fully managed and fully self-hosted.

Head-to-Head: Prefect Cloud

Prefect takes the most Python-native approach to orchestration. Add @flow and @task decorators to regular Python functions. No DSL, no custom operators, no XCom for passing data between tasks — just Python.

This matters most for teams where the data engineers are primarily Python developers who find Airflow’s operator model cumbersome. Dynamic workflows, event-driven triggers, and runtime conditional logic are all first-class features in Prefect. Need to loop over a list of files and process each one as a separate task? That’s a Python for-loop, not an Airflow dynamic task mapping exercise.

Prefect Cloud’s free Hobby tier (2 users, 5 deployments) lets small teams start at $0 — versus MWAA’s $365+ minimum. The Starter tier at $100/month includes bring-your-own-compute, meaning you still pay separately for the infrastructure where your flows actually run. Prefect Cloud is a control plane, not a compute platform — similar to the thin orchestrator pattern, but with Prefect managing the scheduling and monitoring layer instead of Airflow.

The honest limitation: Prefect’s community is a fraction of Airflow’s. Airflow pulls 30 million monthly downloads; Prefect sits around 1.8 million weekly. That gap means fewer Stack Overflow answers, fewer blog posts solving your exact problem, and fewer engineers who already know the tool when you’re hiring. For teams where hiring velocity matters, Airflow’s ubiquity is a genuine competitive advantage — and MWAA inherits that.

Where Prefect genuinely shines is the rapid-prototyping-to-production pipeline. You can convert an existing Python script into an orchestrated workflow by adding decorators — no rewriting into operator patterns, no separating logic from configuration, no learning a new abstraction layer. One engineering team compared this directly to their Airflow experience and found that Prefect flows took roughly half the time to develop and deploy. The trade-off is less structure — Airflow’s opinionated DAG model forces a discipline that Prefect’s flexibility doesn’t enforce.

Prefect Cloud’s Pro tier ($500/month) adds RBAC, audit logs, and custom retention policies. The Enterprise tier (custom pricing) adds SSO, dedicated infrastructure, and priority support. For context, an MWAA mw1.small environment costs $470–540/month all-in — roughly equivalent to Prefect Pro before you add your own compute infrastructure. The total cost comparison depends entirely on how much compute your flows need and where it runs.

One more thing worth noting: Prefect 2 (the current generation) was a ground-up rewrite that broke compatibility with Prefect 1. If you’re evaluating Prefect, you’re evaluating a framework that made the decision to start over once already. That took conviction, and the result is a significantly better product — but it’s worth understanding that the ecosystem is younger than the company’s founding date suggests.

Head-to-Head: AWS Step Functions

Step Functions is the right answer more often than most Airflow advocates want to admit.

For a 10-step workflow running once daily (~300 state transitions per month), Step Functions costs nothing within the free tier. MWAA’s minimum is $365+. Even at medium scale — 100 executions per day — costs stay under $1/month for standard workflows. The Distributed Map mode can process up to 10,000 parallel S3 objects; one AWS demo processed 560,000 CSV files in 100 seconds.

Step Functions also runs natively serverless with zero infrastructure management. No VPC, no NAT gateways, no requirements.txt crashes, no 20-minute environment updates. For teams building AWS-native event-driven architectures — S3 events triggering Glue jobs, Lambda processing, DynamoDB writes — Step Functions integrates with over 220 AWS services directly in the workflow definition.

The limitations are real though. The 256KB payload limit between states constrains data-heavy handoffs. JSON-based Amazon States Language for workflow definitions is verbose and hard to debug. There’s no built-in scheduling (you need EventBridge), no backfill capability, no equivalent of Airflow’s catchup=True for replaying historical runs. The 25,000-event history limit per standard execution bites data engineering workloads — though Express Workflows (5-minute maximum duration, $1 per million requests) eliminate that constraint for short-lived jobs.

For complex data pipelines with interdependencies, branching logic, retries, and backfill requirements, MWAA is genuinely the better tool. For linear workflows, event-driven processing, and microservice orchestration, Step Functions wins on cost, simplicity, and operational overhead.

Beyond Named Platforms: A Pattern Worth Considering

Beyond named platforms, there’s an architectural pattern that deserves consideration alongside them — one that doesn’t come with a vendor attached.

The idea is simple: run Airflow as a lightweight scheduler and UI on minimal infrastructure, and delegate all actual compute to Kubernetes pods or ECS tasks that spin up on demand.

In this model, your Airflow installation is deliberately “dumb.” A small EC2 instance or a modest EKS deployment runs just the webserver, scheduler, and metadata database. Workers don’t exist in the traditional sense. Instead, every task in your DAG uses the KubernetesExecutor (or EcsRunTaskOperator) to launch a purpose-built container that does the actual work, then terminates.

Here’s why this matters:

Per-task isolation. Each task runs in its own container with its own Docker image, its own dependencies, its own resource limits. Your dbt task runs a slim Python image with just dbt-core. Your Spark submission task runs an image with PySpark and your JAR files. Your ML training task gets a GPU-enabled image with PyTorch. No dependency conflicts. No shared memory pressure. No crash-loop risk from a bad requirements.txt — because each task manages its own dependencies independently.

Scale to zero. When nothing is running, you’re paying for the scheduler and webserver — maybe $50–80/month on a small EC2 instance or ECS task. When a hundred tasks fire simultaneously, Kubernetes spins up a hundred pods, each with exactly the resources that task needs. When they finish, those pods terminate and you stop paying. MWAA, by contrast, charges for workers 24/7 regardless of whether tasks are running.

No CeleryExecutor limitations. The KubernetesExecutor gives you everything MWAA’s CeleryExecutor can’t: custom Docker images per task, no 12-hour execution limit (Kubernetes pods can run indefinitely), pod-level resource requests and limits, and the ability to use Spot instances for worker nodes at 50–70% cost savings.

The cold-start trade-off. Spinning up a Kubernetes pod takes 15–30 seconds. For batch workloads that run hourly or daily, this is negligible. For real-time or sub-minute scheduling, it’s not ideal — but those workloads probably shouldn’t be in Airflow anyway.

What does this actually look like in practice? Your DAG file stays clean and declarative. Instead of using PythonOperator to run transformations inside the Airflow worker, you use KubernetesPodOperator pointing at a Docker image that contains your transformation logic and its dependencies:

transform_sales = KubernetesPodOperator(
    task_id="transform_sales_data",
    image="your-ecr-repo/sales-transform:v2.3",
    namespace="airflow-tasks",
    resources={"request_cpu": "2", "request_memory": "4Gi"},
    node_selector={"lifecycle": "spot"},  # 50-70% cost savings
    is_delete_operator_pod=True,
)

Each task declares exactly what it needs. Your dbt task runs a slim image. Your Spark submission gets a heavier one. Your ML training task gets GPU resources. And because each pod is ephemeral, a dependency conflict in one task can never affect another — the isolation is complete.

So is this just a slower version of what MWAA does? No — it’s architecturally different in important ways.

MWAA uses CeleryExecutor with always-on Fargate workers. Tasks run inside those workers, sharing the same Python environment, the same dependencies, the same memory. The workers exist whether tasks are queued or not. When you update requirements.txt, every worker gets rebuilt (hence the 20–40 minute update times).

The thin orchestrator pattern puts compute outside the scheduler entirely. Airflow becomes a pure control plane. The data plane — where actual work happens — is ephemeral and independently deployable. You can update a task’s Docker image in seconds without touching the Airflow environment at all.

The self-hosted overhead is real though. You’re managing the Airflow installation yourself: upgrades, database administration (or using Amazon RDS for the metadata database), security patching, and monitoring setup. Budget 0.25–0.5 FTE for a small-to-medium deployment. At $150K/year fully loaded, even 0.25 FTE equals $3,125/month in engineering time — which can exceed MWAA’s premium for teams that don’t already have Kubernetes expertise.

This pattern works best when your team already runs Kubernetes for other workloads (so the infrastructure cost is shared), you need per-task dependency isolation, or you’re running compute-heavy tasks where Spot instance pricing makes a material difference.

One team running 80 pipelines on self-hosted ECS reported $400–500/month total compute costs. A comparable MWAA setup would run $600–900/month. But the engineering time to maintain the self-hosted setup is the real cost comparison — and it varies enormously depending on team capability.

When MWAA Is — and Isn’t — the Right Choice

MWAA earns its place when three conditions align: deep AWS commitment (your stack is S3, Redshift, Glue, EMR), existing Airflow investment (DAGs, team expertise, operator familiarity), and a preference for managed infrastructure over platform engineering. Teams running 50–500 DAGs on AWS with two or more data engineers and no multi-cloud requirements will find MWAA productive and cost-effective relative to the operational burden of self-hosting.

Here’s where it falls short:

Intermittent or light workloads — unless you’re willing to accept MWAA Serverless’s operator restrictions, the always-on cost floor is too high. Step Functions or Prefect’s free tier are dramatically cheaper.

Multi-cloud requirements — MWAA is AWS-only. Astronomer or self-hosted Airflow provides portability across clouds.

Asset-centric observability — if data lineage, freshness tracking, and column-level quality monitoring are priorities, Dagster Cloud is purpose-built for this. Bolting these capabilities onto MWAA requires multiple additional tools and significant integration work.

KubernetesExecutor or per-task isolation — MWAA is locked to CeleryExecutor. Teams needing custom Docker images per task or pod-level resource isolation should look at Astronomer, the thin orchestrator pattern on self-hosted, or Dagster Cloud.

Rapid Airflow version adoption — MWAA lags upstream releases by 2–6 months. Astronomer typically offers day-zero support for new versions.

Cost sensitivity at scale with multiple environments — per-team MWAA environments compound quickly. Three teams each needing their own environment is $1,100+/month before any work happens. Astronomer’s scale-to-zero workers or self-hosted with Spot instances become significantly cheaper at this scale.

For teams genuinely evaluating from scratch, here’s the landscape at a glance:

Here’s how I’d frame the decision:

Choose MWAA if you’re on AWS, your team knows Airflow, you want AWS to handle the scheduler and metadata database, and your workloads are substantial enough to justify the cost floor. It’s a solid, mature service that has improved dramatically in 2024–2025.

Choose Astronomer Astro if developer experience, KubernetesExecutor support, and fast Airflow upgrades matter more than AWS-native billing integration. It’s MWAA but better in almost every operational dimension — at a comparable or lower cost for active workloads.

Choose Dagster Cloud if you’re starting fresh, prioritise observability and data quality, and your team is comfortable learning a new paradigm. The asset-centric model is genuinely superior for understanding the health of your data platform.

Choose Prefect Cloud if your team is Python-first, dislikes Airflow’s operator model, and values dynamic workflows with minimal boilerplate. The free tier makes experimentation trivial.

Choose the thin orchestrator pattern (self-hosted Airflow + Kubernetes/ECS compute) if you already run Kubernetes, need per-task isolation, want maximum cost control, and have the engineering capacity to maintain the Airflow installation.

Choose Step Functions if your workflows are linear, event-driven, AWS-native, and don’t require backfill or complex scheduling. For many data engineering use cases, Step Functions plus EventBridge is simpler, cheaper, and less to maintain than any Airflow deployment.

The 2024–2025 Updates That Changed the Equation

MWAA’s pace of development accelerated dramatically in this period, and several updates addressed the community’s loudest complaints.

Environment class expansion in April 2024 (mw1.xlarge and mw1.2xlarge) and November 2024 (mw1.micro) gave teams both the headroom for heavy workloads and the affordable entry point for dev/test that had been missing since launch. The micro class at ~$37/month is particularly valuable — before its introduction, every dev environment cost the same $365+ as production.

Airflow versions progressed from 2.8 through 2.9 and 2.10, with Airflow 3.0 landing on MWAA in October 2025. The 3.0 release brought a redesigned React UI, event-driven scheduling via Assets (Airflow’s answer to Dagster’s Software-Defined Assets), a Task SDK with least-privilege execution, and Python 3.12 support. The Asset concept is significant — it means Airflow is converging toward the same asset-centric thinking that Dagster pioneered.

MWAA Serverless (November 2025) was the most consequential launch — true pay-per-execution pricing that directly addresses the always-on cost complaint. But as mentioned earlier, the limitations (no Airflow UI, YAML-only definitions, no PythonOperator) make it a different product for a different use case, not a cheaper version of MWAA.

Graceful updates (May 2025) eliminated the other major pain point: environments can now be updated without interrupting running tasks. The SigV4 REST API (October 2024) simplified programmatic access by supporting standard AWS credentials instead of login token management.

Community sentiment sits at mixed-positive. One engineering team’s May 2025 evaluation captured the tension well: Airflow remains the industry standard with a proven track record, but the improvements in Airflow 3 are following features that Dagster and Prefect already have. Apache Airflow pulled 320 million downloads in 2024, dwarfing competitors — but Dagster’s growth trajectory and commit activity signal a genuine shift in the market.

Closing Thoughts

The orchestration landscape is converging. Airflow 3.0 adopted asset-aware scheduling — a concept Dagster pioneered. Prefect’s event-driven model influenced Airflow’s trigger and asset-watcher features. Dagster is building broader operator support. They’re all moving toward the same destination from different starting points.

That convergence means the “right” choice increasingly depends not on feature parity but on team expertise, existing investment, and cloud strategy. And honestly? The worst decision you can make is choosing MWAA by default because you’re on AWS. Run the numbers. Evaluate the developer experience gaps. Try spinning up a Dagster Cloud or Prefect Cloud instance alongside your MWAA evaluation — both have free tiers, and the comparison will be illuminating.

I think about that first debugging session on the inherited Step Functions setup often. Hours to find something that should have taken minutes. The platform we eventually chose didn’t change the underlying compute or the logic of the pipelines — it changed whether we could see what was happening. Pick the orchestrator that gives your team that visibility. The one they’ll trust when something breaks at an inconvenient hour. That’s the one that wins.