WOSS
Apache Airflow: Pipelines as Code, Grown Up
Aman Mundra · 2026-07-12 · 4 min read
Contents
TL;DR - Apache Airflow is the leading workflow orchestration platform: you author, schedule, and monitor data and ML pipelines as code, expressed as DAGs (directed acyclic graphs). Airflow 3 (April 2025) was the biggest release in its history - a service-oriented architecture, a rewritten React UI, native DAG versioning, first-class Assets, and a Task Execution API that decouples workers from the metadata database. It is the orchestration stage between ingestion (Kafka) and compute (Spark) in a cloud-native AI pipeline.
Every data platform eventually needs an answer to a deceptively hard question: what runs, in what order, when, and what happens when a step fails? Airflow's answer - express the whole thing as code - was influential enough that it became the industry default.
Pipelines as directed graphs
An Airflow pipeline is a DAG: a directed acyclic graph of tasks, where edges encode dependencies. "Extract must finish before transform; transform before load; notify only if all three succeed." You write that graph in Python, and Airflow handles the rest - scheduling runs, respecting dependencies, retrying failures, tracking state, and giving you lineage and observability across the whole thing.
The "as code" part is what made Airflow stick. Because a pipeline is Python in a Git repository, it gets everything code gets: version control, code review, testing, and reuse. A vast ecosystem of operators and providers lets a single DAG coordinate work across databases, cloud services, and processing engines without you writing the integration glue yourself. Pipelines stopped being brittle cron jobs and shell scripts and became reviewable, auditable software.
Airflow 3: the engine, rebuilt
Airflow 2 was successful enough that its architectural assumptions became limitations. Airflow 3, released April 2025, rebuilt the foundations - and like its Apache siblings Kafka and Spark, the central theme was decoupling a tightly-bound dependency.
The headline changes:
- A Task Execution API (AIP-72). Previously, tasks needed direct access to the metadata database and ran inside the Airflow process tree. In Airflow 3, tasks talk to the scheduler through a dedicated API server, so workers are fully decoupled from the core stack. This is the architectural shift that lets tasks run anywhere, with full isolation from the database, and it lays the groundwork for tasks written in languages other than Python - a capability the 3.3 Coordinator layer (below) began to deliver.
- DAG versioning - the single most-requested feature for years. A DAG run now completes against the version it started with, even if you deploy a new version mid-run. No more pipelines quietly changing shape under a running execution.
- Assets replace Datasets. The data-aware scheduling concept from Airflow 2 was elevated to a first-class Asset with a clean
@assetdecorator, making event-driven, data-triggered pipelines a native pattern rather than a workaround. - A rewritten React UI on a FastAPI backend, retiring the old Flask-AppBuilder interface - faster, and finally modern.
The project has kept a brisk cadence since: 3.1 added human-in-the-loop workflows, 3.2 brought asset partitions and multi-team isolation within a single deployment, and 3.3 (July 2026) added an experimental Coordinator layer that lets individual tasks be written in non-Python languages while the DAG and its scheduling stay in Python.
One practical note if you are upgrading: Airflow 3 has real breaking changes, but the project ships a Ruff-based upgrade check (ruff check --preview --select AIR30 --fix) that flags and auto-fixes the most common DAG-level breakages, plus airflow config lint for configuration.
Where Airflow fits in an AI stack
Airflow is the conductor. In a data or ML platform it sequences the steps: pull events that Kafka ingested, trigger the Spark jobs that transform them and build features, kick off model training, run evaluation, and publish results - each step ordered, retried on failure, and observable. Those three projects form the data-infrastructure backbone under many cloud-native AI systems, and Airflow is the layer that makes the sequence reliable and repeatable instead of a fragile chain of scripts.
For ML specifically, the Asset model and event-driven scheduling map cleanly onto the real shape of the work: "when fresh training data lands, retrain; when the new model passes evaluation, promote it." That is a data-aware pipeline, and Airflow 3 makes it a first-class expression rather than a clever hack.
Getting involved
Airflow's contribution surface is broad and welcoming: provider packages (the integrations, such as the Kafka provider), core, and the UI, plus documentation, examples, and tests that make excellent first contributions. Larger changes go through AIPs (Airflow Improvement Proposals). It is a large, active community with a well-worn onboarding path.
