In Airflow, each task has access to an XCOM object, which is used to store and retrieve data. When a task wants to share data with another task, it can use the xcom_push method to store the data in the XCOM object. The data is then available for other tasks to retrieve using the xcom_pull method.
def push_function(**context): context['ti'].xcom_push(key='message', value='Hello from Task 1')
Here's an example of using XCOM in a DAG:
✅ Use cloud storage (S3/GCS/Azure Blob) or a temp file system. Pass only the reference path via XCom. xcom airflow
While XComs are powerful, they are not a universal solution for data transfer. Following best practices from Astronomer and the official Airflow docs is vital for performance. ✅ Best Use Cases
task1 = BashOperator( task_id='task1', bash_command='echo "Hello, World!"', xcom_push_key='greeting', dag=dag, )
| Aspect | XCom | |--------|------| | Purpose | Small data exchange between tasks | | Storage | Airflow metadata DB | | Max size | ~1 MB (configurable) | | Best for | Paths, flags, counts, JSON | | Avoid | DataFrames, large files, images | | Alternative for large data | S3/GCS + pass reference | In Airflow, each task has access to an
XComs are the glue that holds complex Airflow DAGs together. By understanding the balance between explicit pushes and automatic return values—and respecting the storage limits of the metadata database—you can build robust, data-aware pipelines that scale with your needs.
| Limitation | Details | |------------|---------| | | Default is 1 MB per XCom (can be changed, but not recommended) | | Not for big data | Do not pass large DataFrames, images, or file contents | | Database performance | Stored in metadata DB → many large XComs slow down Airflow | | Pickling risks | Custom objects may break across workers / versions |
airflow.cfg :
A subsequent task "pulls" that specific value using a unique key, task ID, and DAG ID.
In Apache Airflow, each task instance is typically isolated. Tasks cannot share variables or data directly. (short for cross-communication ) is the mechanism that allows tasks to exchange small pieces of data.