Retry Failed Tasks in a DAG Run
A 5-step recipe with Apache Airflow. We can help you get access.
Recipe · 5 steps
Retry Failed Tasks in a DAG Run
with Apache Airflow
Dry-run a clear, execute it, and wait for the resulting re-run to finish.
- Read the DAG run and require the failed state. Clearing a healthy run would throw away good work, so the workflow refuses to proceed on anything else.
- Call the clear endpoint with dry_run enabled. Nothing changes; Airflow just returns the task instances the real call would reset, which is the blast radius the operator should approve before continuing.
- Perform the clear for real. reset_dag_runs returns the DAG run itself to the queued state so the scheduler picks the cleared tasks back up; without it the run stays failed and the tasks never re-run.
- Poll the DAG run until Airflow stamps a fresh end_date, marking the re-run as finished. Retries every 15 seconds for up to 80 attempts (~20 minutes).
- Write the retry note onto the DAG run so the history explains why the run was cleared and what happened on the second attempt.
What you'll need to have ready
- dagId: The DAG identifier the failed run belongs to.
- dagRunId: The run identifier whose failed tasks should be cleared and re-run.
- taskIds: An optional explicit list of task ids to clear. When omitted the clear applies to every failed task in the run.
- note: The note to record against the DAG run once the re-run settles.
For your developer
Arazzo workflow apache-airflow:retry-failed-tasks, 5 operations: getApiV2DagsByDagIdDagRunsByDagRunId → post_clear_task_instances → post_clear_task_instances → getApiV2DagsByDagIdDagRunsByDagRunId → set_dag_run_note. Specs on APIs.io.