Diagnose a Failed DAG Run
A 6-step recipe with Apache Airflow. We can help you get access.
Recipe · 6 steps
Diagnose a Failed DAG Run
with Apache Airflow
Find the failed task in a DAG run, retrieve its log, and annotate the finding.
- Read the DAG run and require the failed state so the workflow is not run against a healthy pipeline.
- Filter the run's task instances to the failed and upstream_failed states. The state filter repeats to express an OR condition, so both the task that actually broke and the tasks blocked behind it come back in one call.
- Read the target task instance to resolve the try_number of its most recent attempt, which the log endpoint requires as a path parameter.
- Pull the complete log for the failing attempt. full_content returns the whole log body rather than a paginated slice, which is what a diagnosis needs.
- Record the diagnosis on the task instance so it surfaces in the Airflow UI next to the failure itself.
- Record the same diagnosis on the DAG run so anyone scanning the run list sees why this run failed without drilling into a task.
What you'll need to have ready
- dagId: The DAG identifier the failed run belongs to.
- dagRunId: The run identifier to diagnose.
- taskId: The task instance to pull logs for, normally one of the task ids returned by the listFailedTasks step.
- diagnosis: The human written finding to record against the task instance and the DAG run once the log has been read.
For your developer
Arazzo workflow apache-airflow:diagnose-failed-dag-run, 6 operations: getApiV2DagsByDagIdDagRunsByDagRunId → get_task_instances → get_task_instance → get_log → set_task_instance_note → set_dag_run_note. Specs on APIs.io.