capabilities.work

Trace Dataset-Driven Lineage

A 4-step recipe with Apache Airflow. We can help you get access.

Recipe · 4 steps

Trace Dataset-Driven Lineage

with Apache Airflow

Trace a dataset's producers, consumers, update events, and the run it triggered.

  1. Search the dataset inventory by URI pattern to resolve the exact dataset, and to confirm the deployment is registering datasets at all.
  2. Read the dataset to get both sides of the dependency in one payload: the tasks that produce it and the DAGs that are scheduled on it.
  3. List the update events recorded against this dataset, newest first. An empty or stale list here is the answer when a consuming DAG has not run: nothing updated the dataset, so nothing triggered.
  4. Read the dataset events that triggered a specific run of the consuming DAG. This is the direct causal link between an upstream producer and a downstream run, which no cron schedule can express.
What you'll need to have ready
  • uriPattern: A substring to search dataset URIs against, e.g. "warehouse" or "s3://curated".
  • datasetUri: The full dataset URI to profile. It is a path parameter, so it must be URL-encoded — the slashes in "s3://bucket/key" included.
  • consumerDagId: The dataset-scheduled DAG whose trigger is being traced.
  • consumerDagRunId: The run of the consuming DAG to explain. Its upstream dataset events are the updates that caused it to start.
For your developer

Arazzo workflow apache-airflow:dataset-lineage, 4 operations: get_datasets → get_dataset → get_dataset_events → get_upstream_dataset_events. Specs on APIs.io.

More recipes with Apache Airflow

Audit DAG Runs Across the FleetCollect DAG Run Results from XComDeployment Preflight CheckDiagnose a Failed DAG RunForce a Task Instance StateOnboard a DAGProvision a User with a RoleRegister a Connection After Testing ItRetry Failed Tasks in a DAG RunTrigger a DAG Run and Wait for Completion