capabilities.work

Create a Job and Run It to Completion

A 4-step recipe with Azure Databricks. We can help you get access.

Recipe · 4 steps

Create a Job and Run It to Completion

with Azure Databricks

Create a notebook job, run it now, and wait for the run to finish.

  1. Create a job containing a single notebook task bound to an existing cluster. The canonical job_id is returned.
  2. Trigger an immediate run of the newly created job and capture the run_id for polling.
  3. Retrieve the run state. Repeat until the run life_cycle_state is TERMINATED, then branch on the result_state.
  4. Retrieve the notebook output for the successful run, capturing the value passed to dbutils.notebook.exit().
What you'll need to have ready
  • token: Databricks personal access token for the Authorization header.
  • jobName: Name for the new job.
  • taskKey: Unique key identifying the single task within the job.
  • notebookPath: Absolute workspace path of the notebook to run.
  • existingClusterId: Id of an existing all-purpose cluster to run the task on.
For your developer

Arazzo workflow microsoft-azure-databricks:create-job-and-run, 4 operations: createJob → runJobNow → getJobRun → getJobRunOutput. Specs on APIs.io.

More recipes with Azure Databricks

Back Up a Notebook by Export and Re-importCancel an Active Job RunClean Up the Latest Completed Job RunCluster Health DiagnosticsCreate a Directory and Import a NotebookSafely Delete a Workspace DirectoryImport a Notebook and Run ItPin the First Listed ClusterPreflight and Create a ClusterProvision a Cluster and Run a Job on It