capabilities.work

Provision a Cluster and Run a Job on It

A 5-step recipe with Azure Databricks. We can help you get access.

Recipe · 5 steps

Provision a Cluster and Run a Job on It

with Azure Databricks · IT Infrastructure Management

Create a cluster, wait for RUNNING, then create and run a notebook job.

  1. Create the cluster that will host the job. The cluster starts in PENDING and the cluster_id is returned.
  2. Retrieve the cluster state. Repeat until the cluster is RUNNING, then create the job; branch to failure if it terminates.
  3. Create a notebook job bound to the newly provisioned cluster and capture the job_id.
  4. Trigger an immediate run of the job and capture the run_id for polling.
  5. Retrieve the run state. Repeat until the run life_cycle_state is TERMINATED, then end with the final result_state.
What you'll need to have ready
  • token: Databricks personal access token for the Authorization header.
  • clusterName: Name for the new cluster.
  • sparkVersion: Databricks Runtime version key for the cluster.
  • nodeTypeId: Azure VM node type id for the cluster.
  • numWorkers: Number of worker nodes for the cluster.
  • jobName: Name for the job to create.
  • taskKey: Unique key identifying the single task within the job.
  • notebookPath: Absolute workspace path of the notebook to run.
For your developer

Arazzo workflow microsoft-azure-databricks:provision-cluster-and-run-job, 5 operations: createCluster → getCluster → createJob → runJobNow → getJobRun. Specs on APIs.io.

More recipes with Azure Databricks

Back Up a Notebook by Export and Re-importCancel an Active Job RunClean Up the Latest Completed Job RunCluster Health DiagnosticsCreate a Directory and Import a NotebookCreate a Job and Run It to CompletionSafely Delete a Workspace DirectoryImport a Notebook and Run ItPin the First Listed ClusterPreflight and Create a Cluster