capabilities.work

Provision and Wait for Cluster

A 4-step recipe with Azure Databricks. We can help you get access.

Recipe · 4 steps

Provision and Wait for Cluster

with Azure Databricks · IT Infrastructure Management

Create a Databricks cluster and wait until it is RUNNING.

  1. Create the cluster from the supplied configuration. The cluster starts in a PENDING state and the canonical cluster_id is returned.
  2. Retrieve the current cluster state. Repeat this step until the cluster reports RUNNING; branch to failure if it reports TERMINATED.
  3. Re-read the cluster once it is RUNNING to capture its final metadata for the workflow outputs.
  4. Read the cluster one last time to capture the termination reason when provisioning did not reach RUNNING.
What you'll need to have ready
  • token: Databricks personal access token for the Authorization header.
  • clusterName: Human-readable name for the new cluster.
  • sparkVersion: Databricks Runtime version key (from listSparkVersions).
  • nodeTypeId: Azure VM node type id for the workers and driver.
  • numWorkers: Number of worker nodes for a fixed-size cluster.
  • autoterminationMinutes: Idle minutes before the cluster auto-terminates.
For your developer

Arazzo workflow microsoft-azure-databricks:provision-cluster, 4 operations: createCluster → getCluster → getCluster → getCluster. Specs on APIs.io.

More recipes with Azure Databricks

Back Up a Notebook by Export and Re-importCancel an Active Job RunClean Up the Latest Completed Job RunCluster Health DiagnosticsCreate a Directory and Import a NotebookCreate a Job and Run It to CompletionSafely Delete a Workspace DirectoryImport a Notebook and Run ItPin the First Listed ClusterPreflight and Create a Cluster