Provision Cluster Then Create Job On It
A 3-step recipe with Databricks. We can help you get access.
Recipe · 3 steps
Provision Cluster Then Create Job On It
with Databricks · IT Infrastructure Management
Create a cluster, wait for RUNNING, then create a job on it.
- Create the cluster that the job task will run on. Returns the cluster_id reused by the job task.
- Read the cluster status and inspect the life cycle state. Loop back while PENDING; continue once it is RUNNING.
- Create a job whose single notebook task runs on the newly provisioned cluster via existing_cluster_id.
What you'll need to have ready
- cluster_name: Name for the new cluster.
- spark_version: The Spark runtime version for the cluster.
- node_type_id: The node type for the cluster.
- num_workers: The number of worker nodes.
- job_name: The name for the new job.
- task_key: The unique task key within the job.
- notebook_path: The workspace path of the notebook the task runs.
For your developer
Arazzo workflow databricks:provision-cluster-and-create-job, 3 operations: createCluster → getCluster → createJob. Specs on APIs.io.