capabilities.work

Deploy and Execute Spark Job Definition

A 4-step recipe with Azure Synapse Analytics. We can help you get access.

Recipe · 4 steps

Deploy and Execute Spark Job Definition

with Azure Synapse Analytics

Publish a Spark job definition, run it, and poll the launched batch.

  1. Create or update the Spark job definition in the workspace.
  2. Read the Spark job definition back to confirm it was persisted before executing it.
  3. Execute the Spark job definition, which submits a Spark batch job and returns the batch identifier.
  4. Poll the launched Spark batch job by id. Loop while the livy result is still Uncertain, and end once a terminal result is reported.
What you'll need to have ready
  • apiVersion: The Synapse data plane API version (e.g. 2020-12-01).
  • sparkJobDefinitionName: The name of the Spark job definition to create or update.
  • sparkJobDefinition: The SparkJobDefinitionResource definition to publish.
  • sparkPoolName: The Spark pool name used to poll the launched batch job via the Livy API.
For your developer

Arazzo workflow microsoft-azure-synapse-analytics:deploy-execute-spark-job-definition, 4 operations: SparkJobDefinition_CreateOrUpdateSparkJobDefinition → SparkJobDefinition_GetSparkJobDefinition → SparkJobDefinition_ExecuteSparkJobDefinition → SparkBatch_GetSparkBatchJob. Specs on APIs.io.

More recipes with Azure Synapse Analytics

Build Data Integration PipelineDeploy and Run PipelineDeploy and Start TriggerOpen Spark Session and Run StatementProvision and Resume SQL PoolProvision Linked Service and DatasetProvision Spark PoolPublish and Verify NotebookPublish and Verify SQL ScriptSubmit and Poll Spark Batch Job