capabilities.work

Submit and Poll Spark Batch Job

A 2-step recipe with Azure Synapse Analytics. We can help you get access.

Recipe · 2 steps

Submit and Poll Spark Batch Job

with Azure Synapse Analytics

Submit a Spark batch job and poll it until it reaches a terminal result.

  1. Submit a new Spark batch job to the pool using the supplied batch options.
  2. Read the batch job by id. While the livy result is still Uncertain, loop back and poll again; once a terminal result is reported, end the workflow.
What you'll need to have ready
  • sparkPoolName: The name of the Spark pool to run the batch job on.
  • batchOptions: The SparkBatchJobOptions body. Must include name and file at minimum.
For your developer

Arazzo workflow microsoft-azure-synapse-analytics:submit-and-poll-spark-batch-job, 2 operations: SparkBatch_CreateSparkBatchJob → SparkBatch_GetSparkBatchJob. Specs on APIs.io.

More recipes with Azure Synapse Analytics

Build Data Integration PipelineDeploy and Run PipelineDeploy and Start TriggerDeploy and Execute Spark Job DefinitionOpen Spark Session and Run StatementProvision and Resume SQL PoolProvision Linked Service and DatasetProvision Spark PoolPublish and Verify NotebookPublish and Verify SQL Script