Scrape and Deliver Snapshot to Cloud Storage
A 3-step recipe with Bright Data. We can help you get access.
Recipe · 3 steps
Scrape and Deliver Snapshot to Cloud Storage
with Bright Data
Trigger a scrape, poll until ready, and deliver the snapshot to cloud storage.
- Trigger an asynchronous scraping job for the dataset, returning a snapshot id to poll and deliver.
- Poll the snapshot progress endpoint until the status reaches a terminal value. A ready status means the snapshot can be delivered.
- Schedule delivery of the ready snapshot to the configured cloud storage destination in the requested format.
What you'll need to have ready
- apiToken: Bright Data API token used as a Bearer credential.
- datasetId: Bright Data dataset identifier of the collector to run.
- records: Array of per-record input objects passed to the collector.
- deliverType: Delivery destination type (s3, gcs, azure, snowflake, webhook).
- bucket: Destination bucket or container name.
- credentials: Destination credentials object.
- format: Delivery format (json, ndjson, csv, parquet).
For your developer
Arazzo workflow bright-data:scrape-and-deliver-snapshot, 3 operations: triggerScrape → getScrapeProgress → deliverSnapshot. Specs on APIs.io.