capabilities.work

Crawl Site and Retrieve

A 3-step recipe with Hyperbrowser. We can help you get access.

Recipe · 3 steps

Crawl Site and Retrieve

with Hyperbrowser

Crawl a site from a seed URL and return the crawled page content.

  1. Submit a crawl job for the seed URL with the supplied page budget and capture the returned jobId.
  2. Poll the crawl job status. The status moves through pending and running before reaching completed, failed, or stopped; loop back while still in progress and branch out on a terminal state.
  3. Fetch the first batch of crawled pages from the completed crawl job along with the total crawled page count and batch counters.
What you'll need to have ready
  • apiKey: Hyperbrowser account API key sent in the x-api-key header.
  • url: The seed URL to start the crawl from.
  • maxPages: Maximum number of pages to crawl.
  • followLinks: Whether to follow links discovered on crawled pages.
For your developer

Arazzo workflow hyperbrowser:crawl-site-and-retrieve, 3 operations: post-api-crawl → get-api-crawl-id-status → get-api-crawl-id. Specs on APIs.io.

More recipes with Hyperbrowser

Browser Use Task RunExtract Structured DataHyperAgent Task RunPersistent Profile SessionSession LifecycleSession Recording RetrievalWeb Crawl and RetrieveWeb Search then Fetch