capabilities.work

Create Scenario and Run It

A 4-step recipe with Runloop. We can help you get access.

Recipe · 4 steps

Create Scenario and Run It

with Runloop

Create a scenario, run it, wait for scoring, and complete the run.

  1. Create a scenario with the supplied problem statement and a single bash script scorer weighted at 1.0.
  2. Start a scenario run, which provisions a devbox for the evaluation.
  3. Poll the scenario run until it reaches the scored state, looping while it is running or scoring, and ending the flow on failure, timeout, or cancel.
  4. Complete the scored run, shutting down the underlying devbox.
What you'll need to have ready
  • apiToken: Runloop API bearer token.
  • scenarioName: Name to give the scenario.
  • problemStatement: The problem statement the scenario asks the agent to solve.
  • scorerName: Name of the scoring function (only [a-zA-Z0-9_-] allowed).
  • bashScript: Bash script that scores the run and prints score=[0.0..1.0] to stdout.
For your developer

Arazzo workflow runloop-ai:create-scenario-and-run, 4 operations: createScenario → startScenarioRun → getScenarioRun → completeScenarioRun. Specs on APIs.io.

More recipes with Runloop

Bootstrap a Devbox from an Object ArtifactBuild Blueprint and Launch DevboxGraceful Snapshot and ShutdownProvision Devbox and Run a CommandPublish an Object ArtifactScore a Running Scenario RunSnapshot and Restore a DevboxSuspend and Resume a DevboxWrite, Execute, and Read a File on a Devbox