capabilities.work

Provision a Full Data Pipeline

A 5-step recipe with Airbyte. We can help you get access.

Recipe · 5 steps

Provision a Full Data Pipeline

with Airbyte · Information & Data Management

Create source and destination, wire a connection, and trigger the first sync.

  1. Create the source connector in the workspace using the supplied name and configuration blob.
  2. Create the destination connector in the same workspace that will receive the synced data.
  3. Resolve the available stream properties for the new source/destination pair so the catalog can be confirmed before the connection is created.
  4. Create the connection that binds the source to the destination. The connection is created with a manual schedule so the first sync is triggered explicitly by the next step.
  5. Trigger the inaugural sync job for the freshly created connection so data begins moving from source to destination.
What you'll need to have ready
  • workspaceId: The UUID of the workspace to provision the pipeline in.
  • sourceName: Human readable name for the new source (e.g. dev-postgres).
  • sourceConfiguration: Connector configuration JSON blob for the source (must include sourceType or a definitionId).
  • destinationName: Human readable name for the new destination.
  • destinationConfiguration: Connector configuration JSON blob for the destination (must include destinationType or a definitionId).
  • connectionName: Optional name for the connection that binds source to destination.
For your developer

Arazzo workflow airbyte:provision-pipeline, 5 operations: createSource → createDestination → getStreamProperties → createConnection → createJob. Specs on APIs.io.

More recipes with Airbyte

Bootstrap an Application and Access TokenFind and Cancel a Running JobCreate and Verify a DestinationFind a Connection and Trigger Its SyncReset a Connection and Re-SyncCreate and Verify a SourceCreate a Tag and Apply It to a ConnectionTrigger a Sync and Poll to CompletionBootstrap a Workspace