Execution model

Pipes uses a durable execution engine that guarantees workflows run to completion, even through failures, deployments, and infrastructure disruptions.

StartTrigger fires

ExecuteRun node logic

CheckpointPersist state

SuccessWorkflow complete

ResumeRetry from checkpoint

How execution works

When a trigger fires, Pipes creates an execution, a tracked instance of the workflow running against a specific input. Each execution progresses through nodes, checkpointing state after every step.

// Execution lifecycle
// 1. Trigger fires -> execution created (status: "running")
// 2. Each node executes -> state checkpointed
// 3. On success -> execution complete (status: "completed")
// 4. On failure -> retry from last checkpoint (status: "retrying")
// 5. On exhausted retries -> DLQ (status: "failed")

Checkpointing

After each node completes, Pipes persists the execution state. If a failure occurs, the workflow resumes from the last checkpoint rather than restarting from the beginning.

// Node A completes -> checkpoint saved
// Node B completes -> checkpoint saved
// Node C fails -> retry from Node C (not Node A)

This is critical for workflows that make external API calls or modify state. You never double-charge a customer or send duplicate emails.

Retry policies

Configure retries at the workflow or node level:

const workflow = await npayload.pipes.create({
  name: 'payment-processor',
  trigger: { type: 'event', channel: 'orders' },
  retryPolicy: {
    maxRetries: 5,
    backoff: 'exponential',
    initialDelay: 1000,  // 1 second
    maxDelay: 60000,     // 1 minute
  },
});

Strategy	Delay pattern	Use case
`fixed`	Same delay each retry	Transient network errors
`exponential`	1s, 2s, 4s, 8s...	API rate limits
`linear`	1s, 2s, 3s, 4s...	Gradual backoff

Circuit breaker

When a node fails repeatedly, the circuit breaker trips to prevent cascading failures:

workflow.addNode({
  type: 'http',
  url: 'https://api.payment.com/charge',
  circuitBreaker: {
    failureThreshold: 5,     // Trip after 5 consecutive failures
    resetTimeout: 30000,     // Try again after 30 seconds
    halfOpenRequests: 1,     // Allow 1 test request in half-open state
  },
});

States:

Closed: Normal operation, requests flow through
Open: All requests fail immediately (fast-fail)
Half-open: One test request allowed; success closes, failure re-opens

Dead letter queue

Executions that exhaust all retries are routed to a DLQ for inspection and manual replay:

// Failed executions appear in the workflow DLQ
const failed = await npayload.pipes.dlq.list({
  workflow: 'payment-processor',
  limit: 50,
});

// Inspect and replay a failed execution
await npayload.pipes.dlq.replay(failed[0].executionId);

Backpressure

When downstream systems slow down, Pipes automatically applies backpressure to prevent overload:

Queue depth monitoring: Pauses trigger consumption when queue depth exceeds threshold
Concurrency limits: Caps the number of parallel executions per workflow
Rate limiting: Throttles execution rate to match downstream capacity

const workflow = await npayload.pipes.create({
  name: 'batch-processor',
  trigger: { type: 'event', channel: 'events' },
  concurrency: 10,       // Max 10 parallel executions
  rateLimit: {
    requests: 100,
    window: '1m',         // 100 executions per minute
  },
});

Priority queue

Workflows can process high-priority items first:

const workflow = await npayload.pipes.create({
  name: 'support-router',
  trigger: {
    type: 'event',
    channel: 'support-tickets',
    priorityField: 'payload.severity',  // Higher values processed first
  },
});

Execution visibility

Every execution is fully observable:

// Get execution status and history
const execution = await npayload.pipes.executions.get(executionId);

console.log(execution.status);     // "completed" | "running" | "failed" | "retrying"
console.log(execution.startedAt);  // When execution began
console.log(execution.duration);   // Total execution time
console.log(execution.nodeResults); // Output of each node

Next steps

Node types for available nodes
Building workflows for composing nodes
Saga workflows for multi-step orchestration with compensation

Was this page helpful?

Execution model

On this page