Idempotency and reliability

Prevent duplicate messages with idempotency keys and build reliable publish patterns

Network failures, process restarts, and timeouts can cause your application to retry a publish call that already succeeded. Without protection, this creates duplicate messages. npayload provides idempotency keys to guarantee exactly-once publish semantics.

The problem

// This code might publish twice if the first request succeeds
// but the response is lost due to a network timeout
const message = await npayload.messages.publish({
  channel: 'payments',
  payload: { event: 'payment.completed', paymentId: 'pay_123' },
});

If the network drops after npayload stores the message but before your application receives the response, your retry logic will publish the same message again.

Using idempotency keys

An idempotency key is a unique string that identifies a specific publish operation. If npayload receives a second publish with the same key within the retention window, it returns the existing message instead of creating a new one.

const message = await npayload.messages.publish({
  channel: 'payments',
  idempotencyKey: `payment-completed-${paymentId}`,
  payload: { event: 'payment.completed', paymentId: 'pay_123' },
});

// Safe to retry - same key returns the same message
const retry = await npayload.messages.publish({
  channel: 'payments',
  idempotencyKey: `payment-completed-${paymentId}`,
  payload: { event: 'payment.completed', paymentId: 'pay_123' },
});

// message.gid === retry.gid (same message, no duplicate)

Choosing idempotency keys

Good keys are deterministic and unique to the operation:

Pattern	Example	Use case
Entity + event	`order-${orderId}-created`	Domain events
Request ID	`req-${requestId}`	API request deduplication
Transaction ID	`tx-${transactionId}`	Financial operations
Composite	`${userId}-${action}-${timestamp}`	User actions

Avoid random UUIDs as idempotency keys. The key must be the same across retries to provide deduplication. Derive it from the operation's natural identity.

Retention window

Idempotency keys are valid for the channel's retention period (default: 7 days). After the key expires, a publish with the same key creates a new message.

Batch publish with idempotency

Each message in a batch can have its own idempotency key:

const result = await npayload.messages.publishBatch({
  channel: 'events',
  messages: [
    {
      idempotencyKey: `order-${orderId}-created`,
      payload: { event: 'order.created', orderId },
    },
    {
      idempotencyKey: `inventory-${sku}-reserved`,
      payload: { event: 'inventory.reserved', sku, quantity: 2 },
    },
  ],
});

If some messages in the batch are duplicates, they are returned as successful (with their existing GID) while new messages are created normally.

Transactional publish with idempotency

Transactional publish is inherently atomic, but you should still use idempotency keys to protect against retries of the entire transaction:

await npayload.messages.publishTransactional([
  {
    channel: 'orders',
    idempotencyKey: `order-${orderId}-created`,
    payload: { event: 'order.created', orderId },
  },
  {
    channel: 'inventory',
    idempotencyKey: `order-${orderId}-stock-reserved`,
    payload: { event: 'stock.reserved', orderId, items },
  },
]);

Reliable publish patterns

Pattern 1: Retry with backoff

async function reliablePublish(channel: string, payload: unknown, key: string) {
  const maxRetries = 3;

  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await npayload.messages.publish({
        channel,
        idempotencyKey: key,
        payload,
      });
    } catch (error) {
      if (attempt === maxRetries) throw error;

      const delay = Math.min(1000 * Math.pow(2, attempt - 1), 10000);
      await new Promise((resolve) => setTimeout(resolve, delay));
    }
  }
}

Pattern 2: Outbox pattern

For applications that need to update a database and publish a message atomically:

Write the message to an outbox table in your database (same transaction as the business logic)
A background process reads the outbox and publishes to npayload with an idempotency key derived from the outbox record ID
Mark the outbox record as published after successful delivery

// Step 2: Background worker publishes from outbox
for (const record of outboxRecords) {
  await npayload.messages.publish({
    channel: record.channel,
    idempotencyKey: `outbox-${record.id}`,
    payload: record.payload,
  });

  await db.markPublished(record.id);
}

The outbox pattern guarantees that your database state and npayload messages are always consistent, even if your process crashes between the database write and the publish call.

Next steps

Publishing guide for all publish options
Error handling for handling publish failures
Transactional publish for atomic multi-channel publishing

Was this page helpful?

Idempotency and reliability

On this page