Resilience & Timeouts

@deuz-sdk/core makes one network attempt resilient and keeps it deterministic. Before the first byte of a response stream arrives, a failed request can be retried with exponential backoff and full jitter; once streaming has started, any error is final. Timeouts come in three independent layers driven by the injected clock, and a user abort is treated as a clean finish rather than a failure. All of this is wired through the Dependencies seam so it is reproducible in tests with no real clock and no real network.

Pre-first-byte retry only

The inference pump retries only before the first byte of the response body. The moment the stream begins emitting deltas, a mid-stream error is propagated as-is and never retried.

The reason is correctness, not laziness: a streaming response is stateful. Partial text, reasoning, and tool-call fragments have already been pushed to the consumer. A transparent retry would replay tokens the caller already saw, corrupting the stream and any UI bound to it. So the contract is: retry the connection and the non-2xx handshake, but never the stream.

This also means streamChat keeps its never-throw guarantee. A pre-first-byte failure that exhausts the retry budget surfaces as an error part on fullStream and rejects the usage / finishReason promises — it is not a synchronous throw.

The retry policy

The defaults live in DEFAULT_RETRY inside core/resilience.ts:

// internal — shown for reference
const DEFAULT_RETRY = { maxRetries: 2, baseMs: 500, capMs: 30_000 };

Override the budget per call with maxRetries:

import { streamChat } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';

const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });

const result = streamChat({
  model: anthropic('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'hi' }],
  maxRetries: 4, // raise the pre-first-byte budget for this call
});

Which errors retry

Retryability is decided by the isRetryable flag carried on APICallError and its subclasses — not by HTTP status guessing at the call site. Errors that are not HTTP-shaped (TimeoutError, AbortError, NoObjectGeneratedError, UnsupportedCapabilityError) carry no flag and are always final.

The verdict is a one-liner you can reuse for your own fallback logic:

import { APICallError } from '@deuz-sdk/core';

function isRetryable(err: unknown): boolean {
  return err instanceof APICallError && err.isRetryable;
}

See Error Handling for the full taxonomy.

Backoff and jitter are deterministic

The backoff formula is full jitter: Math.floor(min(cap, base * 2^attempt) * random()). The crucial detail is that random is not Math.random() — core never calls it. Instead the random unit comes from hashing deps.generateId() with FNV-1a into the [0, 1) range. Pin generateId and the retry delays become reproducible:

import { streamChat } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';

const result = streamChat({
  model: createAnthropic({ apiKey: 'k', fetch: mockFetch })('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'hi' }],
  // Same id every attempt → same jitter → same backoff in every test run.
  deps: { generateId: () => 'fixed-id' },
});

The wait between attempts is itself abortable: it is scheduled through deps.clock.setTimeout, and if the user signal aborts during the backoff, the delay rejects immediately rather than sleeping out the full interval.

Retry-After takes precedence

When a provider returns a Retry-After header (integer seconds or an HTTP-date), it is parsed to milliseconds and used directly instead of the computed backoff — still capped at the 30s ceiling. This is honored for the errors that carry it, notably RateLimitError (429) and OverloadedError (529).

import { streamChat } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';

// A mock that 429s once with Retry-After, then succeeds on the next attempt. The
// pump waits the header's value (capped at 30s) before retrying — no guesswork.
let attempt = 0;
const fetch = (async () => {
  if (attempt++ === 0) {
    return new Response(JSON.stringify({ type: 'error', error: { type: 'rate_limit_error' } }), {
      status: 429,
      headers: { 'content-type': 'application/json', 'retry-after': '2' },
    });
  }
  // Your canned success: a Response whose body is an SSE ReadableStream.
  return okSseResponse();
}) as typeof fetch;

const result = streamChat({
  model: createAnthropic({ apiKey: 'k', fetch })('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'hi' }],
});

The parsed value lives on the error too, so you can read it from a rejected promise: err.retryAfterMs on any APICallError.

Timeout layers

Timeouts are clock-driven (not AbortSignal.timeout, which is non-injectable and trips a Cloudflare Workers bug). Each layer is independent, and the model-call layers merge into the fetch signal via combineSignals:

The defaults are DEFAULT_TIMEOUTS in core/timeout.ts (ttftMs: 60_000, totalMs: 300_000). The TTFT timer is armed at pump start, when output is first accessed rather than at the synchronous streamChat return. It is cancelled the instant the first content delta lands. The total timer covers the entire request including the streaming phase.

chunkMs closes the gap between those two. Once content is flowing, a provider that stalls mid-answer would otherwise hold the call open until the total ceiling. With chunkMs set, any silence longer than that between two stream parts fails the call. Before the first content the TTFT layer still governs, so a model that thinks before it answers is not cut off.

A fired timer aborts the request with a TimeoutError whose layer names the budget that ran out: 'ttft', 'total', 'chunk', 'step' or 'tool'. Because the user signal and the timers are combined into one signal handed to fetch, whichever trips first wins.

TimeoutError is a failure; a user abort is not

This distinction is deliberate and load-bearing:

• A TimeoutError is a genuine failure. It surfaces as an error part on fullStream, and usage / finishReason reject with it. Neither layer is retried. A fired timer aborts fetch with the TimeoutError as the abort reason, and the pump rethrows any TimeoutError straight out of the retry loop before the budget is even consulted (inference.ts: if (err instanceof TimeoutError || isUserAbort(...)) throw err). So both a ttft timeout (during the handshake or before the first content delta) and a total timeout are final failures — never retried.
• A user abort (you called controller.abort()) is not an error. The pump resolves finishReason with 'aborted' and resolves usage with the partial token counts gathered so far. No error part is emitted, and the abort is never retried.

import { streamChat, TimeoutError } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';

const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });
const controller = new AbortController();

const result = streamChat({
  model: anthropic('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'Write an essay.' }],
  signal: controller.signal,
});

// Stop generation from elsewhere — resolves cleanly as 'aborted'.
setTimeout(() => controller.abort(), 1_000);

const finishReason = await result.finishReason; // 'aborted' on cancel
if (finishReason === 'aborted') {
  const usage = await result.usage; // partial counts, still resolves
  console.log('cancelled after', usage.outputTokens, 'output tokens');
}

Controlling timeout durations

Set any layer per call with timeout. Unset layers keep their defaults, a bare number is shorthand for { totalMs }, and 0 disables a layer:

import { streamChat } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';

const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY! });

const result = streamChat({
  model: anthropic('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'Summarise the report.' }],
  // Fail fast on a stalled stream; keep the default first-content wait.
  timeout: { totalMs: 25_000, chunkMs: 10_000 },
});

A stalled stream then rejects usage and finishReason with a TimeoutError whose layer is 'chunk'. A buffered call treats it like 'ttft' and 'total' for cross-provider failover. A streaming call has already delivered content by then, so the error is final there.

You also control the clock that drives the timers, since every timer is scheduled through deps.clock.setTimeout. In tests this lets you fire the short backoff timers eagerly while never firing the long ttft/total timers:

import type { Clock } from '@deuz-sdk/core';

// Fire short (backoff) timers immediately; never fire the long ttft/total timers.
export function fastClock(): Clock {
  return {
    now: () => 0,
    setTimeout: (fn, ms) => {
      if (ms < 60_000) {
        const id = setTimeout(fn, 0);
        return () => clearTimeout(id);
      }
      return () => {}; // ttft/total never fire
    },
  };
}

Pass it through deps to drive every timeout and backoff deterministically:

import { streamChat } from '@deuz-sdk/core';
import { createAnthropic } from '@deuz-sdk/core/anthropic';
import { fastClock } from './test/helpers';

const result = streamChat({
  model: createAnthropic({ apiKey: 'k', fetch: mockFetch })('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'hi' }],
  deps: { clock: fastClock(), generateId: () => 'fixed-id' },
});

To enforce a shorter wall-clock budget in production, wrap your own AbortController with a timer and pass its signal — your abort merges with the built-in layers, so the tightest bound wins:

const controller = new AbortController();
const timer = setTimeout(() => controller.abort(), 10_000); // 10s budget

const result = streamChat({
  model: anthropic('claude-opus-4-8'),
  messages: [{ role: 'user', content: 'hi' }],
  signal: controller.signal,
});

// Note: this surfaces as 'aborted' (a clean finish), not a TimeoutError.
result.finishReason.finally(() => clearTimeout(timer));

The circuit breaker (live since 1.7)

The inference pump runs a per-model circuit breaker: a model that keeps failing at the transport level stops receiving requests for a cooldown window and fails fast instead — which is exactly the signal cross-provider failover hops on.

How it works, all in core/inference.ts + core/resilience.ts:

• Keyed per provider:model (e.g. anthropic:claude-opus-4-8), state in deps.breakerStore. One sick model never blocks its siblings.
• What counts: only provider-health failures that end a call — network errors, timeouts, and 5xx/overload/retryable API errors (isBreakerCountable). Client errors (4xx auth/validation) say nothing about provider health and never count; a BreakerOpenError itself is a fast-fail read, never a new signal. Failures the retry budget absorbs don't count either — only the final error of a call does.
• Opening: BREAKER_THRESHOLD (5) consecutive countable failures open the breaker for BREAKER_COOLDOWN_MS (30s). Opening logs a deps.logger.warn('circuit breaker opened', …).
• While open: a call to that model fails immediately — before any network request — with a BreakerOpenError (code: 'breaker_open', carrying provider, modelId, and cooldownUntil). Per G2, on streamChat this surfaces as an error part and rejected promises, never a synchronous throw.
• Half-open probe: once cooldownUntil passes, the next call goes through as a probe.
• Reset on first byte: the moment a response body's first byte arrives, the provider is proven healthy again and the failure count resets to 0. Consecutive means consecutive.
• Best-effort store: a throwing breakerStore never blocks or changes a call — reads and writes are wrapped, failures ignored.

The state shape and seam:

interface BreakerState {
  failures: number;
  openedAt?: number;
  cooldownUntil?: number;
}

interface BreakerStore {
  get(key: string): BreakerState | undefined | Promise<BreakerState | undefined>;
  set(key: string, state: BreakerState): void | Promise<void>;
}

The critical wiring rule is per-client resolution (G11): with createClient, the breaker store is resolved once for the client and shared across every call — a fresh in-memory store per call could never accumulate the consecutive-failure state. The default is an in-memory Map per client. Both methods may return a Promise, so a multi-instance deployment can share breaker state across the fleet by pointing every instance at the same backend:

import { createClient } from '@deuz-sdk/core';
import type { BreakerStore, BreakerState } from '@deuz-sdk/core';

// e.g. Redis / Durable Object-backed; shared across calls and instances (G11).
const states = new Map<string, BreakerState>();
const breakerStore: BreakerStore = {
  get: (key) => states.get(key),
  set: (key, state) => {
    states.set(key, state);
  },
};

export const deuz = createClient({
  apiKeys: { anthropic: process.env.ANTHROPIC_API_KEY! },
  deps: { breakerStore },
});

Cross-provider failover (1.7)

Because the whole conversation is canonical (Message[]/Part[]), a chat can hop providers mid-conversation: the next candidate receives the identical request the failed one got — history, tools, and all. Two equivalent surfaces:

// 1. The call option — inline sugar:
const result = streamChat({
  model: anthropic('claude-opus-4-8'),
  messages,
  fallbackModels: [openai('gpt-5.2'), google('gemini-3-pro')],
  // onFallback is available on the middleware form below
});

// 2. The middleware — reusable wiring:
import { wrapModel, withFallback } from '@deuz-sdk/core/middleware';

const resilient = wrapModel(anthropic('claude-opus-4-8'), [
  withFallback([openai('gpt-5.2'), google('gemini-3-pro')], {
    onFallback: ({ from, to, error }) => log('failover', from.modelId, '→', to.modelId),
  }),
]);

The rules:

• Pre-first-content only. A streaming attempt is buffered until its first content part (text-delta, reasoning-delta, tool-call-delta, tool-call); a failure before that hops to the next candidate. Once content has arrived, a mid-stream error is final — the existing no-replay rule, for the same reason mid-stream retries are forbidden. Buffered (generateText) calls hop on any fallback-worthy rejection.
• What hops (defaultShouldFallback): BreakerOpenError (an open breaker fails fast, so the hop is immediate — this is the breaker/failover synergy), NetworkError, TimeoutError, and retryable / 5xx APICallErrors. Client errors (bad request, auth) never hop — they would fail identically everywhere. Override with the shouldFallback hook on the middleware.
• Retries come first. Each candidate still gets its own pre-first-byte retry budget; failover engages only after a candidate's final failure.
• capabilities follows the hop (2.3). A per-call capabilities override describes every candidate, not only the first. Before 2.3 a fallbackModels candidate fell back to its registry row, such as Anthropic's 4096-token placeholder max_tokens for an unknown slug; the withFallback middleware already passed the override on.
• The audit trail: the winning result carries providerMetadata.deuz.failedOver = { from, to, reason } (keys are provider:model strings, reason is the failing error's code). On streaming it rides the terminal finish part's providerMetadata. The onFallback hook fires once per hop.
• G2 holds: the streaming form returns synchronously; an exhausted candidate list surfaces as an error part + rejected promises. On streaming, durable identity is stable too — a missing session.runId is minted once before the first attempt (and exposed synchronously on the result), so every candidate checkpoints under the same run.
• Reasoning stays with its provider (2.3). Earlier turns' reasoning only the API that produced it can read back. Each wire replays its own family's blocks and leaves the rest in the history, so an OpenAI encrypted item never reaches a Claude fallback as a thinking block, and a Claude block never reaches Gemini. A hop between two Claude models forwards the blocks unchanged. See Reasoning stays with its provider.

Honest caveats: provider capabilities differ, so pick fallbacks whose capability profile covers your call (a tool-heavy call falling back to a model without tool support will fail on the fallback too — the registry's conservative defaults apply). Cost differs per model too — the loop (and the cost part) prices against the model that actually ran, not the primary you asked for.

Related

• streamChat — the synchronous, never-throws streaming entry point and StreamChatResult shape.
• Error Handling — the DeuzError taxonomy, isRetryable, and retryAfterMs.
• Dependencies & Clients — the Dependencies seam, clock, generateId, and per-client breaker resolution.
• Middleware — wrapModel and the withFallback ordering rules.
• Edge runtimes — why the core stays pure and Web-APIs-only.