Failover and the first byte

We route between hosts of the same model, never between models.

What failover is here

If the preferred host times out or errors before your response starts, the request moves to the next healthy host. You see a slower first token and nothing else — same model, same id. We never substitute a different model, however similar or cheaper. The x-minirouter-provider header on every response names who actually served it.

The honest constraint — the first byte

Before the first content byte reaches you, a retry is free: nothing has been delivered. Once the first byte is sent, provider switching is impossible — splicing a second provider’s tokens into a half-delivered completion corrupts output, tool-call framing, and your client’s parser state.

  • EventTTFT timeout
    Before first byteAbort, retry next healthy route
    After first byteCannot occur — first byte arrived
  • EventUpstream 500
    Before first byteRetry; you pay only the successful attempt
    After first byteTerminal error; settle what was produced
  • EventFallback chain
    Before first byteLive — full route list available
    After first byteClosed — the route is committed

You pay only for the attempt that succeeded. After the first byte, an upstream failure ends the stream with a terminal error; under 50 output tokens, the charge is written off entirely.

Why TTFT is the health signal

Time to first token is the only latency measured while we can still act on it, so it drives route health — and it is the number published per upstream on every model page.

Errors on this path

  • request_timeout — no first token from any upstream within the window. Safe to retry.
  • upstream_error — the upstream failed after streaming began, past the point of retry.
  • model_unavailable — every upstream failing; the error names working alternatives an agent can act on.