---
title: "Evaluation API"
description: "Jev evaluation API: typed questions, native TypeSafe format and pricing."
canonical_url: "https://minirouter.sh/docs/evaluations"
markdown_url: "https://minirouter.sh/docs/evaluations.md"
last_updated: "2026-09-19"
---

# Evaluation API

Evaluate shared state against typed questions and receive probabilities, choices and scores.

## Availability

Jev is available when it appears in the [evaluation catalog](https://minirouter.sh/models?evaluationOnly=true) and your API key permits it. If no supported route is available, requests return HTTP 503 before a credit hold or provider call. Authenticated native discovery lists the models your key may use.

## Endpoints

| Method | Path | Format |
| --- | --- | --- |
| POST | /v1/evaluate | Boolean, choice and score questions |
| POST | /typesafe/v1/systemone | Native TypeSafe noul, choice and score questions |
| GET | /typesafe/v1/models | Authenticated model discovery |

Use a MiniRouter integration API key as a bearer token and explicitly set `model` to `typesafe-ai/jev`. Chat endpoints, native aliases and omitted model defaults are unsupported.

## First request

```sh
curl --fail-with-body https://api.minirouter.sh/v1/evaluate \
  -H "Authorization: Bearer $MINIROUTER_API_KEY" \
  -H 'Content-Type: application/json' \
  --data '{
    "model": "typesafe-ai/jev",
    "state": "The subscription was charged twice.",
    "questions": {
      "billing": {
        "type": "boolean",
        "instructions": "Does this describe a billing problem?"
      }
    }
  }'
```

A successful primary response has this shape:

```json
{
  "model": "typesafe-ai/jev",
  "answers": {
    "billing": { "type": "boolean", "probability": 0.98 }
  },
  "usage": { "inputTokens": 283, "outputTokens": 21 }
}
```

These are illustrative values, not a live result.

## Question types

- **Boolean:** use `type: "boolean"` and instructions. Optional criteria describe the true and false cases. The answer contains a probability from 0 to 1.
- **Choice:** use `type: "choice"` with a criteria object mapping option names to descriptions. The answer contains the selected choice and each option's probability.
- **Score:** use `type: "score"` with at least two ordered descriptions in a criteria array. The answer contains an interpolated score and each level's probability.

State and instructions accept strings, objects or arrays. A request accepts up to **64 questions**, **64 options per choice question**, and **2–10 levels per score question**. The normalized JSON sent to the provider must fit within **32 KiB (32,768 UTF-8 bytes)**, including routing fields added by MiniRouter. Keep payloads below this ceiling to leave room for those fields. These are request limits, not a token estimate.

## Native TypeSafe format

Send native requests to `/typesafe/v1/systemone` on `https://api.minirouter.sh`. Use `noul` in place of boolean; its answer uses `{ "type": "noul", "noul": 0.98 }`. Native usage fields are `input_tokens` and `output_tokens`. Choice and score answers include confidence, and score also includes the requested level legend.

Native errors use `{ "message": "…", "error_type": "…" }`. Invalid request shapes return 422; malformed JSON returns 400. Missing or invalid MiniRouter keys return 401. Native model discovery requires authentication and lists only enabled models your key may use.

## Pinned SDK examples

These examples use JavaScript SDK **0.6.0** and synchronous Python SDK **0.7.0** with the native TypeSafe endpoint. Use the request fields and question formats documented above.

Install `@typesafe-ai/sdk@0.6.0`, set `MINIROUTER_API_KEY`, and run this JavaScript as an ES module:

```js
import { TypeSafeClient, noul } from '@typesafe-ai/sdk';

const client = new TypeSafeClient({
  apiKey: process.env.MINIROUTER_API_KEY,
  baseURL: 'https://api.minirouter.sh/typesafe',
  retry: { maxRetries: 0 },
});
const result = await client.systemOne({
  model: 'typesafe-ai/jev',
  state: 'The subscription was charged twice.',
  questions: { billing: noul('Does this describe a billing problem?') },
});
console.log(result);
```

For Python, install `typesafe-sdk==0.7.0` and set the same environment variable:

```python
import os
from typesafe_sdk import TypeSafeClient, RetryPolicy, Noul

with TypeSafeClient(
    api_key=os.environ['MINIROUTER_API_KEY'],
    base_url='https://api.minirouter.sh/typesafe',
    retry=RetryPolicy(max_retries=0),
) as client:
    result = client.system_one(
        model='typesafe-ai/jev',
        state='The subscription was charged twice.',
        questions={'billing': Noul(instructions='Does this describe a billing problem?')},
    )
    print(result)
```

Keep the explicit model and zero-retry setting; do not override retries on individual requests. `X-Should-Retry: false` does not disable SDK retries. Two explicit calls remain two separate requests and may produce two charges.

Include question instructions and keep state and questions within the request limits. Native aliases, omitted instructions, structured criterion descriptions and extra request fields are unsupported. The Python example uses the synchronous client. Pointing an AI SDK Gateway client directly at MiniRouter is unsupported.

## Pricing and settlement

The final charge is verified provider inference cost plus **10%**, rounded once to nano-USD. Read the standard input, output and request rates in the [model catalog](https://minirouter.sh/models?evaluationOnly=true). Catalog prices already include the markup. A verified zero-cost request has a zero customer charge. Pending or missing cost does not mean free. Provider promotions may change; no free period is guaranteed.

A credit hold reserves **65,536 tokens** of input at the standard tariff, even for a smaller request; it is not a minimum bill. The unused amount is released after settlement. The response header `X-Minirouter-Billing-State: pending` means final provider cost is still being reconciled. Read the settled charge in account activity. Output usage remains counted even when output is free.

## Retries and unsupported fields

Disable automatic client retries. Each new incoming request has a separate billing identity. A timeout or lost response can still incur provider cost. Each accepted request makes at most one upstream attempt; `X-Should-Retry: false` is advisory and does not disable a library's retry policy.

Streaming, chat parameters, fallback models, caller providerOptions, duplicate JSON keys and unknown request fields are rejected. An invalid provider response can still incur a charge. MiniRouter does not automatically retry it.

## References

- [Vercel evaluation API](https://vercel.com/docs/ai-gateway/modalities/evaluation)
- [TypeSafe API reference](https://docs.typesafe.ai/api)
- [MiniRouter authentication](https://minirouter.sh/docs/authentication)
- [MiniRouter billing](https://minirouter.sh/docs/billing)