---
name: minirouter-evaluate-switch
description: Plans and verifies a bounded local model comparison with explicit approval before paid requests. Use when testing a model switch against representative tasks, cost limits and deterministic scoring.
compatibility: Node.js 22 or newer with filesystem access. Online commands require HTTPS access to minirouter.sh and api.minirouter.sh.
---

# Evaluate a model switch

**IS:** a local, reproducible comparison with a bounded paid execution step. **IS NOT:** automatic retries, free inference, routing changes or live tool execution.

Resolve paths against this installed SKILL.md directory and run the bundled CLI. Read [evaluation input and key requirements](references/input.md) before creating an evaluation. Use `examples/evaluation.json` as a deliberately non-runnable template, replacing model IDs, evidence ID and deadline from current advice and the user's tasks.

1. Prepare representative cases and deterministic scorers, a baseline and at most two candidates, exact efforts/providers, maximum requests/output, deadline, pass threshold and spend limit. Explain that retainOutputs keeps response evidence only in a private local directory.
2. Use an account with no enabled BYOK pools; do not change those pools automatically. Have the user provision a dedicated key through their existing Keys UI with the exact model allowlist, concurrency one, daily/monthly caps no greater than the run budget and expiry before the current UTC day ends. Set MINIROUTER_KEY locally. Planning reads key policy but spends nothing.
3. Run `node <skill-directory>/scripts/model-advisor.mjs evaluate plan --input <evaluation.json> --out <new-output-directory>`. Present the concrete manifest, its hash, configurations and maximum spend. Paid execution requires the user's explicit approval of that manifest hash. An existing matching approval is sufficient; changing the manifest requires a new approval.
4. After approval, run `node <skill-directory>/scripts/model-advisor.mjs evaluate run --input <evaluation.json> --manifest <directory>/manifest.json --approve <approved-hash>`.
5. Run `node <skill-directory>/scripts/model-advisor.mjs evaluate verify --artifact <directory> --input <evaluation.json>`. Report pass counts, sample size, observed latency, settled/pending charges and reproducibility. Include proposed configuration and rollback configuration; do not apply either.

Continue through execution and verification when the manifest is approved. Otherwise finish with the concrete plan awaiting that approval.

## Gotchas

- A timeout or unknown outcome stops the run. Restart does not replay an attempt; never retry an ambiguous paid request manually.
- Caps include outstanding holds. A local price estimate is not a spend cap; changed key policy invalidates the plan.
- Missing/404 generation records do not mean zero cost. Preserve unresolved attempts.
- No routers, media, BYOK, server tools or paid model judges. Tool-call scorers inspect returned structures without executing actions.
- Without required local evidence, verification reports quality-unverifiable. Small samples do not establish p95 latency.

## Install this release

Download the [complete skill archive](https://minirouter.sh/agent-skills/releases/83b1177c2858898aef4af631fd5d5060d3afcac0c17b4719d19370b084d57721/minirouter-evaluate-switch.tar.gz) and [release manifest](https://minirouter.sh/agent-skills/releases/83b1177c2858898aef4af631fd5d5060d3afcac0c17b4719d19370b084d57721/manifest.json). Verify the archive SHA-256 against the manifest before extracting into a new local directory. Keep the full extracted skill folder; SKILL.md alone does not include the executable. Run `node <skill-directory>/scripts/model-advisor.mjs --help` to verify Node 22+ installation.
