A project to build with System One models

A cascade that answers with a small model first and escalates only when unsure

Answer every request with a decision model first, send only the uncertain ones to an LLM, and measure how much time and money it saves.

LLM routing and guardrailsAdvanced4 stepsSuggested by @biplov

The decision

The same typed question your LLM answers today, whether it is a classification, a yes/no or a rating, asked first of a decision model. When its top probability clears the threshold, that is the answer; otherwise the LLM answers.

How to build it

Pick one LLM task in your product that has a fixed set of answers.
Run the decision model and the LLM side by side on a week of traffic, and log both answers.
Find the threshold where the decision model agrees with the LLM, for example 99% of the time.
Switch: the decision model answers above the threshold, and the LLM handles the rest.

Make it better

Publish the numbers: the share of traffic that never reached the LLM, the latency, and the cost per 1,000 requests. That is the case for decision models in one chart.

More projects to build

All projects

LLM routing and guardrailsIntermediate

A guardrail that checks every tool call before an agent runs it

Ask a decision model "is this command safe to run?" before a coding or ops agent executes it, and block or ask for approval below a threshold.

jared-palmer/kev mapika/decider alibi-serikbay/jevk5

LLM routing and guardrailsIntermediate

Route each prompt to the cheapest LLM that can answer it

A decision model reads the prompt and picks the smallest model likely to handle it, cutting LLM spend without hurting quality.

vllm-semantic-router/decision jared-palmer/kev convai-innovations/laya

LLM routing and guardrailsIntermediate

A RAG router that decides whether to retrieve, and from which index

Before a retrieval runs, a decision model reads the question and picks the index that holds the answer, or decides none is needed, so the LLM reads less noise and you run fewer searches.

convai-innovations/laya vllm-semantic-router/decision supersonic-labs/julia-1