Intelligent Model Routing Guide FinOps & Latency

Intelligent Model Routing allows organizations to dynamically optimize LLM model selection across connected AI agent harnesses. By classifying tasks and routing prompts to the most cost-effective and capable models using adaptive reinforcement learning, Intutic helps you achieve peak performance while minimizing token expenses.

---

How It Works

1. Proxy Interception: Every outbound prompt from your developer tools is routed through the proxy gateway.
2. Gateway Classification: The proxy performs high-speed (sub-5ms) keyword matching to classify the prompt into one of five task types: testing, deployment, review, debugging, or coding.
3. Thompson Sampling Selection: Intutic evaluates historical reward parameters ($\alpha, \beta$) for the (Model × SOP Tier × Task Type) Beta distribution to select the optimal model.
4. LLM-as-a-Judge Reward Feedback: Background LLMProbe workers audit trajectory outputs, evaluating response quality and SOP compliance. High-quality responses increment success parameters ($\alpha$), while failures increment ($\beta$) in Valkey.

 ┌────────────────┐       ┌─────────────────────┐       ┌─────────────────────┐
 │ Outbound Prompt│ ──1──>│  Thompson Sampling  │ ──2──>│  Selected Model     │
 └────────────────┘       │  Model Selection    │       │ (e.g., gpt-4o-mini) │
                          └──────────┬──────────┘       └──────────┬──────────┘
                                     ▲                             │
                                  4. Reward                        │ 3. Response
                                 Update (α, β)                     │
                                     │                             ▼
                          ┌──────────┴──────────┐       ┌─────────────────────┐
                          │  LLM-as-a-Judge     │ <─────│ Async Background    │
                          │  (LLMProbe Audit)   │       │ Trajectory Logger   │
                          └─────────────────────┘       └─────────────────────┘

---

Setup & Activation

Step 1: Enable Routing in the Dashboard

1. Open the Compute Metrics Dashboard (e.g., localhost:5174 or your local console at http://localhost:5174).
2. Navigate to Settings from the sidebar navigation.
3. Click on the Smart Model Routing & Response Cache tab.
4. Check the Enable Intelligent Model Routing option.

---

Step 2: Configure Custom Task Trigger Words

You can customize the words that trigger model redirection to fit your team's tech stack and vocabularies:

1. In the Intelligent Model Routing settings section, locate the keyword configuration fields.
2. Input comma-separated lists of trigger keywords for the following categories:
   • Testing: e.g., test, spec, vitest, jest, unittest, assert
   • Deployment: e.g., deploy, release, kubernetes, docker, gke, pipeline, ci/cd
   • Review: e.g., review, audit, lint, eslint, pr
   • Debugging: e.g., fix, bug, issue, error, crash, debug
3. Click Save Keywords to push the updates to Valkey.

NOTE

Custom keywords are validated at the API layer. Keywords must be alphanumeric strings (or ci/cd) and between 2 and 19 characters long.

---

Step 3: Route Agent Traffic

To route agent traffic, you must ensure your AI agent harnesses are connected to the Intutic proxy gateway:

Option A: Using the CLI (Recommended)

The Intutic CLI sync daemon scans and automatically updates the configurations of all supported harnesses in your local repository:

npm install -g @intutic/cli
intutic login
intutic init
intutic connect

Option B: Standalone Proxy Redirects

For custom agent configurations, point your agent's API base URL environment variables directly to the proxy gateway:

export OPENAI_API_BASE="http://localhost:4000/v1"
export ANTHROPIC_API_BASE="http://localhost:4000/v1"

Once connected, your prompts are automatically routed to the most optimal model based on local rules and current learning rates.