The Execution Trace Telemetry Model

Intutic monitors, accounts for, and evaluates every AI agent request through a structured, append-only telemetry model known as the Execution Trace.

---

Telemetry Schema

Every trace tracks token counts, routing tiers, active cost profiles, anomalies, and policy enforcement decisions in the database:

interface ExecutionTrace {
  traceId: string;               // Unique trace ID prefixed with 'tr_'
  sessionId: string;             // Owning agent session ID
  requestId: string;             // Unique ID generated per LLM exchange
  timestamp: Date;               // When the request occurred
  
  // Model Routing & Selection
  requestedModel: string;        // Model requested by the agent harness
  actualModelRouted: string;     // Model actually used (after optimization routing)
  routingTier: string;           // Selected routing tier ('cost', 'quality', 'speed')
  complexityScore: number;       // Inferred task complexity score (0.0 to 1.0)
  
  // Token Auditing
  rawInputTokens: number;        // Original tokens sent
  compressedInputTokens: number; // Tokens sent after prompt optimization compression
  outputTokens: number;          // Tokens generated by the model
  reasoningTokens?: number;      // Reasoning/thought tokens (e.g. for o1/o3 models)
  
  // Cost Accounting (Append-Only)
  rawCostUsd: number;            // Projected cost under raw model routing
  actualCostUsd: number;         // Actual cost incurred under the routed model
  savingsUsd: number;            // Total cost saved (rawCostUsd - actualCostUsd)
  cacheHit: boolean;             // Whether semantic prompt cache was hit
  cacheSavingsUsd: number;       // Savings generated via cache hits
  
  // Token Utility & Classification
  tokenUtility: 'USEFUL' | 'WASTED' | 'AMBIGUOUS';
  tokenUtilityScore: number;     // Utility index value (0.0 to 1.0)
  wasteCategory: 'NONE' | 'TOKEN_WASTE' | 'LOOP_WASTE' | 'RETRY_WASTE' | 'CONTEXT_BLOAT' | 'MODEL_MISMATCH';
  
  // Enforcement & Compliance
  complianceScore: number;       // Trust/rule alignment score (0.0 to 1.0)
  enforcementAction: 'ALLOW' | 'BLOCK' | 'HIJACK' | 'KILL' | 'BYPASS';
  
  // Anomalies
  anomalyDetected?: 'TOKEN_WASTE' | 'LOOP_DETECTED' | 'HALLUCINATION' | 'SECURITY_VIOLATION';
  anomalyConfidenceScore?: number;
}

---

Cost Calculations

Incurred and saved costs are dynamically calculated using the prices configured in the model registry:

Pricing Formulas

Costs are calculated by multiplying token usage against pricing weights:

$$\text{Cost}{\text{in}} = \text{Input Tokens} \times \text{Price per Input Token}$$ $$\text{Cost}{\text{out}} = \text{Output Tokens} \times \text{Price per Output Token}$$ $$\text{Actual Cost} = \text{Cost}{\text{in}} + \text{Cost}{\text{out}}$$

Cost Savings

Cost savings are generated in two ways:

1. Dynamic Routing Savings: Saved by routing tasks to a less expensive, yet functionally capable model instead of the requested premium model: $$\text{Savings}_{\text{routing}} = \text{Raw Cost} - \text{Actual Cost}$$
2. Semantic Cache Savings: Saved when the semantic prompt cache is hit, resolving the query locally without making external calls: $$\text{Savings}_{\text{cache}} = \text{Cache Hit Savings}$$

---

Token Utility & Waste Classification

Every trace is processed by a deterministic 5-rule heuristic engine to classify token utility. If confidence for a rule falls below 0.7, the trace is flagged as AMBIGUOUS for manual or LLM-as-a-judge follow-up.

The Classification Pipeline

[ Incoming Trace ]
        │
        ├── Rule 1: Was request KILLED? ────────────> Yes: WASTED (Confidence 1.0)
        │
        ├── Rule 2: Anomaly detected? ──────────────> Yes: WASTED (Confidence 0.85)
        │
        ├── Rule 3: Output > 150% of baseline? ─────> Yes: WASTED (Confidence 0.80)
        │
        ├── Rule 4: Expensive model, tiny task? ────> Yes: AMBIGUOUS (Model Mismatch)
        │
        └── Rule 5: Low rule alignment compliance? ──> Yes: AMBIGUOUS (Token Waste)
                │
                └── Default ────────────────────────> USEFUL (Confidence 0.90)

Rule 1: Kill Enforcement

If the trace shows an enforcementAction of KILL, the request was blocked mid-run by security policies. Zero value was generated, so all tokens are classified as WASTED with a utility score of 0.0.

Rule 2: Anomaly Detections

If anomaly patterns such as LOOP_DETECTED or HALLUCINATION are flagged on the trace, the utility is set to WASTED and the category is marked as LOOP_WASTE or TOKEN_WASTE.

Rule 3: Baseline Overshoot

The system tracks rolling token averages in Valkey (tok:baseline:*) grouped by model, task type, and prompt length bucket. If a trace output exceeds the 95th percentile baseline by $>150%$, the extra tokens are flagged as TOKEN_WASTE.

Rule 4: Model Mismatch

If an expensive reasoning model (such as gpt-4o or o1) is utilized for a trivial task ($<500$ input tokens, $<200$ output tokens), it triggers a MODEL_MISMATCH warning. The trace is marked AMBIGUOUS to flag potential optimization opportunities.

Rule 5: Low Compliance

If alignment compliance scores fall below 0.5, the tokens are flagged as potential TOKEN_WASTE since they were spent on non-SOP aligned activities.