Semantic Caching

​

How It Works

1. Theo generates a deterministic cache key from the request parameters (prompt, mode, skills, tools)
2. Checks the server-side cache for a matching entry
3. On hit: returns the cached response immediately — zero cost, sub-10ms latency
4. On miss: runs the full orchestration pipeline, caches the result, and returns it

​

Cache Key Computation

• Prompt text (exact match)
• Mode (auto, fast, think, etc.)
• Active skill slugs
• Inline tool definitions

​

TTL Behavior

​

Cache Scope

• Cache is per-user — your cached responses are not shared with other accounts
• Cache is per-key scoped when org-level keys are used

​

Detecting Cache Hits

const res = await theo.complete({ prompt: "What is DNS?" });

if ((res as any)._cached) {
  console.log("Cache hit! Cost: 0");
} else {
  console.log(`Cache miss. Cost: ${res.usage.cost_cents}¢`);
}

​

Cost Impact