Evaluating LLM Features With Offline Test Sets and Online Metrics
Quick answer: evaluate LLM features in two stages. Offline, run a curated test set against every candidate change and grade results with programmatic checks,...
Blog
All troubleshooting articles, newest first.
LLM Cost Control With Prompt Caching and Output Length Limits
Quick answer: LLM cost is mostly tokens multiplied by price, so reduce both. Measure cost per feature first, then trim prompts, retrieve fewer and better...
Human-in-the-Loop Design for AI Agent Actions With Side Effects
Quick answer: let an AI agent act freely on read-only and easily reversible tasks, and require human approval for actions that send messages, move money,...
LLM Observability: What to Log and Trace in Production AI Features
Quick answer: log every LLM call as a trace with the prompt version, model and parameters, redacted inputs and outputs, retrieved document IDs, tool calls,...
Fine-Tuning vs RAG vs Prompting: Choosing an Approach for Domain Knowledge
Quick answer: start with prompting, because it is the cheapest to try. Use retrieval-augmented generation when answers depend on private, fresh, or frequently...
Dedicated Vector Stores vs Relational Databases With Vector Search
Quick answer: if your data already lives in a relational database and your vector count is modest, adding vector search there is usually simpler and keeps data...
LLM Rate Limit Handling: Retries, Backoff, and Queueing at Scale
Quick answer: LLM providers limit both requests and tokens per minute, and exceeding them returns rate limit errors. Retry only retryable errors, use...
Token Budgeting and Context Window Management in Long Conversations
Quick answer: every model has a limited context window, and long conversations eventually exceed it or become expensive. Set a token budget for each part of...
Embedding Model Versioning and Reindexing Strategies for Vector Search
Quick answer: vectors produced by different embedding models, or different versions of one model, live in different spaces and cannot be compared. Store the...