How It Works
The evaluation system runs predefined prompts through the agent and scores the output using both deterministic checks and optional LLM-based judging. Each evaluation case specifies:- An input question (what the user asks)
- Expected behavior (which tools should be used, what keywords should appear in the answer, whether citations are expected)
- Optionally, a reference answer for LLM-based quality scoring
Evaluation Criteria
Deterministic Checks
These checks produce pass/fail results:LLM Judge (Optional)
When a reference answer is provided and a judge model is configured, an LLM evaluates the response for correctness and completeness on a 0.0-1.0 scale. Judge failures are logged but do not fail the evaluation.Evaluation Categories
Evaluations cover a range of agent behaviors:Observability
When an observability platform (Langfuse) is configured, evaluation results are synced as dataset experiments with per-item traces and scores. This provides:- Historical tracking of agent quality over time
- Per-query traces showing exactly what the agent did
- Score trends across model changes or prompt updates