Evals
Systematic evaluation of LLM and agent behaviour — test sets, graders and regression checks that make non-deterministic systems shippable.
Evals are automated tests for LLM systems: curated inputs, expected behaviours and graders (rules, reference answers or a model acting as judge) run on every change, so prompt, model or tool updates can be compared instead of guessed at.
How I use it
- Conversational AI Agents — automated evals are part of the multi-tenant agent platform in my Independent AI & Full-Stack Engineer (Freelance) work, alongside Langfuse / OpenTelemetry tracing.
- local-rag-chatbot — a built-in LLM grading step judges whether retrieved context is enough to answer before generating, which is the same pattern used in RAG evals.
- Offline-testable agents — Akashi, langgraph-langserve-starter and langgraph-ts-agents all run their test suites on fake models, so behaviour is checked deterministically in CI without API keys.
Related
-
AI Agent Products & Remote MCP — production use in my freelance work
-
LangSmith · Langfuse · LLMs · RAG (Retrieval-Augmented Generation)