INTERMEDIATE · 8 MODULES · 64 LESSONS
Evaluation Frameworks
Find out with evidence whether your LLM assistant works: read its conversations, find where it fails, measure each failure with trustworthy evaluators, and decide when it's ready to launch.
- For you if
- You have an LLM assistant, RAG system, or agent and you evaluate it by eye today.
- By the end
- You'll know how to write an LLM judge per failure, validate it against a human expert, and add an eval gate to CI.
- You need
- Intermediate Python and experience calling an LLM from code. No API key needed.

Curriculum
01Error analysis: read before you measure
- Introduction: read first, measure second
- Traces: the raw material
- Reading a trace: where the failure starts
- Open coding: free notes and a verdict
- Axial coding: from notes to a failure taxonomy
- Counting and prioritizing: what to fix now and what deserves an evaluator
- Saturation and criteria drift: when to stop and when to review again
- Project: Ridgeline's failure report
02From failures to evaluators: code first
- Introduction: the cheapest evaluator is the one that needs no other model
- Choosing an evaluator for each failure
- Assertions over the whole trace
- Checking tool calls
- Reference facts and expected answers
- Why generic metrics mislead
- An eval harness in plain Python
- Project: the code evaluator suite and the v2 prompt
04Validating the judge against human labels
05Synthetic data and cold-start evals
06Evaluating by component: RAG, conversations and agents
Reviews
What students say
These reviews are from enrolled students who completed at least 50% of the course. We moderate reviews only on content grounds (spam, offensive language, personal data), never for being critical or negative.
No approved reviews yet.
Be the first to share your experience!