Agentic AI Evaluation Bootcamp
Reference implementations from Vector Institute's Agentic AI Evaluation Bootcamp, covering agent evaluation fundamentals and four example agents evaluated with LLM-as-judge metrics, batch evaluation and a misalignment probing experiment across several commercial models.
This is a collection of reference implementations for Vector Institute's Agentic AI Evaluation Bootcamp, covering different aspects of building and evaluating agent-based systems: introductory notebooks on why evaluation is hard, the four quality dimensions, grader types, and a shared evaluation harness built on Langfuse.
A ReAct agent using Google ADK and Google Search answers questions grounded in live web content, evaluated on the DeepSearchQA benchmark with LLM-as-a-judge metrics. An anti-money-laundering investigation agent queries a SQLite database of financial transactions through a read-only SQL tool and produces structured analysis with batch evaluation. A report-generation agent turns natural-language queries into downloadable Excel reports from a relational database, with a Gradio demo and Langfuse-integrated evaluation. A misalignment probing module is a YAML-driven experiment runner that tests whether reckless examples can nudge model responses toward harmful behavior, across five context-injection conditions, six commercial models and three task categories, storing traces and scores in Langfuse.
Built with Python, using Langfuse, Google ADK, Gradio and uv for dependency and environment management.