# Agentic AI Evaluation Bootcamp

Reference implementations from Vector Institute's Agentic AI Evaluation Bootcamp, covering agent evaluation fundamentals and four example agents evaluated with LLM-as-judge metrics, batch evaluation and a misalignment probing experiment across several commercial models.

This is a collection of reference implementations for Vector Institute's Agentic AI Evaluation Bootcamp, covering different aspects of building and evaluating agent-based systems: introductory notebooks on why evaluation is hard, the four quality dimensions, grader types, and a shared evaluation harness built on Langfuse.

A ReAct agent using Google ADK and Google Search answers questions grounded in live web content, evaluated on the DeepSearchQA benchmark with LLM-as-a-judge metrics. An anti-money-laundering investigation agent queries a SQLite database of financial transactions through a read-only SQL tool and produces structured analysis with batch evaluation. A report-generation agent turns natural-language queries into downloadable Excel reports from a relational database, with a Gradio demo and Langfuse-integrated evaluation. A misalignment probing module is a YAML-driven experiment runner that tests whether reckless examples can nudge model responses toward harmful behavior, across five context-injection conditions, six commercial models and three task categories, storing traces and scores in Langfuse.

Built with Python, using Langfuse, Google ADK, Gradio and uv for dependency and environment management.

Website: https://github.com/VectorInstitute/eval-agents