context-graph-benchmark
This benchmark compares three memory architectures for multi-agent LLM systems - raw conversation history, vector-only RAG, and a structured context graph - on questions that require combining facts from different points in a conversation, using no LLM calls.
Vector similarity search can retrieve a chunk that looks like a query, but it has no mechanism for combining two separately stated facts from different points in a conversation into a single answer; the benchmark measures exactly where each memory architecture's ceiling is on that specific kind of question. All three architectures are implemented in pure Python with zero API calls, and a benchmark harness runs each against five scripted multi-agent scenarios, grading answers by deterministic substring match against hand-authored ground truth across eighteen graded queries.
Reported results show the context-graph approach reaching a higher accuracy than either the raw-history or vector-only approaches, at a fraction of the average tokens per query, and reproduced identically on two separate machines. A companion article explains the reasoning and the measurement approach in more depth.
It is written in Python and released under the MIT license.