MemTrace
A diagnostic tool that traces why an LLM memory system produced a wrong answer, modeling extraction, retrieval and generation as an operation-variable execution graph.
MemTrace helps developers understand why an LLM memory system gives a wrong answer. A memory system reads messages, extracts facts, updates or deletes stored memories, retrieves relevant memories, and generates an answer; when that answer is wrong, the real cause is often hidden in an earlier step, such as a missed fact, an overwritten memory or a wrongly retrieved one.
It records that whole process as an operation-variable execution graph, where variables are the concrete things produced during execution, such as messages, memories, retrieved results, prompts and predictions, and operations are the steps that create or use them. Given this graph, MemTrace traces a failed case backward and identifies which operation most likely introduced the error.
The project includes a companion toolkit for recording execution graphs from existing memory systems, a benchmark dataset of human-annotated failure cases, and an annotation interface, and is released under the MIT license.