# glasshouse

A community-run registry for AI agent memory-system benchmark results, requiring per-question evidence records, a fixed reader, judge and prompt set per version, and reproducible harnesses instead of self-reported scores.

glasshouse is a place to publish memory-benchmark results, currently around LoCoMo, LongMemEval and BEAM, where anyone can check them. It was created because the same system run on the same benchmark by different parties has produced very different numbers, and because these benchmarks otherwise have no shared leaderboard or submission process even though the memory systems they measure sit inside other products' AI.

Submissions must publish the per-question record so the aggregate can be recomputed rather than just claimed; each benchmark version fixes its reader, judge and prompts so submitters cannot choose them; a submission must use a harness a customer could actually reach, such as the product's own SDK, and disclose its runs and harness; three runs count as a result and five as certified; company proposals need agreement from three other companies and individual proposals need public support, deliberately without a numeric threshold; and each new version stays fixed for at least fourteen days before the maintainer publishes on it.

Hosted by Wontopos, which also builds memory infrastructure and so is a competitor among the systems being measured; its own submissions go through the same approval as everyone else's, and it says it does not decide who is admitted. Built with Python. Apache-2.0 licensed.

Website: https://github.com/wontopos/glasshouse