llm-eval-layer
A pure-Python decision engine that scores LLM responses for attribution, specificity, relevance and context quality to detect hallucinations and output an accept, review or reject verdict.
llm-eval-layer is a decision engine for LLM responses that goes beyond retrieving documents and calling a model: it decides whether an output should be served, retried or regenerated, and explains why. It splits faithfulness into attribution (grounding) and specificity (concreteness) so it can tell a hallucination, which pairs high specificity with low attribution, from a merely weak answer.
Separate modules score relevance through semantic similarity, best-sentence match and token overlap, and context precision and recall against retrieved documents, with an optional paraphrase-stability consistency check. A pipeline module runs an eight-gate decision with a confidence score, a regression module stores baselines and blocks deployment when quality drops, and an optional OpenAI-backed judge only fires for scores in an uncertain 0.45-0.65 zone, at zero API cost in the default mode.
Built with pure Python and PyYAML, with optional sentence-transformers for semantic scoring, falling back to TF-IDF vectors when it is absent. MIT licensed.