Local coding agent evals
A set of small practical scripts and task packs for comparing local language models and coding-agent harnesses, measuring raw inference speed, tool-choice reasoning and whether an agent can finish real coding tasks.
Local coding agent evals is a small collection of scripts and task packs the author used while comparing local models and coding-agent harnesses such as Qwen Code, Codex and Claude Code running against Ollama-hosted models, published to accompany a written article rather than as a comprehensive benchmark.
It has three parts, each testing a different layer: a speed and memory benchmark measuring Ollama's prefill speed, decode speed, wall time and memory use across longer prompts, testing the model and inference engine together; a hard tool-reasoning benchmark that asks a model to return one tool decision for five harder reasoning tasks without executing them, testing the model's base capability; and an agent problem pack of five small coding tasks run against a real coding-agent harness in an isolated workspace, testing the model and harness combination together. Because a coding agent can read files outside its intended scope, the author recommends running the agent benchmarks in an isolated workspace, a separate user account, a virtual machine, or a separate machine.