MELLM — Lightweight Modular AI Routing Engine
MELLM routes a query to a small, domain-specific expert model instead of a single large general model, keeping only one specialist loaded in VRAM at a time so consumer GPUs can get expert-level answers.
MELLM runs a small classifier model that reads an incoming query, identifies its domain such as code, math, medical or legal, and loads the matching specialist model rather than a single large general-purpose model that has to cover every topic. When a query spans multiple domains, it is split and each part is routed to its own specialist.
The project argues this is not just a workaround for limited hardware: routing to purpose-built specialists is also cheaper and more accurate than always calling a very large general model, even at production scale. It targets consumer GPUs as small as 6GB, using 1.5 to 7 billion parameter specialist models loaded on demand and unloaded when idle.
It is written in Python, built on llama.cpp and GGUF models, and released under the MIT license.