WebLLM
An in-browser large language model inference engine that runs open source models directly in a web browser using WebGPU acceleration, with an API compatible with OpenAI's.
WebLLM is an in-browser inference engine for large language models. It runs entirely inside the browser, with no server-side component, using WebGPU for hardware acceleration.
It exposes an API compatible with OpenAI's, so the same client code can run open source models locally, with support for streaming, JSON-mode structured output and function calling. It natively supports models such as Llama 3, Phi 3, Gemma, Mistral and Qwen, and custom models can be deployed in the MLC format.
WebLLM is distributed as an npm package or by CDN and can be embedded in a web application; it is a companion project to MLC LLM, which handles universal LLM deployment across hardware environments.