node-llama-cpp
Node.js bindings for llama.cpp that run language models locally on Metal, CUDA or Vulkan hardware, with pre-built binaries, JSON-schema-constrained output, function calling and embedding support.
node-llama-cpp provides Node.js bindings for llama.cpp so language models can run locally inside a JavaScript or TypeScript application, with pre-built binaries for macOS, Linux and Windows and an automatic fallback to compiling from source with cmake if no binary matches the platform, requiring neither node-gyp nor Python. It adapts to the available hardware automatically, with Metal, CUDA and Vulkan acceleration, and a single CLI command downloads and compiles the latest llama.cpp release, or starts a chat session with a model without writing any code.
Beyond running inference, it can enforce a JSON schema on a model's output at the generation level, provide the model with callable functions to retrieve information or perform actions, generate embeddings and reranking scores, and support structured decision output with any model, while guarding against special-token injection attacks in user input.
It is written in TypeScript and released under the MIT license.