Goinfer
A local LLM inference proxy built on llama.cpp and llama-swap that lets you load and hot-swap multiple GGUF models and serve them through an OpenAI-compatible HTTP API, with an outbound-connection design for secure remote access to home or office GPUs still on the roadmap.
Goinfer is an inference proxy for swapping between multiple GGUF models and exposing them through an HTTPS API with credentials, built on top of llama.cpp and llama-swap and aimed at being DevOps-friendly to deploy and monitor on remote machines with minimal manual operation.
It loads multiple GGUF models and switches between them at runtime, changing inference parameters as needed; exposes both an OpenAI-compatible /v1/ HTTP API and a llama.cpp-compatible /completions API with streaming responses; supports API-key authentication and CORS control; and offers remote monitoring, uploading or deleting GGUF files, reloading configuration, and rebuilding llama.cpp from the admin side.
The README describes, as a problem not yet implemented, flipping the usual connection direction so a GPU-rich client at home or in an office initiates an outbound connection to a server with a static IP, which then forwards inference requests back to it without inbound ports, VPNs or a custom client on every device. Built with Go.