Working Locally with Open Weight LLMs
Course material for an O'Reilly live training on running open-weight decoder LLMs locally, with notebooks comparing the transformers library, vLLM, SGLang, tabbyAPI and llama.cpp as execution engines.
This repository holds material for an O'Reilly live course on working locally with open-weight, GPT-like decoder LLMs, focused on the common use case of text completion and question answering.
Slides give background and theory, and Jupyter notebooks run the same models, Qwen and Gemma variants, through several execution paths for comparison: the transformers library, vLLM as a separate server reached over the OpenAI API, SGLang including quantized and multi-token-prediction variants, tabbyAPI with exl3 quantization, and llama.cpp; the notebooks double as documentation of how the author ran them.
Built with Jupyter notebooks; also points to LM Studio and Ollama as CPU-friendly options for running LLMs locally, noting neither is fully open source.