GPTCache
A library for building a semantic cache of LLM responses, so repeated or similar queries are served from cache instead of the model, cutting API cost and latency.
GPTCache is a library for creating a semantic cache for LLM query responses, aimed at reducing the cost and latency of applications that make frequent calls to services such as ChatGPT as traffic grows. It is fully integrated with LangChain and llama_index, and a Docker server image lets any language use it as a service, not only Python.
By default only a limited set of libraries is installed to support basic caching, with additional dependencies installed automatically as more features are needed. The project recommends its get and set API for new integrations, since it no longer adds support for new model-specific adapters given how quickly the model API landscape changes.