OpenAI Embeddings Example
A demonstration project showing how to build hybrid search and retrieval-augmented generation with OpenAI embeddings stored in a Pinecone vector database and orchestrated with LangChain, including a PDF-loading pipeline.
This is an example project demonstrating retrieval-augmented generation and hybrid search using Python, OpenAI API embeddings, a Pinecone vector database index, and LangChain for orchestration. It demonstrates several individual techniques: using a system prompt to modify an LLM's text-completion behavior, keeping prompts DRY with templates, setting up a project with LangChain as an alternative to a vendor-specific LLM package, a command-line PDF loader that extracts text, vectorizes it, and loads it into a Pinecone index dimensioned to match OpenAI embeddings, and creating, loading, and querying that Pinecone database.
It also demonstrates, as a secondary goal, how to use Pydantic to manage project settings and handle sensitive credentials safely within a project. The README illustrates the effect of retrieval augmentation concretely by comparing a generic ChatGPT answer about a specific university course against an answer generated after embedding a batch of the course's own PDF lecture notes.
Setup uses a Makefile-driven virtual environment and requires OpenAI and Pinecone API keys in a local .env file.