Generative AI Knowledge Base
A Terraform module that deploys an end-to-end pipeline for extracting question-and-answer pairs from documents and fine-tuning a large language model on them using Google Cloud Vertex AI.
This solution is a Google Cloud sample showing how to extract question-and-answer pairs from documents with generative AI and use them to fine-tune a large language model on Vertex AI. Uploading a new document triggers a Cloud Function webhook, which uses Document AI's OCR to extract the document's text and indexes that text in Vector Search.
A Vertex AI large language model then generates candidate question-and-answer pairs from the extracted text, which are saved to Firestore; after a human validates them, a fine-tuning dataset is generated from that Firestore data, and the fine-tuned model is deployed and saved in the Vertex AI Model Registry. The Terraform module provisions the underlying Google Cloud project resources for this pipeline, including the storage buckets, Vector Search index and endpoint, Firestore database, and Document AI processor, and reports configuration in about two minutes and deployment in about six.