Retrieval augmented generation for building datasets from scientific literature
A pair of Colab notebooks demonstrating a retrieval-augmented generation workflow for extracting structured datasets from scientific paper abstracts, applied here to hydrogen-storage properties of metal alloys and hydrides.
This repository contains the notebooks used to build a dataset of hydrogen-storage properties for metal alloys and hydride compounds by extracting structured data from paper abstracts with retrieval-augmented generation. The RAG_new.ipynb notebook runs the extraction pipeline, and Prompting.ipynb loads a fine-tuned model and prompt for the same task; a preprint describing the method and dataset is linked from the repository.
The RAG notebook installs Ollama in a Google Colab terminal, pulls and serves Llama 3 locally, loads a CSV of selected abstracts, and runs a defined extraction query against them. The prompting notebook instead loads a 4-bit quantized unsloth/llama-3-8b-bnb-4bit checkpoint through FastLanguageModel and applies a prompt to the same abstracts; both notebooks are meant to be run on a Colab T4 GPU. The resulting dataset was also tested as training data for downstream machine-learning models predicting alloy properties.
Built with Python as Jupyter and Colab notebooks, using Ollama and the Unsloth fine-tuning library.