PaLM + RLHF - Pytorch
A PyTorch implementation of Reinforcement Learning with Human Feedback (RLHF) on top of the PaLM architecture, aimed at replicating a ChatGPT-style training pipeline in the open.
This project implements Reinforcement Learning with Human Feedback (RLHF) on top of the PaLM architecture, intended for those interested in replicating something like ChatGPT in the open. It first trains a PaLM autoregressive transformer, then trains a reward model from curated human feedback, and passes both to an RLHFTrainer; fine-tuning the reward model can optionally use LoRA.
It installs with pip install palm-rlhf-pytorch, and its README notes a potential successor approach, Direct Preference Optimization, under which the repository's core logic reduces to roughly a binary cross-entropy loss. Outstanding work tracked in the project's todo list includes adding Hugging Face accelerate and wandb instrumentation.