Support RLHF and other instruction fine-tuning options beyond supervised fine-tuning.

[divyashreepathihalli]

[sandeshkatakam]

Hi, I'd love to contribute to adding RLHF and instruction fine-tuning support to Keras Hub. Could you clarify the scope? Are we focusing on specific model architectures and dataset standards? Will we use parameter-efficient methods (e.g., LoRA), and what instruction fine-tuning families are in focus?
I propose starting with a Trainer class and Config classes for RLHF methods (PPO, DPO, DDPO, GRPO, CPO), including testing, docs, selective layer updates/adapters, RLHF-trained weights for Keras Hub, and tutorial notebooks.
I tried a PoC with GPT-2, a BERT reward model, and PPO in native Keras but ran into tokenizer mismatches with the Anthropic hh-rlhf dataset.
Details are in my Colab notebook ( Integrating RLHF into Keras Hub.ipynb).Happy to refine the PoC based on feedback.
Does this align with your vision? If so, could you assign the issue to my GitHub handle?

[divyashreepathihalli]

Hi @sandeshkatakam we have not scoped this out yet. It would be great if you can add a keras io guide or example and we can probably pull that into the code if it works out.

[sandeshkatakam]

Hi @sandeshkatakam we have not scoped this out yet. It would be great if you can add a keras io guide or example and we can probably pull that into the code if it works out.

Yeah sure, I will work on that!