What to check If the Reward Curve is not Expected When Training Multimodal Models in OPD
A note on what to inspect when the reward doesn’t improve very significantly in OPD training.
Table of Contents
1. Problem Statement
It happens during my first PR#745 of verl-omni, submitting a LTX2.3 DiffusionOPD snippet on v1 trainer. The situation is: the reward of the student doesn’t improve too much (from step 0’s 0.16 to step 100’s 0.25); the teacher I used was trained with 50 steps with v0 trainer.
2. What to Check
As commented by a fellow, we can do the following things
- Validate base, teacher, student using same prompts and seeds, 50 steps, CFG (guidance scale) 4.0,
noise_level0.0. We should then inspect the CLAP and ImageBind score with \(\mu \plusmn \sigma\). We should expect the student clearly beat the base and land within noise of the teacher. - Teacher merge precision. We should expect the exported LoRA and the base should be merged in FP32 to obtain the teacher1 To check this, we should (1) merge manually in FP32; (2) check
teacher - merged
Footnotes:
1
I don’t know if this is specific to verl-omni, but verl-omni does report a teacher degradation in SD3.5 runs if merged in BF16.