Evaluation
Periodic trainer evaluation, reward suites, and external benchmarks.
Quality Signal During Training
The reward components configured on a recipe are the primary quality signal.
Each rollout scores generated media and logs per-sample and per-component reward
(visible in WandB when enabled). Built-in scorers under unirl/reward/local/
include PickScore, HPS, OCR, GenEval2, VideoPickScore, and rule-based matchers.
See Rewards for configuration and the generated Reward Package README.
Periodic Evaluation
Set eval_interval above zero to enable a trainer's periodic path; 0
disables it. The implementations are not interchangeable:
| Trainer | Current evaluation behavior |
|---|---|
| Diffusion | Generates from eval_data_path, scores with the configured reward, and logs mean reward. Supports eval_num_prompts, eval_samples_per_prompt, eval_chunk_prompts, evaluation sampling overrides/eval_eta, and eval_rewards suites with shared or separate eval data. A reward block is required when eval is enabled. |
| AR | Runs batched generation and reward scoring, reporting avg@k-style accuracy/reward. Uses eval_num_prompts, eval_batch_size, eval_samples_per_prompt, and eval_temperature. It does not use the diffusion eval_rewards suite mechanism. |
| SFT | Performs validation forward passes and logs weighted eval/loss; it does not generate rollouts or call a reward model. Uses eval_batch_size and eval_num_samples; -1 means the complete eval set. |
| PE | Evaluates the composed AR→diffusion rollout and image reward. Supports eval_num_prompts, diffusion eval_cfg_text_scale/eval_eta, and optional eval_rewards suites. |
| Unified model | Evaluates the unified AR+diffusion generation path and image reward. Supports eval_num_prompts, eval_cfg_text_scale/eval_eta, and optional eval_rewards suites. |
| Async AR / async diffusion | Reuse the corresponding AR/diffusion evaluator, but first stop launching work, drain in-flight rollouts, and synchronize at an empty RolloutManager hard boundary. They do not evaluate arbitrary stale in-flight state. |
For MultimodalRLDataSource,
data_source.args.run.eval_data_path selects a deterministic, unshuffled eval
source and falls back to data_path when omitted. Recipes commonly interpolate
EVAL_DATA_PATH. SFT uses its SFT data source's validation iterator instead.
The listed trainers run an initial evaluation when enabled and then at the configured cadence. The agentic trainer is not covered by this periodic-eval contract. Use the selected recipe and trainer constructor as the exact source of truth for available fields.
Offline / External Benchmarks
The [eval] extra installs the current OCR reward dependencies: PaddlePaddle,
PaddleOCR, torchvision, and Levenshtein support. It is not a general benchmark
harness.
The legacy GenEval/OpenMMLab scorer must run as a Python 3.10 sidecar, separate from the core Python 3.12 / CUDA 13 engine environment. Follow Geneval MMCV Setup; do not install its MMCV stack into a core training environment.
If You Need Eval Now
- Enable the trainer's periodic eval fields and read
eval/*metrics in W&B. - Add an
eval_rewardssuite when diffusion eval needs additional scorers. - Or score generated media offline with the standalone reward service in
unirl-reward-service/.