Overview

UniRL uses Hydra entrypoints with domain-specific trainer lifecycles. This page is a navigation overview; the code-adjacent contracts are generated from unirl/README.md and the package READMEs embedded as pages in each sidebar section.

Synchronous RL:
  train_diffusion | train_ar | train_pe | train_unified_model | train_agentic
  -> rollout -> reward -> advantage -> train -> optional weight sync

Supervised:
  train_sft
  -> records -> build Part -> train

Asynchronous RL:
  train_async_diffusion | train_async_ar
  -> separate train and rollout placement slabs
  -> buffered rollout/reward/advantage/train with versioned weight publication

Each entrypoint validates its recipe and delegates to its own trainer. They share the single-controller Remote / placement substrate and train-stack contracts, not one placement layout or one RL lifecycle. The earlier v1 actor-group runtime has been retired.

Main References

• Code Architecture README for the module map and runtime data flow.
• Rollout Package README for rollout modes and request/response contracts.
• Train Stack README for the FSDP backend, the train-step contract, and structural injection.
• Algorithms Package README for per-track loss contracts and the reward→advantage path.
• SDE Package README for SDE strategy rules, schedules, and runtime kernels.
• Weight Sync README for dedicated rollout synchronization.

Data Flow

RL trainer
  -> build a request Sample, dispatch to the rollout engine
  -> filled Sample with generated Parts (conditions, segments, primitives, media previews)
  -> RewardService.score_and_attach(sample) -> frontier Part.rewards
  -> Part.compute_advantages             -> Part.advantages
  -> TrainStack.train_track(part)        shards across train workers, runs the mini-batch loop
  -> optional weight sync back to dedicated rollout workers

Sample is the end-to-end rollout boundary: every engine accepts and returns a Sample, whose ordered Sample.parts are Part objects. New engines should adapt backend-specific outputs into these types from unirl/types/sample.py instead of leaking backend objects into training code.

Each trainer (unirl/trainer/<domain>.py) owns its placement blocks, worker construction, and phase ordering. SFT has no rollout, reward, advantage, or weight-sync phase; async AR/diffusion use disjoint train and rollout slabs.

For deployment modes and engine-specific requirements, use the generated Rollout Package README as the source of truth.

One Algorithm per Track

The training loss for a generated Part is its configured StageAlgorithm, such as diffusion unirl.algorithms.flowgrpo.FlowGRPO or autoregressive unirl.algorithms.grpo.GRPO. There is no separate driver-side "rollout control" object:

• reward→advantage z-scoring lives on Part.compute_advantages (unirl/types/sample.py);
• SDE-index selection lives on DiffusionSamplingParams.resolve_sde_indices (unirl/types/sampling.py).

A single-stage recipe binds one cfg.algorithm; PE nests one algorithm: node per generated Part (diffusion.algorithm, ar.algorithm) and runs sibling TrainStacks. Unified-model training instead uses UnifiedModelTrainStack to accumulate AR and image losses into one shared-backbone optimizer step. See Trainer & Training Stack.

Roadmap

Near-term direction spans three tracks — Infra, Algorithm, and Model. See the Roadmap.