Concepts & Glossary

Read this first if the rest of the docs use unfamiliar terms. It is a conceptual primer, not an API reference; code-adjacent contracts live in the package pages embedded in each section, such as Code Architecture.

Lifecycles

RL, supervised, and asynchronous runs do not share one lifecycle:

Synchronous RL:
prompts -> rollout (fill generated Parts in a Sample)
        -> reward  (score media into per-sample rewards)
        -> advantage (normalize rewards within/across groups)
        -> train   (replay trajectories, compute loss, backward, optimizer step)
        -> [optional] weight sync back to dedicated rollout engines

SFT:
records -> build a supervised Part -> train

Async AR / diffusion:
separate rollout slab -> buffered Samples -> reward/advantage -> separate train slab
                      <- versioned weight publication <-

SFT has no rollout, reward, advantage, or weight-sync phase. Async AR and diffusion overlap work across two disjoint placement slabs while preserving publication and checkpoint barriers.

One Algorithm per Generated Part

A rollout engine transforms a request Sample into a filled Sample. Sample.parts is an ordered lineage of Part objects; a generated Part carries its stage segment, conditions, primitives, rewards, and advantages. Each trainable generated Part binds one loss algorithm — a StageAlgorithm that replays the stage, computes a loss, and calls backward().

There is no separate driver-side "rollout control" object. Reward→advantage shaping and SDE-index selection live on typed objects:

Concern	Where it lives
Loss	`cfg.algorithm` → a `StageAlgorithm` (e.g. diffusion `FlowGRPO` or AR `GRPO`)
Reward → advantage	`Part.compute_advantages` (`unirl/types/sample.py`)
Which inference steps run SDE	`DiffusionSamplingParams.resolve_sde_indices` (`unirl/types/sampling.py`)

A single-stage recipe binds one top-level cfg.algorithm. PE nests one algorithm: node per generated Part (diffusion.algorithm, ar.algorithm) and runs sibling TrainStacks. Unified-model training uses one UnifiedModelTrainStack for its shared AR + image backbone.

Glossary

Orchestration

Term	Meaning
Driver	The process running the training loop: an entrypoint (`unirl.train_diffusion`, …) that builds a `<Domain>Trainer`.
Trainer	`unirl/trainer/<domain>.py` — owns that domain's placement block(s), workers, and lifecycle.
Remote / placement	Single-controller layer (`unirl/distributed/group/`): a `Remote` is a logical worker; a `placement` block assigns sibling Remotes to one GPU slab and carries `RankInfo` (DP/TP/PP/SP/EP ranks).
Bundle	A model package's trainable + frozen modules (transformer, VAE, text encoders) exposed to training and rollout.
Pipeline	The model's sampling pipeline (denoising / generation) used to produce media and trajectories.

Rollout

Term	Meaning
Rollout engine	The sampler backend: `trainside`, `sglang`, `sglang_llm`, `vllm_omni`, or `composed`.
Direct sampling	`trainside`: the FSDP-wrapped training module IS the sampler; no separate engine, no weight sync.
Dedicated sampling	A separate engine (SGLang / vLLM-Omni) holds its own weights and needs trainer→rollout weight sync.
Colocate vs separate	Whether train and rollout share GPU bundles (`colocate`) or run on distinct GPU slabs (`separate`).
`Sample`	The typed rollout boundary. Every engine accepts and returns a `Sample`; see `unirl/types/sample.py`.
Part / Segment	`Sample.parts` is an ordered lineage. Each `Part` may hold a stage `segment` (e.g. a diffusion trajectory with per-step log-probs), conditions, primitives, rewards, and advantages.
Group	The `sampling.samples_per_prompt` siblings of one prompt; advantages are normalized within a group.

SDE & log-probs

Term	Meaning
SDE strategy	Per-step stochastic kernel that produces a step log-prob (`FlowSDEStrategy`, `DanceSDEStrategy`, ...).
Sigma schedule	The σ values across denoising steps; pinned on the generated Part's sampling params as a single source of truth.
`old_logp` / `new_logp`	Rollout-time vs current-weights log-probs; their ratio drives GRPO's clipped objective.
Log-prob replay	Recomputing log-probs at train time by replaying the stage; old log-probs stay fixed across updates.

Training

Term	Meaning
Backend	The training-state `Remote` (`FSDPBackend`) owning structural injection + optimizer + scheduler + EMA.
Stage / StageAlgorithm	A trainable stage and the loss object that replays it and runs forward/backward.
Train stack	`TrainStack` — implemented in `unirl/train/stack/base.py` and exported by `unirl.train.stack`; PE uses sibling stacks, while HI3 uses `unified_model_stack`.
EMA rollout	Sampling rollouts with EMA-smoothed weights (NFT opts in; GRPO / DPPO must not).
Batch geometry	`micro_batch_size`, `num_updates_per_batch`; multi-update freezes π_old once per rollout shard.

Reward

Term	Meaning
RewardService	`unirl/reward/service.py` — holds one backend: local scorers or the remote HTTP client.
Reward component / scorer	A unit that scores media (PickScore, HPS, OCR, GenEval2, VideoPickScore, ...).
Advantage	The normalized reward signal the loss multiplies against.

Data-plane vs weight-plane

Term	Meaning
Weight sync (`cfg.sync`)	Sends fresh trainer weights to dedicated rollout engines (NCCL broadcast, tensor, IPC).
Tensor transport	The data plane (`unirl/distributed/tensor/`) for moving bulky rollout outputs between workers.

These solve opposite directions; do not conflate them.

Where to Go Next

Runtime map: Overview.
Algorithm loss math and the reward→advantage path: Algorithms Package README.
Picking a recipe: Experiment Recipes.

Concepts & Glossary

On this page