Multinode Runs

Training semantics live in recipe YAML under examples/. Shell scripts only prepare the runtime environment, start Ray, set path/logging defaults, and forward the recipe name plus Hydra overrides.

Launchers

Use the single-node launcher for local or one-node jobs, and the multinode launcher when workers need to join a Ray head before training starts. The first argument is the recipe path below examples/ (<path-under-examples>, without .yaml); ENTRY selects a non-diffusion entrypoint (train_ar, train_sft, train_async_ar, train_async_diffusion, train_pe, train_unified_model, or train_agentic).

bash examples/run_experiment_single_node.sh diffusion/sd3/sd3_trainside
bash examples/run_experiment_multinode.sh diffusion/sd3/sd3_sglang_rollout_colocate
ENTRY=train_ar bash examples/run_experiment_multinode.sh ar/qwen_vl_grpo_geo3k_mc_sglang_4x8
ENTRY=train_sft bash examples/run_experiment_multinode.sh sft/qwen3_sft
ENTRY=train_agentic bash examples/run_experiment_multinode.sh deep_research/deep_research_search_judge

The launcher defaults to scheduler-fanned LAUNCH=spmd. For an interactive head node with passwordless SSH to workers, set LAUNCH=ssh and provide NODE_IP_LIST=IP:GPUS,.... NUM_NODES, GPUS_PER_NODE, NODE_RANK, HEAD_IP, and RAY_PORT override topology discovery.

Environment-Derived Overrides

Launchers preserve the environment; each recipe decides which values it interpolates (commonly PRETRAINED_MODEL, data paths, and W&B settings). Inspect the selected YAML instead of assuming every recipe reads the same variables.

Cluster Geometry

For different cluster geometry, override the recipe's placement and batch-geometry fields together, for example the device count and the train-stack mini-batch shape:

bash examples/run_experiment_multinode.sh diffusion/sd3/sd3_sglang_rollout_colocate \
  num_devices=16 \
  stack.micro_batch_size=1 \
  stack.num_updates_per_batch=2

Hydra composition does not prove that this geometry is valid. Live geometry checks run later in trainer/worker construction, and their exact constraints differ by trainer. Review the owning trainer plus the selected recipe whenever changing topology or batch fields.

Pre-Run Checks

For any large run, first compose and resolve the recipe without launching Ray work:

python -m unirl.train_diffusion \
  --config-name=diffusion/sd3/sd3_sglang_rollout_colocate \
  --cfg job --resolve

This command checks Hydra composition, overrides, and interpolation/environment resolution only. It does not instantiate dataclasses or run trainer, cross-component, or batch-geometry validation. A small real smoke launch on the target topology is still required to exercise construction-time checks.