Reinforcement Learning Training

This tutorial follows the RP1 reinforcement learning workflow from environment setup and model/data preparation to task configuration and policy training. Keep the model, task, and policy configurations matched throughout. After training, continue to Evaluation and Simulation for policy export, MuJoCo Sim2Sim validation, and deployment handoff.

Velocity-command tasks use RP1.3-Flat / RP1.3-Rough, while reference-motion tracking uses RP1.3-BeyondMimic. Choose the target task first, prepare its inputs, and check the body action count, observations, and history layout before using a model or policy from another workflow.

If you already have a policy, begin with policy inputs and outputs and Evaluation and Simulation, then proceed to deployment. See PartyOS Overview for the relationship between the platform and its methods.

Use roboparty_train for Isaac Lab training, policy playback, and export, then check transfer through MuJoCo Sim2Sim before verifying the robot's inference configuration.

Prepare the Isaac environment using these versions and check the corresponding CUDA, graphics driver, and PyTorch requirements. Python dependencies are defined in robolab/setup.py and rsl_rl/pyproject.toml.

Place roboparty_train outside the Isaac Lab source tree and work in the Python environment used by Isaac Lab:

robolab/ and rsl_rl/ are source directories. Bulk task imports may reference the RPO asset submodule; if assets are missing, investigate the actual path reported in the exception.

When checking RP1.3 tasks, inspect their registration and configuration. list_envs.py launches Isaac Sim and filters only for RPO names. See Tasks and robot configurations for the task definitions.

The 28-action pipeline still requires corresponding tasks, rewards, observations, action mappings, and policies; 30/42-axis training pipelines are not currently available.

First identify the target task and body configuration, then prepare the robot assets and data. Flat/Rough velocity tasks and BeyondMimic motion tracking tasks use different inputs; reference motion videos cannot directly replace the motion data required for training.

These three RP1.3 paths use the 24-axis training assets in RP1_3_CFG. The visualization models in RP1_description, its 28-axis scenes with the same names, and the interactive-head GLB are used separately; the selected task determines the training assets.

Prepare and retain the URDF / MJCF, meshes, joint names and order, default pose, actuator configuration, and collision settings from the same source. Changes to the head, hands, or number of body axes require corresponding task, observation, and action definitions.

The pinned training source is roboparty_train@613193a725b5.

The BeyondMimic loader reads the following NPZ fields. When supplying your own reference motion, the data must match the target model, tracked body parts, and task timing definitions.

Check field dimensions, body order, units, coordinate frames, quaternion conventions, and the number of valid frames. Record the reference FPS separately from the control or policy frequency; velocities after resampling must be consistent with positions and timing.

When using human motion or motion from another robot, you can adapt it to the target body through the hhtools H2R / R2R workflow, then check joint limits, contacts, and motion continuity. A retargeted motion that can be visualized still needs validation in the target training task to establish whether it can be tracked dynamically.

For each set of training inputs, retain:

• The target model's source, version, and hash, plus joint and body mappings.
• Data files, their sources and usage licenses, and conversion methods and configurations.
• FPS, units, coordinate frames, valid segments, and training/evaluation splits.
• The corresponding Task ID, motion reference path, and configuration files.

Obtain dataset files and licenses from their actual sources; do not infer training data or checkpoints from demonstration videos. Once preparation is complete, proceed to Training and resuming runs; see the input/output contract for policy interfaces.

RP1.3 training tasks are registered in robolab/robolab/tasks/direct/base/__init__.py; both tasks use the BaseEnv direct environment. When modifying a task, track its robot assets, environment configuration, agent configuration, and observation definitions together.

The configuration files are rp1_3_env_cfg.py and agents/rp1_3_agent_cfg.py, respectively. The robot definition RP1_3_CFG is in robolab/robolab/assets/robots/roboparty.py and points to rp1.3/urdf/rp1_24dof.urdf.

These values apply to the 24-action training configuration. Confirm hardware travel ranges and safety limits against the corresponding robot specifications; see Policy inputs and outputs for the relationship between single-frame actor inputs and the history buffer.

Both rp1_flat.xml and rp1_rough.xml in the training repository reference robot_24dof.xml. The scenes with the same names in RP1_description reference robot.xml, the model with 28 motion joints; some meshes also differ between the two repositories.

Use the URDF, MJCF, and meshes from the corresponding roboparty_train version for training; when changing models, check joint definitions, geometry, inertia, and mappings together.

Before creating a separate 28-action task, provide the corresponding task registration, robot and actuator configurations, action/observation dimensions, rewards and termination conditions, agent configuration, joint order, history buffer, and Sim2Sim mappings. Maintain these definitions together as a separate configuration.

Before deployment, match input dimensions, fields, history order, coordinate frames, normalization, and joint mappings item by item.

Matching dimensions alone is not sufficient: check fields, order, and semantics as well. History ordering may also differ between 243-dimensional training and default deployment inputs. The 24-axis BeyondMimic policy and the default 28-axis policy have different body sets and input definitions and cannot be directly interchanged.

rp1_24dof/configs/beyondmimic.yaml lists both the default velocity policy and a motion tracking policy: the former uses 243 dimensions and the latter 126 dimensions, with references to policy_dance1.onnx and dance1.npz. Check the actual exported file for the selected policy branch; do not infer inputs from the YAML filename alone.

BaseEnv concatenates single-frame inputs in the following order:

A single frame totals 81 elements. This configuration uses 3 frames of history, giving a final actor input of 243 elements. The training environment updates CircularBuffer before flattening it; confirm the history direction against the Isaac Lab version in use.

See base_config.py for observation scaling and clipping. Actions are clipped first, then multiplied by action_scale and added to the default joint positions. Record numerical scaling, the statistical normalizer, and joint order separately.

The Actor for RP1.3 motion tracking assembles single-frame observations in the following order:

The total is 126 input dimensions and 24 output dimensions, with one frame of history. Retain the reference motion, model, tracking anchor, and timing definitions with the policy; see the motion tracking configuration and BM Sim2Sim entry point.

Let A and B represent two observation fields, and 0 and 1 represent two frames from oldest to newest:

The actual fields are vectors; this example illustrates only their ordering.

Both configurations have a policy period of 20 ms; their underlying simulation/control steps are 5 ms and 4 ms, respectively. Measure the actual runtime periods separately. Deployment then maps to motors through urdf2motor and closed-chain transformations; check the indices at each layer separately.

When using a newly exported policy, first confirm its history layout and mapping, then match the deployment configuration. Existing policies retain their corresponding training and export definitions.

1. Pinned versions/hashes of the checkpoint, exported files, training configuration, and deployment configuration.
2. Actual ONNX input/output names, shapes, dtypes, batch dimensions, opset, and external data.
3. Field order, coordinate frames, units, default pose, scaling, and clipping.
4. History length, layout, oldest-to-newest direction, initial-frame padding, and reset behavior.
5. Statistical normalizer values and whether normalization is embedded or handled externally.
6. Policy order→model order→motor order, signs, zero positions, and closed-chain transformations.
7. Inference/control timing configuration and action post-processing.

The deployment implementation requires a single input and checks its element count; verify the remaining semantics against the checklist above. The 28-axis default uses 10 frames, frame_major, and a 28-dimensional mapping, requiring its own I/O contract.

The training entry point is robolab/scripts/rsl_rl/train.py. It starts Isaac Sim, loads the task and agent configurations, then invokes the runner for training. Complete environment preparation first.

The numbers in the video indicate training iterations, allowing comparison of posture and walking behavior at different iterations.

States during training · Simulation

The numbers in the video indicate training iterations and help show posture and walking behavior at different iterations.

Download video

Run the following from the roboparty_train root directory in the matching Isaac Lab Python environment:

Use 1 environment and 1 iteration to check the loading and training pipeline. Set the parallel environment count and number of iterations separately for actual experiments; the policy produced by this check is only for verifying the pipeline.

Record the parallel environment count, number of iterations, seed, and runtime conditions for every experiment.

First set RP1_RUN_PATTERN and RP1_CHECKPOINT_PATTERN to regular expressions matching an existing run and file, then execute:

The training entry point uses re.match to search previous directories and files. In training, --checkpoint takes a matching expression, whereas the parameter with the same name in play takes an explicit file path. Use a sufficiently precise expression and check the checkpoint resolved in the startup output to avoid loading a different experiment.

The resume search takes place before the new log directory is created. Matching runs are selected by modification time, and empty runs are skipped. The runner saves and restores the model, optimizer, iteration count, and other state; new logs are written to a new run directory, preserving the original run.

The root log directory is logs/rsl_rl/rp1.3_flat for Flat and logs/rsl_rl/rp1.3_rough for Rough. Each run saves params/env.yaml, params/agent.yaml, and checkpoints; the runner saves at the configured interval and saves a final checkpoint when training ends.

Continue to Evaluation and export.

BeyondMimic trains motion tracking policies using body positions, orientations, and velocities from reference motions. RP1.3-BeyondMimic uses the RP1.3 model, with rp1.3_bm/dance1.npz as its default reference motion.

Dance motion · Simulation

Multiple simulated models perform a dance motion.

Download video

The training entry point is robolab/scripts/rsl_rl/train.py, and the policy playback entry point is robolab/scripts/rsl_rl/play_bm.py. Before playback, check the Task ID, robot configuration, motion file, and checkpoint; the observation and command contracts for a motion tracking policy must follow the corresponding task configuration.

RPO-BeyondMimic and RPO-Getup-Mimic are separately registered RPO tasks with their own models, motions, and agent configurations. They cannot directly replace RP1.3 tasks or policies.

Continue to Training and resuming runs and Evaluation and policy export.

1. During data preparation, fix the 24-axis model, reference motion, tracked bodies, and timing definitions.
2. Train and play back with RP1.3-BeyondMimic, retaining the checkpoint, configuration, and corresponding reference motion.
3. Verify the 126-dimensional input and 24-dimensional output, and run the matching TorchScript policy through the BM MuJoCo entry point.
4. When connecting to the physical robot, use the matching ONNX export, motion reference, and 24-axis deployment configuration, then continue with the deployment checks.

The motion tracking and default velocity branches in the deployment configuration must each match their own weights and observations; the history input of the default 28-axis velocity policy cannot replace this motion tracking path. See Acceptance records for the prerequisites for physical robot operation and rollback rules.