UFO User Guide

UFO is an open-source unsupervised reinforcement learning framework for humanoid robots from RoboParty Lab, used to train Behavior Foundation Models (BFM). It pretrains a whole-body control policy and a "behavior space" in simulation. Afterwards, given a reference motion, target pose, or reward function, the same policy can directly perform the corresponding task without retraining for each task.

The framework includes two training algorithms: FB (reproducing and improving BFM-Zero) and TeCH.

Resources:

This section introduces UFO's capabilities, the two code branches, and the environment setup. After completing this section, you can download the example data and get a quick training run working on a single GPU.

UFO's code is split across two branches with different purposes; do not merge them into each other:

According to the UFO technical report, FB trains in about 6 hours on 8 H200s, and can also be trained on 8 RTX 4090s.

Clone the repository:

Install uv:

Install the project environment:

Optional: W&B logging. To view training curves in Weights & Biases, log in first, then add --use-wandb --wandb-run-name <name> to the training command:

Motion data is not stored in the Git repository. Download the default LaFAN example data:

The script downloads two files from Hugging Face and verifies their SHA-256:

--smoke uses 16 parallel environments and 2048 steps, and does not connect to W&B; it is used to confirm that the environment and data load correctly, and does not represent training quality.

Both training and inference results are written to the directory specified by --work-dir:

run_train.sh also automatically creates a cache/ directory based on --work-dir (for example runs/ufo_fb corresponds to cache/ under the repository), storing caches for uv, PyTorch, Triton, Warp, etc.; you can also specify the location with the environment variable UFO_CACHE_DIR.

This section imports robot motion data into UFO, generates a data manifest for training, and explains how to mix multiple data sources by ratio.

Note UFO does not perform motion retargeting. Human motions or motions from other robots must first be converted into joint trajectories of the target robot using hhtools, GMR, or your own retargeting pipeline, and then imported into UFO.

UFO's recommended import format is RobotState: each frame contains the root position root_pos, root orientation root_quat, and joint angles dof_pos, plus a frame rate fps or time column time.

CSV columns: without a header, they are, in order, the root position (x, y, z), root quaternion (x, y, z, w), then the joint angles, in the same order as the actuated joints in the MuJoCo XML; a time column may be prepended. With a header, you can specify in the data manifest which columns each item corresponds to using columns (see configs/data/example_robot_state_auto_build.yaml).

Before importing, check whether the data and robot config match:

Set --robot to the config file of the corresponding robot under configs/robots/; new robot configs are generated per Section 6.

This step generates:

• The data manifest YAML (at the location specified by --out), used by both training and inference to read the data;

• A pkl of the complete motions, for inference;

• A pkl clipped into about 10-second segments, for training.

The data manifest can list multiple data sources and set a sampling weight for each:

• Between data sources, sampling follows the fixed weight ratio; within each data source, motions that the policy fails to track are prioritized for sampling.

• This is suitable for injecting a small amount of rare, high-difficulty skills while preserving the base motion distribution, for example using 5% cartwheel data to teach the policy a cartwheel.

• train_path is used for training and inference_path for inference.

• When injecting motions such as cartwheels, you can add --cartwheel-aux-safe to the training command, which removes the contact and foot-shape penalties related to walking and reduces the motion change-rate penalty.

You can also skip the manifest and give multiple files and weights directly in the training command: --data-path a.pkl b.pkl --data-mix-weights 0.95 0.05.

Note

• Each data source's name must be unique; weights must be non-negative finite numbers and sum to more than 0.

• RobotState import does not resample the frame rate; make sure --fps matches the actual data.

• Currently only hinge joints are supported.

This section trains a policy with FB or TeCH. Training runs on the main branch and is launched by run_train.sh.

TeCH was called TLDR in early versions; --agent tldr still works but prints a deprecation notice.

FB:

TeCH:

When using your own data, replace --data-path with --data-manifest <manifest path>.

The model is saved in <work-dir>/checkpoint/, overwritten every --checkpoint-every-steps steps, along with the replay buffer (checkpoint/buffers/). Reward inference needs to sample states from the replay buffer, so do not delete this directory.

This section uses the trained model for three kinds of inference and exports the ONNX for deployment. The three kinds of inference correspond to three ways of specifying a task:

All three kinds of inference run in MuJoCo; you can save a video with --save-mp4; --disable-dr and --disable-obs-noise are used to disable randomization and observation noise, making it easier to observe the policy's own behavior.

• Use the complete motion sequence (e.g. lafan_29dof.pkl), not the 10-second training segments.

• --motion-list specifies the motion indices to track; you can give several at once.

• When using a data manifest, instead use --data-manifest <manifest> --dataset <data source name>.

• --max-steps can limit the number of frames replayed, for quick previews.

Results are written to <model-folder>/tracking_inference/:

Motion tracking exports ONNX by default; add --export-onnx false when not needed.

• The policy reaches a set of target poses in sequence within one episode, switching targets every 100 steps by default (--goal-switch-interval), with an episode length of 2100 steps (--episode-len).

• The default robot uses the repository's built-in target pose files; other robots need to provide their own target pose files with --goal-json, which cannot be shared between different robots.

• Results are written to <model-folder>/goal_inference/, including goal_reaching.pkl and videos.

• Reward inference samples states from the replay buffer saved during training (150,000 by default), scores these states with the reward function, and thereby computes the corresponding z. So it requires checkpoint/buffers/ in the training directory; you can also specify the replay buffer location with --buffer-path.

• The default tasks include standing in place, walking in each direction, turning, raising a hand, squatting, etc.; use --tasks to run only some of them, for example move-ego-0-0.7.

• To compute only z without generating videos, add --skip-rollouts.

• Results are written to <model-folder>/reward_inference/, including reward_locomotion.pkl and a video for each task.

Note New robots currently only support locomotion and turning tasks (move-ego-*, rotate-z-*). Tasks such as raising a hand, squatting, and sitting on the ground depend on the default robot's body semantic definitions and will error out directly.

Goal reaching and reward inference do not export ONNX by default; add --export-onnx when needed. In the exported policy ONNX, the input partition is derived by the model itself; the accompanying JSON records the robot name, config path, XML path, actuated joints, each input dimension, the z dimension, and the output action dimension.

Note The exported ONNX is bound to the robot, motion dimension, and observation dimension it was trained with, and cannot be used for other robots.

This section uses the deploy branch to validate the policy in MuJoCo, then deploy it to the real robot and teleoperate with PICO. The deploy branch is only responsible for running, not training; each robot requires a dedicated deployment config, denoted <robot> below.

When running for the first time on a new machine, new model, or new teleoperation environment, proceed in this order:

1. Sim2Sim on the workstation;

2. Teleoperation Sim2Sim on the workstation;

3. Sim2Real with the robot suspended or supported;

4. Onboard PICO teleoperation: first only observe the real-time z and robot state without enabling the policy;

5. Using the remote controller, enter standing, enable the policy, and start tracking in sequence, and test reset and stop;

6. Deliberately disconnect PICO, XRoboToolkit, or the network, and confirm the watchdog stops the policy output.

Policy inference uses CPU by default. When a GPU is needed, install the onnxruntime-gpu matching your CUDA and set onnx_providers in config/policy/<robot>_policy.yaml.

PICO teleoperation needs another environment, ufo-teleop, as well as the XRoboToolkit service and xrobotoolkit_sdk; see scripts/teleop/README.md in the deploy branch for the installation steps.

The model files are large and not stored in Git; they go under model/<robot>_policy/ (see the Model Files section of the deploy branch README for the download command for released models). The downloaded directory should be:

After downloading, verify the SHA-256 in release_manifest.yaml using the script in the README.

Note The policy, encoder, z files, and deployment code are released as a validated whole. Do not mix old model files with the latest deployment code unless compatibility is explicitly stated.

Terminal A starts MuJoCo:

Terminal B starts the policy:

--task determines which task to execute: config/exp/tracking/tracking.yaml (tracking offline motions), config/exp/goal/goal.yaml (goal reaching), config/exp/reward/locomotion.yaml (reward task).

Control with the keyboard in the policy terminal:

On the workstation, open four terminals and start them in order:

See the Teleop Sim2Sim section of the deploy branch README for the full parameters of Terminal C; if the CUDA version of ONNX Runtime is not installed, change --device cuda to --device cpu.

Real-time z is in a "frozen" state by default. Once the policy is ready, press the right-hand A button on the PICO to start following; the left-hand X button freezes the current z.

On the robot's onboard compute unit, you need:

Use rsync to sync the repository and models from the workstation to the robot. Then on the robot:

• Confirm that all CPU cores of the onboard compute unit are online: cat /sys/devices/system/cpu/online;

• Run the preflight check that does not drive the motors:

  ONBOARD_PY=<venv path>/bin/python \
    scripts/onboard/run_preflight_suite.sh --profile ordinary
  

On the robot, after setting the environment variables for CycloneDDS and the low-level control bindings (see the deploy branch README), start the policy:

UFO_REAL_ROBOT_OK=1 is the confirmation switch for real-robot control; set it only when about to control the real robot.

Operate with the robot's remote controller (the specific keys follow the robot's deployment config):

For emergencies, use the physical emergency stop.

PICO connects directly to the robot, and all programs run on the robot's onboard compute unit:

1. Start the XRoboToolkit service and set the target IP to the robot's IP in the PICO app.

2. Robot Terminal A, start the pose bridge: scripts/teleop/teleop_pose_50hz_onboard.sh

3. Robot Terminal B, start real-time z: Z_PY=<venv path>/bin/python scripts/realtime/run_realtime_z_server_onboard.sh

4. Robot Terminal C, start the policy that subscribes to real-time z (must set UFO_REAL_ROBOT_OK=1; see the deploy branch README for the launch script).

Division of labor between the two controllers:

PICO buttons only control the reference motion; they cannot enable the policy, release the stop lock, or replace the physical emergency stop.

Safety mechanisms: when the teleoperation data is stale or invalid, the real-time z service stops publishing; if the policy does not receive a valid 256-dimensional z within ctx_zmq_timeout_ms (200 ms by default), it stops outputting; the final joint targets are clipped to the robot's joint velocity limits.

For the configuration of split teleoperation (retargeting and real-time z on the workstation), see Section 5B of the deploy branch README.

Before enabling the policy, confirm each item:

-[ ] The physical emergency stop is within reach and has been tested;

-[ ] For the first run, the robot is suspended or supported;

-[ ] All CPU cores of the onboard compute unit are online;

-[ ] Both ONNX files can be loaded on the robot;

-[ ] The interface name in <robot>_real.yaml is correct;

-[ ] Both Sim2Sim and teleoperation Sim2Sim have passed;

-[ ] The global stop lock of the remote controller has been tested;

-[ ] The real-time z watchdog has been tested by disconnecting PICO, XRoboToolkit, or the network;

-[ ] Set UFO_REAL_ROBOT_OK=1 only when about to control the real robot.

Note The MP4 included with the released models is only a simulation replay and does not prove real-robot safety. What has been verified so far is that ordinary Sim2Real and onboard PICO teleoperation can run on the real robot; systematic fault injection, long-duration free walking, and impact-limit tests have not yet been completed.

This section generates configs for a new robot and starts training. This route is experimental: the repository's built-in default robot config is the most thoroughly tested, and new robots typically still need adjustments to the controller, rewards, contacts, and termination conditions.

1. The target robot's MuJoCo XML;

2. An optional corresponding URDF;

3. RobotState motion data already retargeted to the target robot (see Section 2).

• The tool extracts body names, actuated joint order, motion dimensions, joint ranges, and the floating base from the XML. The XML is the sole basis for the joint order and motion layout.

• The URDF is optional auxiliary information, used to supplement hardware parameters such as torque, velocity, damping, and friction; when the two conflict on joint limits, the XML takes precedence by default. When the joint names in the XML and URDF differ, use --urdf-joint-name-map to specify the correspondence.

• When --hydra-out is added, the Hydra robot config needed by the training environment is also generated.

The generated config is marked as a draft (review_status: draft), and its semantic fields are heuristic guesses. Before formal training, at least check:

• The base (base_body), feet (feet), hands (hands), and key body parts (key_bodies);

• The actuated joint order and initial pose;

• PD gains and actuator limits;

• Contact bodies, and fields related to rewards and termination conditions.

Per Sections 2.2 and 2.3, replace --robot with the new robot's config to inspect and import the data. Then run a quick test:

When robot_config is written in the data manifest, --robot-config can be omitted; when both are present, UFO checks whether they point to the same robot. After the quick test passes, remove --smoke and set the GPU and training parameters for formal training.