P-BFM User Guide

P-BFM (Perceptive Behavioral Foundation Model) is a perceptive behavior foundation model released by RoboParty Lab for humanoid whole-body control across terrain. It maps motion references, target body states, and reward functions into one intent space. A frozen policy combines depth perception with explicit heading and velocity commands to execute those intents on the terrain underfoot.

P-BFM trains with just 40 flat-ground LaFAN motions and online interaction with simulated terrain, without terrain-specific motion demonstrations. The single trained policy transfers zero-shot to a real robot with an onboard depth camera, supporting motion tracking, goal reaching, reward inference, teleoperation, and disturbance recovery.

Resources:

This section covers how P-BFM works, release contents, and environment setup. By the end, you can install the environment, download motion data, and check the setup with tests and smoke training.

1. Unified intent space. P-BFM uses forward–backward representations. The Backward encoder maps robot states into intent space: frames of a reference motion, a target pose, or reward-weighted state samples all produce intent vectors z in the same space.

2. Perceptive execution. The Actor reads proprioceptive observations, intent z, and depth images (8 frames of 18 × 32) to execute the intent on the current terrain. The same intent produces different actions on stairs, slopes, or boxes.

3. Explicit root commands. Latent-only control is insufficiently precise about global travel distance and direction. P-BFM supplies heading and planar velocity commands alongside z to control root motion separately.

4. Velocity estimation. The policy includes a depth-supervised velocity estimator (721 → 512 → 256 → 3) that outputs current velocity in the yaw frame. At deployment, the policy uses this estimate without simulator ground-truth velocity.

5. Training. Unsupervised reinforcement learning uses 40 flat-ground LaFAN motions as behavioral priors while the robot interacts online with simulated terrain. Four independent Critics correspond to the behavior prior (QD), auxiliary rewards (QAux), heading (QH), and velocity (QV).

• Released: Code for training, inference, terrain evaluation, and ONNX export; training configuration (pbfm/configs/legacy_large.json); depth input contract (pbfm/configs/depth_contract.json); robot models and configurations.

• Downloaded separately: LaFAN training motion data (Section 2.1).

• Not provided: Pretrained weights. Neither the repository nor project page provides a weight download; inference and export require your own run directory trained as in Section 3. To preview behavior, open the project's interactive browser demo and try motion references, target poses, and reward presets on a shared terrain map (a desktop browser is recommended).

• Not included: A controller that sends joint commands to the robot. Section 6 explains what the exported model includes and what you must implement.

Note

P-BFM is based on UFO and retains its upstream CC BY-NC 4.0 license, restricted to noncommercial use.

--extra test also installs pytest and onnxruntime. ONNX export in Section 6 uses onnxruntime to verify the output, so installing it now is recommended.

Download training data first (Section 2.1), then run unit tests:

Tests requiring external data or a GPU skip automatically when unavailable. Passing tests verifies code behavior, not policy convergence or real-robot safety.

With 8 available GPUs, also run smoke training as in Section 3.3 to check the training pipeline.

See Section 3.6 for the training output (--work-dir) layout.

This section prepares training motion data: download the authors' LaFAN data or convert your own motions into a format P-BFM can read. Run commands from the repository root.

The script downloads two files from the Hugging Face dataset xuewang/UFO-MotionData, verifies SHA-256, and places them in pbfm/data/:

Existing files with correct hashes are skipped. The script needs the hf command or Python package huggingface_hub, already present after uv sync.

Tip

When no --data-path is found, inference defaults to pbfm/data/lafan_29dof_10s-clipped.pkl relative to the current directory. You can therefore omit the path when running inference from the repository root. Training requires an explicit --data-path; see Section 3.2.

P-BFM reads robot-state CSV or NPZ motions. The CSV example at pbfm/data/examples/g1_robot_state_sample.csv has one frame per row:

1. Check that the data can be read correctly:

   uv run python -m pbfm.motion.inspect \
     --robot pbfm/configs/robots/g1_29dof.yaml \
     --source <your-CSV-or-NPZ-file-or-directory>
   

2. Generate a data manifest and build the training cache:

   uv run python -m pbfm.motion.build \
     --robot pbfm/configs/robots/g1_29dof.yaml \
     --source <your-CSV-or-NPZ-file-or-directory> \
     --name <dataset-name> \
     --out <manifest-path>.yaml
   

3. Use --data-manifest <manifest-path>.yaml instead of --data-path for training; they are mutually exclusive.

To mix multiple sources, pass multiple files to --data-path and specify weights with --data-mix-weights, for example --data-mix-weights 0.95 0.05.

Note

For terrain training (anything except --terrain-mode plane), preflight checks that each motion's maximum horizontal displacement, plus perception radius and margin, stays within the terrain's safe radius. Motions exceeding it stop training with an error; success logs terrain coverage preflight passed.

This section trains P-BFM from scratch. Full training requires 8 GPUs. Run commands from the repository root.

UFO_CACHE_DIR is the root for temporary files, PyTorch / Triton / CUDA / Warp compilation caches, and expert-data caches. Use a writable disk with enough space; the default is cache/ under the current directory.

Replace /path/to/new-run with a new output directory. torch.distributed.run launches 8 local processes, each using one GPU and 1024 environments, for a total of 192,020,480 global environment steps. --gpu-ids must be single; torch.distributed.run handles multiple GPUs.

Note

Always provide --data-path or --data-manifest. The configuration template's default data path is specific to the authors' machine and does not exist elsewhere.

Note

Use scripts/train_legacy_scratch.py for training from scratch. It starts the terrain curriculum at step 0 and rejects work directories containing checkpoints. Running python -m pbfm.train directly uses the continuation schedule: 384,040,960 total steps, with the terrain curriculum starting at step 192,020,480.

Append --smoke to the Section 3.2 command and use a new --work-dir:

Smoke mode runs 32,768 global steps with at most 16 environments per process and reduced replay capacity. It disables training-time evaluation, replay checkpoint writing, and W&B. It only verifies that the pipeline runs.

Note

Smoke replay is small and does not represent full-training GPU memory usage. Before full training, ensure each GPU can hold full replay capacity (default 5,120,000 transitions per rank), model and optimizer states, and training batches, and that disk space can hold every rank's replay checkpoints.

plane is the flat-ground control condition: the surface is completely flat, but terrain observations are retained. For W&B, append --use-wandb --wandb-project PBFM --wandb-run-name pbfm.

The default rp1_simple terrain consists of 5 m square tiles, with the robot starting at each tile's center. Terrain proportions are:

Steps are 10–20 cm high, roughness amplitude is 2–6 cm, and boxes are 5–15 cm high. Difficulty increases through a curriculum starting at step 0 for scratch training, advancing every 19,200,000 global steps.

Note

Stair names describe elevation changes when moving toward the tile center. The robot starts at the center and moves outward, so low_stairs_down actually means ascending and low_stairs_up descending. Check the start, heading, and actual geometry before labeling videos.

Inference and ONNX export need only the model. Reward inference in Section 4.3 also requires replay.

Resuming depends on replay format and depth-observation conventions:

• Training replay must record explicit_next=true. If absent, load only the model into a new training directory, rebuild replay with a collection warmup, then start updates.

• Do not modify replay metadata to force incompatible data to load.

This section runs three inference modes with a trained policy in simulation: motion tracking, goal reaching, and reward inference. They generate intent z from motion references, target poses, and reward functions, respectively. Run from the repository root. --model-folder points to the training --work-dir, which must contain checkpoint/.

The policy encodes the reference motion into intent frame by frame and tracks it in simulation. --save-mp4 produces a side-by-side video: reference on the left and policy execution on the right.

To track on a training terrain tile, for example low_stairs_down at difficulty row 7:

Terrain geometry is read from the checkpoint's training configuration.

Outputs:

The policy reaches a sequence of target poses taken from specific motion-data frames. The default list, pbfm/data/robots/g1/goal_frames_lafan29dof.json, contains 8 groups, each specifying a motion name (motion_name), motion index (motion_id), and frame indices (frames). Replace it with --goal-json.

Outputs are <model-folder>/goal_inference/goal_reaching.pkl (intent vectors for each goal) and goal_inference/videos/goal.mp4.

Reward inference rescores training replay samples with the specified reward function and computes intent vectors. It therefore requires training replay: train_reduced, train_rank_<n>, or train under checkpoint/buffers/. Smoke training does not write replay and cannot support reward inference.

By default, 38 tasks run in sequence; select a subset with --tasks. Examples of task names:

Outputs are <model-folder>/reward_inference/reward_locomotion.pkl and one video per task (requires --save-mp4).

Alongside intent z, the policy reads heading and planar velocity commands. The three inference modes handle them differently:

• Motion tracking: Intent advances frame by frame through an 8-frame reference window, shortened at the motion's end. Heading and valid planar velocity commands come from the same aligned reference motion.

• Goal reaching and reward inference: Reference velocity commands are not generated automatically.

• Valid versus invalid: A valid zero velocity means “stop”; invalid means “no velocity command.” For custom motion names, explicitly choose the intended --velocity-context.

When saving or sending intent, keep the vector, heading and velocity, validity flags, frame rate, and coordinate conventions together. The intent vector alone is not a complete input to a command-conditioned policy.

This section executes the same intent on different terrains and compares behavior to assess terrain adaptation. Run commands from the repository root.

Results are written as JSON and a same-named CSV in the same location, with one row per terrain.

See all options with uv run python -m pbfm.evaluation.terrain --help.

This section exports ONNX models for deployment and explains input conventions and the components you must implement. The repository does not include a real-robot controller. Run commands from the repository root.

The script loads the checkpoint on CPU, exports ONNX, and verifies it with the ONNX checker and onnxruntime comparisons. Success prints a verification result beginning with Verified <output-path>:.

The exported policy.onnx is a complete deployment graph, not a split model containing only the action network:

Inputs contain neither Critics nor simulator-only velocity labels. Input names and shapes may differ across checkpoints; use the metadata generated at export rather than another checkpoint's interface.

The Backward encoder maps robot states into intent space and computes intent vectors from reference motions or target poses at deployment. Export it separately:

The depth contract is recorded in pbfm/configs/depth_contract.json; deployment preprocessing must match it:

Note

ONNX depth input is float in [0, 1]. Convert simulated uint8 depth to float32 and divide by 255 exactly once. Do not divide already normalized float depth again.

scripts/verify_depth_policy_trace.py uses recorded robot observations to compare actions and velocity estimates from ONNX and the original checkpoint:

This checks interface consistency, not control performance. ONNX parity tests also cannot validate real-robot integration steps such as camera preprocessing, temporal histories, action scaling, or PD control.

The repository also includes a read-only recorder, pbfm/perception/capture.py, that synchronizes depth-camera images and robot states for perception debugging. It neither loads a policy nor sends motor commands.

• Camera-depth preprocessing matches Section 6.3 and has been verified with real traces (Section 6.4)

• Depth is normalized exactly once, with inputs in [0, 1]

• Intent vectors, heading and velocity commands, validity flags, frame rate, and coordinate conventions are sent together (Section 4.4)

• Input names and shapes match the metadata exported from this checkpoint

• Action scaling, joint order, and PD gains match the training robot configuration

• Tracking and terrain evaluation have run in simulation with the same checkpoint (Sections 4 and 5)

• The robot is suspended or protected for its first run, and the hardware emergency stop has been tested separately from software stopping