I-BFM User Guide

I-BFM is an open-source behavior foundation model for humanoid–object interaction. It uses forward–backward representations and unsupervised reinforcement learning to learn a joint representation of the humanoid, objects, and contacts. A downstream reward function is enough to infer an intent vector, allowing the same policy to carry, push, or kick boxes without task-specific retraining.

For box carrying, I-BFM achieves 94.3% success without disturbances and 89.3% after a robot fall. By comparison, the planning-based baseline achieves only 1.3% after a fall.

Resources:

This section covers how I-BFM works, release contents, and environment setup. By the end, you can download the released model and run a box-carrying task in simulation.

For a quick preview, open the Live MuJoCo Session on the project page to try box carrying and keyboard control in your browser.

1. Shared representation. I-BFM uses forward–backward (FB) representations to learn the coupled dynamics of the humanoid, objects, and contacts. The Backward encoder maps states into a 256-dimensional intent space, and the policy outputs actions conditioned on intent vector z.

2. Unsupervised pretraining. Pretraining targets no particular task. It mixes LaFAN motions (30%), box-carrying motions (40%), and box-kicking motions (30%) as behavioral priors while interacting online in simulation.

3. Dual-timescale objectives. Training combines short-term interaction objectives with longer-term objectives (target horizon: 8 steps), enabling interaction control at different timescales.

4. Reward inference. Given a downstream reward function, replay samples weighted by that reward determine the task's intent vector z, without policy retraining.

5. Task controllers. Carrying, pushing, and kicking each have a controller that switches between pre-inferred intent vectors across phases—approaching, aligning, lifting, carrying, placing, recovering from falls, and so on—to form a closed loop.

The repository does not include real-robot deployment software.

Note Training source retains upstream BFM-Zero's CC BY-NC 4.0 license and is restricted to noncommercial use.

1. git lfs pull downloads both weight parts, LaFAN motions, and carrying/kicking motion caches.

2. restore_checkpoint.sh checks both parts' SHA-256 hashes, joins them into checkpoints/ibfm-166m/model/model.safetensors, and verifies the result. It prints Restored … on success, or Checkpoint is already restored. if already restored.

3. uv sync installs dependencies from uv.lock into ibfm/.venv.

Note Without git lfs pull, the weight parts are only pointer files of a few hundred bytes, causing restore_checkpoint.sh to exit on a SHA-256 mismatch. Run git lfs pull, then restore again.

Tip All subsequent commands in this guide run inside ibfm/.

On a machine without a display, run 100 steps of box carrying in headless mode:

The script generates a random scene, loads the released model, and runs it. At completion, it prints the trajectory, controller-summary, and height-test paths (Trace:, Controller:, Height tests:), confirming that the model and environment are ready.

With a desktop environment, run ./tasks/carry_box/run_interactive.sh directly to open the MuJoCo viewer and record video; see Section 2.2.

This section runs three interaction tasks in MuJoCo with the released model. All three share one model; their task controllers and intent vectors differ. Run commands inside ibfm/.

With the viewer open, use the keyboard to change targets or introduce disturbances:

The viewer uses GLFW and requires a desktop environment. If DISPLAY is unset, the carrying script defaults to :1; set the correct DISPLAY first if your display number differs.

The robot approaches the box, aligns, lifts it, carries it to the target, and puts it down. If it falls, it gets up before continuing.

The script performs two steps:

1. Generate a random scene: robot–box distance 1.5–2.0 m and target height 0.65 m, saved to tasks/carry_box/random_height_scene/.

2. Run the carrying controller with a 0.30 m cubic box weighing 0.70 kg.

For carrying, R also scores the current trial, places the box at a random height between 0.15 and 0.80 m, and starts a new trial. Each trial records successful lifting and the 3D distance between the box center and target; placement succeeds when this distance is below 0.20 m.

Common environment variables:

Outputs are written to tasks/carry_box/interactive/random_height_<RECORD_TAG>/:

The robot moves behind the box, aligns, and pushes it toward the target with its hands. The target defaults to 2.0 m along X from the box's initial position. Success requires the box–target horizontal distance to be within the threshold (default 0.40 m), and the controller also checks that the box is stationary and flat on the ground. The first argument can set the output directory name, for example ./tasks/push_box/run_interactive.sh test1.

Outputs are <OUTPUT_ROOT>/<name>/push_box.mp4, rollout.csv, and summary.json.

The robot contacts the box with its foot and kicks it toward the target, located 2.0 m along X from the box's initial position. The target marker has a radius of 0.20 m. By default, the script opens the viewer, does not record video, and runs inference on CPU.

Outputs are rollout.csv and summary.json, plus kick_box.mp4 when recording.

Note The kicking script defaults to GLFW even with VIEWER=0. On a machine without a desktop, also set MUJOCO_GL=egl and a finite MAX_STEPS, for example VIEWER=0 MUJOCO_GL=egl MAX_STEPS=3000 ./tasks/kick_box/run_interactive.sh.

Simulation success rates (%) published on the project page:

This section describes the training motion data and how to convert your own motions. Run commands inside ibfm/.

The released model's data mixture is defined in configs/data/g1_lafan_carry_kick_30_40_30.yaml. All three sources are downloaded with git lfs pull:

Weights apply per data source, independently of the number of motions in each source.

Your motions can use robot-state CSV or NPZ format. A CSV has one frame per row, containing root position (such as root_pos_x/y/z), root quaternion (such as root_quat_x/y/z/w, in xyzw order), and joint angles in robot XML order. Specify the frame rate with --fps.

1. Check that the data can be read correctly:

   uv run python -m humanoidverse.tools.data_inspect \
     --robot configs/robots/g1_29dof.yaml \
     --source <your-CSV-or-NPZ-file-or-directory>
   

2. Generate a data manifest and build the cache:

   uv run python -m humanoidverse.tools.data_build \
     --robot configs/robots/g1_29dof.yaml \
     --source <your-CSV-or-NPZ-file-or-directory> \
     --name <dataset-name> --fps <frame-rate> \
     --out <manifest-path>.yaml
   

   By default, clips are split at 10 seconds (--clip-seconds); clips shorter than 10 seconds but longer than 1 second are retained.

3. Train with --data-manifest <manifest-path>.yaml (Section 4.4). To mix with released data, follow configs/data/g1_lafan_carry_kick_30_40_30.yaml: list multiple sources in one manifest and set their weight values.

See configs/data/example_robot_state_auto_build.yaml for a complete manifest example.

This section trains I-BFM from scratch with the released configuration. It requires 8 H200 GPUs. Run commands inside ibfm/.

The argument names this run, with outputs written to runs/ibfm_reproduction/. Names cannot contain slashes; the script refuses to overwrite an existing directory.

The script uses the released configuration:

The released model was taken at step 166,559,744 of this type of training run.

Training randomization from the task configuration: nominal box mass 0.7 kg, randomized over 0.5–1.0 kg; joint PD gains scaled by 0.75–1.25; control delay 0–1 steps (20 ms per step); foot friction randomized over 0.4–1.25.

Policies in policies/ contain weights and configurations, without the optimizer or replay. Replay is written only once to the final rolling checkpoint when training finishes normally.

Resuming restores the model and optimizer from the rolling checkpoint.

Note Intermediate checkpoints do not contain replay, which must be collected again after resuming. Resumed training therefore differs from uninterrupted training.

Run a short training job on one GPU to confirm the environment works:

Smoke mode uses 16 environments and 2048 environment steps, without W&B logging. run_train.sh wraps the humanoidverse.train entry point and sets cache directories before launching. See scripts/train_ibfm_h200.sh for the full training arguments.

Note The final step in scripts/smoke_release.sh reads humanoidverse/data/examples/g1_robot_state_sample.csv, which is absent from the repository, so that step fails. Use the command above if you only need to check training.

This section uses the released model or your own model for motion tracking, goal reaching, and reward inference. Run commands inside ibfm/. --model-folder points to a directory containing checkpoint/: use tasks/common/checkpoint_ref for the release or runs/<name> for your own model.

The policy encodes the reference motion as intent vectors and tracks it in simulation. --save-mp4 produces a comparison video of the reference and policy execution. Default data: humanoidverse/data/lafan_29dof_10s-clipped.pkl.

Outputs are zs_<index>.pkl (intent vectors) and tracking_<index>.mp4.

The policy reaches a sequence of target poses. The default list is humanoidverse/data/robots/g1/goal_frames_lafan29dof.json; replace it with --goal-json. The default run lasts 2100 steps, switching targets every 100 steps (--episode-len, --goal-switch-interval).

Reward inference requires replay. The released model's replay is on Hugging Face: 8 shards totaling about 210 GiB. Download it from inside ibfm/:

After downloading, each directory from runs/ibfm_166560m_replay/checkpoint/buffers/train_rank_0/ through train_rank_7/ contains buffer.hdf5 and config.json. Use one shard for reward inference:

Outputs are <model-folder>/reward_inference/reward_locomotion.pkl and one video per task.

Note Reading a shard loads the entire file into memory. Each shard is about 28 GB, so ensure sufficient RAM before running.

Tip Intent vectors for the interaction tasks have already been inferred and are stored in each task's latents/ directory. Running Section 2's tasks does not require downloading replay.