PredActor User Guide

PredActor (Predictive Action Diffusion) is an open-source humanoid diffusion control policy. From recent proprioceptive observations and optional task commands, it jointly predicts future whole-body states and joint actions. Predicted states remain internal to the policy for guidance; actions go directly to the joint controller without a separate tracker.

On an onboard Jetson Orin NX, PredActor runs at 50 Hz. The full control callback takes a median 16.790 ms and p95 19.383 ms, both below the 20 ms control period.

Resources:

This section covers how PredActor works, release contents, and environment setup. By the end, you can open the local browser evaluation interface and control a simulated robot with text commands.

DAgger data aggregation and real-robot deployment code have not been released. Running in MuJoCo establishes software and weight compatibility, not real-robot safety; deployment requires separate control, safety, and hardware validation.

1. Read proprioception. Control runs at 50 Hz (0.02-second periods). Each cycle reads the latest 4 observations: joint positions and velocities, projected gravity, base linear and angular velocities, and the previous action, totaling 96 dimensions per step. The text-conditioned policy adds a 512-dimensional text semantic vector.

2. Jointly generate states and actions. A Transformer diffusion model jointly denoises 20 future steps (0.4 seconds) of whole-body states—positions and velocities of 30 bodies plus root state—and 29-dimensional joint actions. The released text-conditioned configuration uses 5 DDIM denoising steps per cycle.

3. Execute actions only. The current action goes directly to the joint PD controller. Predicted future states remain inside the policy and are not passed to a tracker.

4. Two forms of guidance. CFG (classifier-free guidance) strengthens conditions learned during training, such as text commands. CG (classifier guidance, called whole-body guidance in the interface) imposes targets on predicted future states—forward, lateral, and yaw velocities, or pelvis height—steering states during denoising and thereby influencing actions.

5. Proprioceptive observations only. During inference, the state stream is denoised directly from conditions; externally estimated whole-body states are not required as inputs.

Models and data are hosted in MasterYip/PredActor_Artifacts on Hugging Face. Downloads go into the Git-ignored Artifacts/ directory at the repository root:

uv sync --locked creates a Python 3.10 environment from uv.lock, including the evaluator, MuJoCo, browser interface, and training dependencies. Evaluation does not require Isaac Sim.

Tip Evaluation commands use --locked to keep the runtime consistent with the lockfile. uv run --locked uses this environment directly; activation is unnecessary.

--filter selects path prefixes in the Hugging Face repository. checkpoints downloads all three checkpoints in Section 1.3; motions/g1_152fk downloads motion data. After downloading, the script verifies file sizes and SHA-256 hashes.

Existing downloads of matching size are skipped. After a network interruption, rerun the same command to fetch only missing files or those with incorrect sizes.

This command loads the PDP051 policy and MotionCLIP encoder, starts MuJoCo, and opens http://127.0.0.1:8765/ in a browser. A visible robot and increasing STEP count indicate a working evaluator. Enter walk in CLIP command and press Enter; the robot should transition from standing to walking.

Check the training environment with a smoke test that requires no downloaded data:

It generates a small deterministic dataset, performs one optimization step with a reduced model, and checks the saved checkpoint. Success prints PREDACTOR_TRAINING_SMOKE_PASS and the record path. This checks execution paths, not model quality.

This section runs the policy in the browser evaluator. Select behavior with text, control direction with velocity guidance, a gamepad, or destinations, and apply external forces with the mouse. Run all commands from the repository root.

Without additional arguments, the released PDP051 policy is used. Common options:

See all options with uv run --locked predactor-eval --help.

On a server without a display, launch headlessly and forward the port to your local browser:

Replace <username>@<server-address> with your server login. If port 8765 is occupied, add --web-port 8766 to use another port.

1. Select TEXTOP in the right-hand CLIP command panel.

2. Enter an action description and press Enter or click ↑. Suggested examples include walk, jog, run, squat down, jump up, raise hand, and kick.

3. The panel shows the active command. The startup default is stand.

MotionCLIP encodes the text into a 512-dimensional vector used as a policy condition.

Note Training data contains only 152 motions; more complex descriptions are less likely to produce the intended behavior. Start with the simple commands above.

INTERP combines weighted “start → end” semantic directions into a continuously varying condition:

1. Select INTERP in CLIP command. Default propositions are zero vector → stand, stand → walk, walk → run, stand → squat down, and get down → raise hand. Only the first has weight 1 by default; the rest are 0.

2. Drag a row's weight slider. If either endpoint is the zero vector (Ø), the range is 0–1; with text at both endpoints, it is −1 to 1.

3. Use + to add propositions, up to 16. You can edit each start and end text.

4. Click APPLY. The weighted sum is normalized and used as the new condition.

The Whole-body guidance panel imposes velocity targets directly on predicted future states:

1. Confirm the switch beside the panel title is Enabled.

2. Set SOURCE OWNER to WEB for sliders or JOY for a gamepad (Section 2.6).

3. Adjust Forward, Lateral, and Yaw to set forward, lateral, and turning velocity targets.

4. Gain Kp controls guidance strength, default 1.00 and range 0–4. Setting it to 0 disables guidance effects.

Advanced Activation manager options select affected body parts and observations for each guidance type; usually no changes are needed. Click ↺ there to restore defaults.

Text and whole-body guidance can be used together: text selects how to move, such as walking or jogging, while guidance sets direction and speed.

Destination follow automatically derives velocity targets from a destination:

1. Enter the target position in world coordinates in X, Y, and Z.

2. Click SET to set the destination and enable following. The robot turns and moves toward it; once horizontal distance is within the arrival radius, velocity targets become zero and it stops.

3. If pushed farther away than the rearm radius after arrival, following resumes.

4. Click CANCEL to disable following and clear the target.

Controller tuning parameters take effect after clicking APPLY:

By default, the evaluator automatically detects Linux gamepads (--web-joystick-device auto) and uses Xbox mappings (--web-joystick-type xbox_new). A gamepad detected at startup sets whole-body guidance to JOY; the Controller panel displays button states. These controls apply with JOY selected:

Click ? in the Controller panel to view the current gamepad mapping.

Launch with --web-perturb to push the robot with the mouse in the simulation view:

An orange arrow shows drag direction and magnitude. Force and torque limits default to 120 and 30; adjust them with --web-perturb-max-force and --web-perturb-max-torque. ↺ (Reset MuJoCo robot) on the simulation toolbar resets the robot. AUTO RESET resets it automatically after falls. ❚❚ at the top pauses evaluation.

Set run_dir to the output directory printed during training (Section 5.4):

The saved resolved_config.yaml determines which policy and text encoder are loaded. The unconditional baseline (Section 5.2) also accepts text commands, but text has no effect on it.

After evaluation, the output directory contains:

This section runs motion-library entries with the released RLTracker teacher in Isaac Lab and records state–action trajectories for PredActor training. Collection uses a separate conda environment. Run commands from the repository root.

Download the teacher policy and 152 motions in the uv environment:

Default download locations:

The installation script performs these steps:

1. Verify Python 3.10; exit otherwise.

2. Install PyTorch 2.5.1 (CUDA 12.1) and Isaac Sim 4.5.0.

3. Download a pinned Isaac Lab v2.1.0 commit into PredActor/collection/.deps/IsaacLab, installing only the 4 extensions required for collection.

4. Install the repository's RLTracker tasks, RSL-RL runtime, and CUDA forward-kinematics library.

5. Pin versions throughout with PredActor/collection/constraints.txt, then run pip check.

Success ends with PredActor collection environment is ready..

Tip If PyPI is slow, use the Alibaba Cloud mirror for Python packages. PyTorch and Isaac Sim still come from their CUDA / NVIDIA indexes: PIP_INDEX_URL=https://mirrors.aliyun.com/pypi/simple bash PredActor/collection/install.sh After a network interruption, rerun the same command; completed downloads and Isaac Lab code are reused.

Run in the collection environment:

The script checks collection code, teacher weights, and motion files, and outputs JSON. Start collection when status is PASS and motion_count is 152.

Deterministic mode (deterministic) executes each motion from start to finish a fixed number of times: 152 motions × 50 repetitions = 7,600 episodes. Failed episodes are reassigned and excluded from the total. Results go to Artifacts/dataset/g1_152fk_raw/motion.zarr.

See all options with bash PredActor/collection/collect_dataset.sh --help.

Note Once progress exceeds 90%, deterministic mode exits early if no new episode finishes for 1800 seconds, avoiding stalls on unexecutable motions. The actual total may therefore be below 7,600; check the final collection log count before training.

1. Place motions at Artifacts/motions/<motion-set>/<motion-name>/motion.npz.

2. Add --motion_file "<motion-set>/*" during collection. Total episodes equal matched motions multiplied by --samples_per_motion.

3. Use the same pattern for annotation's input.motion_pattern (Section 4.3).

This section attaches a 512-dimensional, text-aligned MotionCLIP semantic vector to every collected frame as a label for text-conditioned training. The project calls this “data dyeing.” Run in the uv environment after leaving the collection environment (conda deactivate).

Confirm these files are in place:

Results go to Artifacts/dataset/g1_152fk/motion_dyed_text_t0.01.zarr, adding a motion_latent field with 512 dimensions per frame.

The released configuration encodes original motions, not collected samples: each original motion is encoded once, then attached to all its episodes using motion_idx. Encoding resamples 50 Hz trajectories to 30 Hz, uses 60-frame windows and xyz positions, and aligns them to the text vocabulary's CLIP vectors at temperature 0.01.

Tip The first run downloads OpenAI CLIP ViT-B/32 weights into the CLIP cache (usually ~/.cache/clip) and requires internet access.

Append parameters as Hydra overrides, for example input.zarr_path=/path/to/motion.zarr:

clip_or_drop truncates the original motion when the sample is shorter; when the sample is longer, it drops the episode.

This section behavior-clones a policy from annotated data for evaluation in Section 2's interface. Run in the uv environment from the repository root.

Except for its last line, this matches the paper-configuration training command in the repository docs. The added training.save_terminal_checkpoint=true saves final weights when training ends; see the note below.

--cfg selects a configuration under PredActor/diffusion_policy/config_files/; subsequent key=value arguments override it. logging.mode=offline logs Weights & Biases locally, while the default online requires a W&B login.

Note Training saves checkpoints/latest.ckpt every 10 epochs, counting from epoch 0. Without training.save_terminal_checkpoint=true, a 50-epoch run ends with epoch 40 weights in latest.ckpt, not the final weights.

All configured guidance is automatically disabled during training; it applies only at inference.

The unconditional baseline uses the same annotated data without text conditioning:

Configuration differences:

Use different --exp_name values to distinguish the two output directories.

Training prints an output directory of the form outputs/<month-day-hour-minute-second>-<config-name>-<exp_name>/, for example outputs/October-16-22-30-05-g1prdp_cond_diffuse-predactor_bc/. It contains:

Next, evaluate this model in the browser following Section 2.8.

This section trains MotionCLIP from scratch. It maps 60-frame motions and text descriptions into a shared 512-dimensional space and is used for annotation in Section 4. This workflow is experimental; use the released encoder if you only want to reproduce policy training.

Obtain two datasets under their respective licenses:

• AMASS motions retargeted to this robot's 29 joints, with one NPZ per motion;

• BABEL annotations, train.json and val.json.

The 152 motions in Section 1.3 are not the full encoder training dataset. Arrange files under PredActor/motion_encoder/ as follows:

Directory names and original sequence names must match BABEL's feat_p paths. For example, ACCAD/<subject>/<sequence>_poses_120_jpos.npz corresponds to BABEL's ACCAD/<subject>/<sequence>_poses.npz annotation. The numeric filename suffix may differ. Each NPZ must contain:

The 30 bodies must follow the order in src/datasets/g1_amass_utils.py, with the pelvis first. Isaac Lab uses a different body order, so reorder it first.

Note Preprocess trajectories to 30 Hz. The parser skips files differing from 30 Hz by more than 5 Hz; it does not convert 120 Hz to 30 Hz, and numeric filename suffixes do not indicate frame rate. Before using the public retargeted dataset mentioned in docs/motion_encoder.md, check its format against the table above. Renaming files does not convert coordinates or body order.

Subsequent commands in this section run inside PredActor/motion_encoder/. The parser includes only BABEL-annotated motions and writes three splits:

Empty splits are not saved. Use a new output directory when regenerating.

amass_30fps_db.pt is a loader placeholder, automatically replaced with _train.pt, _vald.pt, or _test.pt; do not create it. The loader divides sequences into 100-frame chunks and samples 60 consecutive frames per training example, so sequences shorter than 100 frames produce no training samples.

First run a smoke check, saving outputs to a separate directory:

Smoke checks execution, not model quality. At least two samples with different descriptions are needed for a meaningful text–motion contrastive loss.

Full training (alternatively, without activating the environment, run uv run --locked bash PredActor/motion_encoder/scripts/train_g1_xyz.sh from the repository root):

The script follows the original recipe: 100 epochs, batch size 20, learning rate 1e-4, an 8-layer Transformer, 512-dimensional latent space, 16 DataLoader workers, and saves every 10 epochs. The motion model starts from random initialization and the CLIP text tower is trained jointly; the released encoder is not loaded. The first run downloads CLIP ViT-B/32 weights.

After completion, exps/g1-model-xyz-clip/ contains checkpoint_0100.pth.tar, opt.yaml, training.log, TensorBoard event files, and training_receipt.json. The checkpoint includes the motion model and trained CLIP weights, without optimizer state.

Return to the repository root and pass the new checkpoint to Section 4's annotation script:

Then retrain the policy following Section 5.

Note Evaluation text must be encoded with the same encoder used to annotate training data. At training startup, resolved_config.yaml records the evaluation encoder path, defaulting to the released MotionCLIP checkpoint. When training on data annotated with your new encoder, append the following to the Section 5 training command: env_runner.task_cond_providers.clip_teleop.checkpoint_path=PredActor/motion_encoder/exps/g1-model-xyz-clip/checkpoint_0100.pth.tar， This records the new encoder path in training outputs so Section 2.8's evaluation command loads it.