INTACT User Guide

INTACT (INtent-To-ACTion) is an open-source, end-to-end JEPA world model. It extends LeWM with an intent-to-action Actor learned from action-labeled, reward-free offline trajectories. At deployment, the model takes the current and goal images and directly outputs an action chunk, without large-scale action search such as CEM.

With just 1 training epoch per task, search-free Direct inference achieves an average success rate of 95.61% across four simulated tasks—PushT, Cube, Reacher, and TwoRoom—with planning latency of 3.9–4.8 ms.

Resources:

This section covers how INTACT works, its two runtimes, released models, and environment setup. By the end, you can install the environment, prepare the data, and complete a single-task smoke training run on your GPU machine.

The released training configurations and evaluations target these four simulated tasks.

INTACT retains LeWM's Encoder and Forward Predictor and adds an Actor:

1. Encode. A ViT-Tiny/14 Encoder maps each 224×224 RGB image to a 192-dimensional latent z_t. The Forward Predictor predicts the next latent from previous latents and actions, answering what will happen after an action.

2. Construct intent. Intent m_t is the difference between two latents, with two instances:

   • Local intent, m_t = z_{t+1} − z_t: the change that actually occurred in this transition, with gradients propagated normally;

   • Goal intent, m_t = sg(z_g) − z_t: the remaining displacement to the goal, with gradients stopped at the goal.

3. Read out actions. Both intents use the same Actor and four-slot input [z_t, m_t, z_t ⊙ m_t, A(a_{t−1})] to produce a diagonal Gaussian action distribution. A(a_{t−1}) encodes the previous action.

4. Train end to end. Each 8-frame window contains 7 adjacent transitions. For each transition, the same demonstrated action supplies separate local-intent and goal-intent negative log-likelihood losses. These are optimized with the forward prediction loss and SIGReg regularization, propagating action gradients into the Encoder. No distance constraint is imposed between the two intents.

5. Direct inference. Encode the current and goal images, use the Actor's mean as the action, and advance one step with the Forward Predictor. Repeat for 5 steps to produce an action chunk. After executing it, replan from the next real observation. No candidate action sequences are evaluated.

The Actor parameter structures differ: paper checkpoints use the older five-slot input, while the current runtime uses four slots. Do not load paper checkpoints with scripts/eval.sh or modify files in paper_runtime/; their fingerprints match those recorded for the paper evaluation.

The main branch of the Hugging Face repository INTACT-JEPA/INTACT contains three model groups. E1 and E5 mean 1 and 5 training epochs, respectively.

The same 72 paper checkpoints are also published at the fixed revision paper-e5-goal-v1, which the download script in Section 5 uses. See Section 5.5 for success rates.

install.sh creates .venv in the repository, installs torch==2.6.0 and torchvision==0.21.0, then all dependencies pinned in requirements-cu124.lock, and finally runs pip check and scripts/verify_install.py. It uses uv when available, otherwise Python 3.10's built-in venv and pip.

Without an NVIDIA GPU, you can only check imports and configurations; training and evaluation are unavailable:

Note After installation, do not separately install unpinned PyTorch or torchvision versions. Mixing CPU and CUDA wheels commonly causes missing-operator errors during import. If the installation script cannot be used, follow “Manual equivalent” in docs/INSTALL.md.

Edit .env and fill in four variables:

Variables already exported in the current shell take precedence over .env. Load and inspect the effective paths:

fleet_env.sh sets stable-worldmodel's cache root, STABLEWM_HOME, to INTACT_OUTPUT_HOME. Checkpoint-relative paths below start at $INTACT_OUTPUT_HOME/checkpoints/.

Tip A few commands locate data through STABLEWM_HOME instead of LOCAL_DATASET_DIR: format conversion in Section 1.8 and paper evaluation in Section 5.2. Setting LOCAL_DATASET_DIR and INTACT_OUTPUT_HOME to the same directory avoids extra arguments for those commands.

INTACT uses the official LeWM datasets; training scripts do not download them automatically. All four archives are in the LeWM Hugging Face collection:

Download and extract them into $LOCAL_DATASET_DIR/datasets/. The hf command is installed in .venv with the pinned huggingface_hub:

PushT training reads Lance format, so convert HDF5 once. Return to the repository directory and run:

The converter reads and writes files under $STABLEWM_HOME/datasets/. Prefixing the command with STABLEWM_HOME="$LOCAL_DATASET_DIR" performs conversion in the data directory; omit it if the two roots are identical.

The resulting directory layout is:

Note Do not delete pusht_expert_train.h5 after conversion. Training reads .lance, but Official evaluation reads initial and goal states from pusht_expert_train.h5.

Note Reacher's evaluation configuration (config/eval/reacher.yaml) reads dmc/reacher_random, or datasets/dmc/reacher_random.h5, rather than the reacher.h5 shown above. A missing-file error for this path means the evaluation dataset is absent locally.

Run the three checks from the repository root, followed by a smoke training run:

A smoke run checks execution paths, not model accuracy. With 4 GPUs, also check four-task training:

Note Output directories are never overwritten. If a log with the same name exists, the launcher exits with Log already exists. Use a different --run-name when rerunning.

This section trains one INTACT model for one task: the paper's single-task E1 setting. Run all commands from the repository root; train.sh loads .env automatically.

The first two arguments to train.sh are the intent coordinates and task, in that order.

Replace pusht with cube, reacher, or tworoom for another task, or goal with waypoint for the comparison coordinates. The paper's single-task setting trains on all data for 1 epoch, as in the command above.

train.sh handles the following options itself and forwards all remaining arguments unchanged to Hydra:

Append Hydra overrides in key=value format, such as loader.num_workers=4. Common parameters and defaults from config/train/intact_goal.yaml are:

Note Training uses Math SDPA (sdpa_backend: math). When comparing with the paper, keep these defaults and change only seed. Also change --run-name, or omit it so the script names the run by seed; reusing a name stops training because the directory already exists.

At startup, the script prints intent coordinates, task, run name, data directory, output directory, and log path. Progress is printed at step 1, every 100 steps, and the final step:

The same records are appended to checkpoints/<run>/metrics/train.jsonl. In run_metadata.json, status progresses through configured and running to complete on success or failed on error, with the exception type recorded. Successful training ends with:

$INTACT_OUTPUT_HOME/checkpoints/<run>/ contains:

Full terminal output is saved in $INTACT_OUTPUT_HOME/logs/<run>.log. If the checkpoint directory already contains files, training stops with FileExistsError instead of overwriting results.

This section trains the paper's four-task shared-Encoder model (E5). Four processes each handle one task on one GPU. They share the Encoder and projector; the Forward Predictor, action encoder, and Actor are task-specific.

• All four datasets are in place, and bash scripts/check_fleet.sh passes.

• Exactly 4 GPUs are available. If CUDA_VISIBLE_DEVICES is set, it must list exactly 4 GPUs or the script exits.

• The process-to-task mapping is fixed: rank 0 is PushT, rank 1 Cube, rank 2 Reacher, and rank 3 TwoRoom.

The script launches 4 local processes with torch.distributed.run, synchronizing shared gradients at every step. Defaults: fused AdamW, constant learning rate 5e-4, weight decay 1e-3, SIGReg weight 0.03, 5 epochs, batch size 256 per task, Math SDPA, and deterministic algorithms enabled.

Note Do not reduce --batch-size when reproducing the paper's shared-Encoder results. If GPU memory limits require changing it, explicitly report that change.

Each process prints progress at the same intervals and writes its own records:

At the end of each epoch, the script checks that shared-parameter hashes match across all 4 processes, stopping with Shared states diverged if they differ. It then saves a checkpoint for each task and prints MULTITASK_EPOCH_COMPLETE=. Output layout:

Tip Task directories contain one .pt file per epoch. Specify a particular file for evaluation, such as intact_multitask_goal_s3072_e5_pusht/weights_epoch_5.pt. Supplying only the directory fails because it contains multiple .pt files.

This section evaluates checkpoints in simulation. scripts/eval.sh is the single entry point for Official LeWM evaluation; CLEAR-LeWM v0.8 uses clear_eval.py separately, as described in Section 4.5.

Guarded A uses planning horizon H=5, replanning horizon RH=5, initial sampling standard deviation 0.25 in raw action space, and retains the top 16 candidates each round. A final deterministic reevaluation of the mean plan is recorded separately and excluded from the 384 candidates.

Note Guarded A parameters are locked by the evaluation code. Changing any of them through Hydra, such as solver.num_samples, stops evaluation with Guarded A configuration mismatch.

Arguments from the sixth onward are Hydra overrides. Each episode samples an initial state from expert data, uses the state 25 steps later as its goal, and executes at most 50 steps.

The paper protocol evaluates each checkpoint with three seeds, 100 episodes per seed:

For a four-task model, the task argument must match the checkpoint directory:

direct and guarded_a use the same causal action history:

1. At the sampled starting time t, the Actor first receives the 5 preceding dataset actions, rows[t-5:t]. Raw zero actions are prepended only if fewer than 5 prior steps exist at the true episode boundary.

2. After each step, the controller's actually executed action is shifted into the history.

3. Neither the current row's action row[t] nor the goal action is supplied to the Actor.

cem disables the Actor and does not use action history. Official and CLEAR are scoring protocols, not history modes.

Results are saved under $INTACT_OUTPUT_HOME/<run>/, outside checkpoints/:

.txt.json records the inference mode, scoring protocol, checkpoint SHA-256, evaluation seed, configuration hash, SDPA settings, host and GPU, evaluation-code file hashes, metrics, sampled episodes and starting points, and summaries of planning time and candidate counts (totals and means).

CLEAR-LeWM is a separately versioned evaluator with reset manifests and success criteria that differ from Official. Install its pinned environment following its instructions, then run from the INTACT repository root:

Replace /path/to/CLEAR-LeWM-v0.8 with your local CLEAR-LeWM directory. --mode accepts direct, pure_cem, or guarded_a.

Note Official and CLEAR success rates answer different questions. Report them separately instead of combining them into one average.

This section downloads the paper's four-task shared-Encoder checkpoints, reproduces its main table with the frozen paper runtime, and explains how to use the single-task models on Hugging Face.

The paper matrix has 6 comparison cells, each with 3 training seeds (0, 42, 3072) and 4 tasks per seed, totaling 72 checkpoints:

The script downloads anonymously from revision paper-e5-goal-v1 of INTACT-JEPA/INTACT. It verifies SHA256SUMS itself and each archive's hash, extracts to $STABLEWM_HOME (that is, INTACT_OUTPUT_HOME), and verifies each checkpoint's size, hash, and companion files. Success ends with Paper checkpoints installed under …/checkpoints.

• To use another directory, append a path argument, for example bash scripts/download_paper_checkpoints.sh matrix all /path/to/cache.

• To use a mirror, set INTACT_HF_REPO, INTACT_CHECKPOINT_REVISION, or INTACT_CHECKPOINT_BASE_URL.

Training seeds must be 0, 42, or 3072; evaluation seed defaults to 0, and episode count to 100. eval_paper_matrix.sh uses CEM 300×30 for lewm and search-free Direct for the other five cells. Both scripts verify checkpoint hashes, then prepend paper_runtime/ to PYTHONPATH for evaluation.

Scripts read checkpoints from $STABLEWM_HOME/checkpoints/. If you downloaded elsewhere, first set PAPER_CHECKPOINT_HOME to that directory, the parent of checkpoints/. Results go to $STABLEWM_HOME/<checkpoint-directory-name>/ as pusht_results.txt, ogb_cube_results.txt, dmc_results.txt, or tworoom_results.txt, plus a corresponding .json file.

Note Paper evaluation scripts look for data in $STABLEWM_HOME/datasets/ and do not read LOCAL_DATASET_DIR. If the roots differ, append cache_dir="$LOCAL_DATASET_DIR", for example bash scripts/eval_paper_direct.sh pusht 0 0 100 cache_dir="$LOCAL_DATASET_DIR". With eval_paper_matrix.sh, supply all first 5 arguments before adding this override.

Official Direct success rates (%) for each training seed in the goal_intact cell. Each value averages evaluation seeds 0, 1, and 42, with 100 episodes per seed:

See Section 5.5 for the full six-cell results.

The directory contains three archives, intact-goal-e1-single-seed<seed>.tar.gz, about 310 MB each. Each contains weights for four tasks at checkpoints/intact_goal_<task>_s<seed>_p77_v1/weights_epoch_1.pt; extract to $INTACT_OUTPUT_HOME to place them under checkpoints/. RESULTS.csv lists success counts over 100 episodes for each of 36 evaluation cells (task × training seed × evaluation seed), allowing cell-by-cell comparison.

Note These archives contain only weights_epoch_1.pt, without config.json. Because scripts/eval.sh requires config.json alongside the .pt file, the extracted checkpoints cannot be evaluated directly.

Tip The MANIFEST.json supplied with INTACT/ archives reports a macro average of 95.75% for these three training seeds. The project's published 95.61% uses a different seed set (3072, 3073, 3074); refer to AUDITED_RESULTS_20260914.json.

INTACT-unified/ contains the same 72 paper checkpoints as paper-e5-goal-v1; download them with Section 5.1's scripts. The Actor in INTACT-no-previous-action/ has three input slots, unlike the current four-slot Actor, and the repository provides no evaluation script for it.

All values below are success rates (%) under Official LeWM. ± denotes sample standard deviation across training seeds.

Current project results

Single-task E1 results by task (training seeds 3072, 3073, 3074)

Four-task shared-Encoder E5 paper matrix (training seeds 0, 42, 3072)

See docs/RESULTS.md for the full single-task inference comparison and variance of the main shared-Encoder cells. Sample standard deviations for all six cells and hashes for every checkpoint are in checkpoints/PAPER_E5_MATRIX_MANIFEST.json.

This section explains which information to retain and how to aggregate success rates when reporting INTACT results, following docs/REPRODUCIBILITY.md.

Every reported result should be traceable to the following information. Training and evaluation scripts record as much as possible automatically (see run_metadata.json in Section 2.5 and .txt.json in Section 4.4); fill in anything missing manually:

• Source commit and fully resolved configuration

• Dataset, scoring protocol, and task

• Training seed, evaluation seed, epoch count, and batch size per task

• Inference mode and SDPA backend

• Checkpoint SHA-256 and training completion status

• Machine-readable evaluator output

1. For each independently trained checkpoint, first average success rates over evaluation seeds 0, 1, and 42, with 100 episodes per seed.

2. Then compute the mean and sample standard deviation across training seeds and report “mean ± standard deviation.”

3. Report single-task and shared-Encoder results separately.

4. Report Official and CLEAR-LeWM results separately.

5. Mark cells that were not run or were stopped intentionally as --, not 0.

6. Label post-hoc selections and preliminary experiments separately; exclude them from controlled comparisons.

• The 3.9–4.8 ms figure measures planning only. It excludes environment rendering, observation transfer, and other overhead, and is not end-to-end robot control latency.

• Candidate count means action sequences evaluated per plan, not environment steps.

• Published LeWM paper numbers use its own 10-epoch + CEM protocol and serve as background, not a paired comparison. Use the paper matrix in Section 5.5 for controlled comparisons.