WLA³ World Latent Action Modeling for Semantics, Dynamics, and Kinematics

Peidong Liu^*,† Zhiyuan Xiang^* Mingyang Li Wenhao Li Jiale Zhang Jiahao Sun Jiawei Li^†

Joy Future Academy, JD Group

^* Equal contribution. ^† Corresponding authors.

arXiv · coming soon

Overview

Abstract

Scaling Generalist Policy Models with heterogeneous data is limited by the lack of unified, low-noise action supervision. Human egocentric videos are abundant, but only a small fraction comes with high-quality hand-action labels. Observed world transitions offer a common source of action-related supervision across data sources. We introduce WLA³ (World Latent Action Modeling for Semantics, Dynamics, and Kinematics), a unified Generalist Policy Model framework built around representations learned by a World Latent Action Model (WLAM). WLAM encodes synchronized camera views and available embodiment-state changes into a compact local latent action and a richer transition feature. WLA³ reuses these representations across semantics, dynamics, and kinematics through latent action-conditioned world modeling, Semantic Latent Aggregate supervision, and joint latent–robot action generation. The final 32D latent action reaches 67.89% average classification accuracy on LARYBench, while WLA³ achieves 81.9% average success across six real-robot tasks versus 66.2% for π_0.5.

Method

World Latent Action Model

WLA³ learns a shared transition representation from human videos, robot trajectories, simulation rollouts, and UMI demonstrations. WLAM jointly encodes synchronized camera transitions and, when available, embodiment-state transitions. A variational bottleneck produces a compact 32D latent action, while the pre-bottleneck feature retains richer segment-level transition information. Cross-modal dropout and overlap consistency encourage the representation to remain robust across modalities and nearby temporal windows.

Generalist Policy Learning

The learned transition space supports three stages of policy learning. Local latent actions condition the world model on physical change; pooled transition features provide direct SLA supervision to the VLM; and the LARA expert jointly predicts latent-action and embodiment-specific robot-action chunks.

LAC-WM pre-training and Generalist Policy Model training pipeline

Data

Heterogeneous Training Data

The 84.1K-hour corpus combines four complementary data families. Human videos provide scale and behavioral diversity, while robot, simulation, and UMI trajectories add structured state transitions and executable supervision across embodiments.

53,874h Human video

14,151h Real robot

11,676h Simulation

4,393h UMI

Composition, examples, statistics, and robot embodiments in the 84.1K-hour training corpus

Experiments

Real-World Evaluation Protocol

Six tabletop tasks on the AgiBot G1 test fine-grained grasping and placement. Each task is evaluated under spatial perturbations, substantially altered illumination, and unseen tabletop backgrounds while retaining the same instruction and success criterion.

Standard, spatial, illumination, and background evaluation settings

Latent-Action Representation on LARYBench

Under the standard representation-level protocol, WLAM ranks first on both semantic-classification tasks and all four control-regression datasets. It reaches 70.75% average classification accuracy and 0.37 average regression MSE, improving over the strongest competing results on both metrics. Results marked † are quoted from LARYBench; all others are evaluated by us.

Method	Semantic classification (%) ↑			Control regression (MSE) ↓
Method	Composite Human	Composite Robot	Avg.	CALVIN	VLABench	RoboCOIN	AgiBotBeta	Avg.
LAPA^†	14.61	23.64	19.13	0.96	0.95	0.96	1.00	0.97
UniVLA^†	19.08	18.56	18.82	0.82	0.74	0.94	0.97	0.87
villa-X^†	17.80	29.90	23.85	0.86	0.72	0.94	0.97	0.87
DiLA	64.41	62.10	63.26	0.55	0.27	0.86	0.82	0.63
MVP-LAM	50.31	51.61	50.96	0.55	0.38	0.90	0.92	0.69
OpenCLAP	55.32	63.02	59.17	0.43	0.17	0.68	0.60	0.47
DreamDojo	63.69	59.41	61.55	0.35	0.12	0.74	0.81	0.51
Ours	73.12	68.37	70.75	0.25	0.09	0.55	0.57	0.37

Real-World Policy Performance

The full WLA³ model reaches 81.9% average success across the six tasks, exceeding π_0.5 by 15.7 percentage points. The cumulative ablation shows consistent gains as LAC-WM, LARA, and SLA are introduced.

Task-level real-world policy success rates and cumulative component ablations

Scaling with Data

With architecture, initialization, post-training data, and evaluation held fixed, average success increases from 57.6% to 65.9% and 81.9% as Generalist Policy Model mid-training data grows from 5K to 10K and 20K hours.

Policy success rate as mid-training data scales from 5K to 20K hours

Cross-Embodiment Representation Transfer

On twelve action classes shared by human and robot videos, WLAM achieves the best accuracy in all four transfer directions. Its 20.98% cross-embodiment average is 5.27 percentage points above DiLA, showing that its action representation remains more readable as the embodiment changes. H and R denote human and robot videos; all methods use the same linear reader and fixed 4×32-token interface.

Method	Within-domain			Cross-embodiment
Method	H→H	R→R	Avg.	H→R	R→H	Avg.
DiLA	40.75	39.00	39.88	16.17	15.25	15.71
MVP-LAM	27.67	25.96	26.82	12.54	13.75	13.15
OpenCLAP	28.88	40.38	34.63	10.50	10.88	10.69
LAPA	22.21	17.33	19.77	14.21	14.42	14.31
UniVLA	29.25	27.00	28.13	12.21	13.38	12.79
villa-X	24.13	26.04	25.09	9.83	12.29	11.06
WLAM (ours)	49.38	47.38	48.38	22.08	19.88	20.98

Cross-Context Transition Transfer

Latent actions extracted from a source trajectory are applied to a new visual context. Across simulation and real-world examples, WLAM follows the source change while better preserving the objects, appearance, and geometry of the target scene.

Citation

Please use the following BibTeX entry for the current preprint.

@misc{liu2026wla3,
  title  = {WLA$^3$: World Latent Action Modeling for Semantics,
            Dynamics, and Kinematics},
  author = {Liu, Peidong and Xiang, Zhiyuan and Li, Mingyang and
            Li, Wenhao and Zhang, Jiale and Sun, Jiahao and Li, Jiawei},
  year   = {2026},
  note   = {Preprint},
  url    = {https://wla-3.github.io/}
}

WLA3 World Latent Action Modeling for Semantics, Dynamics, and Kinematics