Unifying Vision-Language-Action and Latent World Modeling

TL;DR A robot foundation model that predicts future spatial tokens in its VLM's co-evolving visual space and denoises them jointly with continuous actions.

Autonomous closed-loop execution, 3× speed

Vision-language-action models (VLAs) map current observations directly to actions. While they inherit the semantic and visual priors of a pre-trained VLM, they learn from sparse action labels alone and do not explicitly model how a scene will evolve.

World action models (WAMs) capture physical dynamics by adapting a video generator to predict future frames alongside actions, improving control at the cost of a large video diffusion backbone. Recent latent-space policies avoid pixel synthesis, yet either align intermediate policy features to compact vectors without generating a spatial future stream, or predict features of frozen external encoders such as DINOv2. As the policy's own vision encoder adapts to robot data, its perceptual space and the predicted future drift apart.

Z-sight predicts the future directly in the visual token space of its own VLM. An online teacher tracks the policy's adapting vision encoder to supply co-evolving future targets, while a foresight expert and an action expert jointly denoise future spatial tokens and continuous actions.

VLA

VLMvision encoder
Action expert
actions

WAM

Video diffusion
future video
Action expert
actions

Z-sight

VLMvision encoder
Foresight expert
future visual tokens
Action expert
actions
VLAs map current observations to actions without explicit foresight. WAMs build on a video generator to predict future pixels alongside actions. Z-sight co-denoises future spatial tokens and actions in the VLM's own representation space, allowing the action expert to read the forming prediction.

How it works

Z-sight is a 3.18B-parameter mixture-of-transformers with three streams. A Qwen3.5-2B VLM encodes the language instruction, proprioceptive state, and a short camera history, extending its vision tower with parameter-free causal temporal attention to capture motion within a single-frame token budget. Beside the VLM sit two 0.48B flow-matching experts: a foresight expert that predicts future spatial tokens and an action expert that generates a 30-step continuous action chunk.

At inference, the VLM prefix is encoded once with cached keys and values. Both experts then initialize from Gaussian noise and integrate over ten shared Euler steps. At every step, the foresight expert sharpens its estimate of the future visual tokens while the action expert attends directly to that forming prediction to refine the action chunk, without pixel decoding.

Present tokens

Head-camera frame: Unitree G1 hand holding a fruit plate in front of a closed fridge

Future tokens

Later head-camera frame from the same episode: the fridge door is open

Action chunk

step 0 / 10
Schematic of joint denoising over ten shared steps. Each cell shows the average color of an image patch from a Unitree G1 head-camera frame, standing in for the high-dimensional token vector predicted by the foresight expert. The inset in the middle panel shows a later frame from the same episode, which supervises training through the EMA teacher and is unavailable at test time. Action curves are illustrative.

Because perception and prediction share the same encoder family, backpropagating the future-prediction loss into the VLM could cause representation collapse or erode pre-trained knowledge. Z-sight prevents both by supervising the foresight expert against a stop-gradient exponential-moving-average (EMA) teacher of the vision encoder and blocking future-prediction gradients from entering the VLM. The vision encoder adapts solely through action supervision, while its EMA teacher keeps future targets aligned in the same space.

Training proceeds in three stages across 4,731 hours of video and robot trajectories. We first pre-train the VLM with discrete FAST action tokens on roughly 2,000 hours of egocentric human video from EgoDex and EgoVerse. We then introduce both experts under knowledge insulation on roughly 2,750 hours of cross-embodiment robot data from AgiBot World and Humanoid Everyday, replaying human video at 20% of the sampling mass. Finally, we post-train end-to-end on demonstrations from the target robot.

On a real humanoid

We evaluate Z-sight on a physical Unitree G1 equipped with two 7-DoF dexterous hands and a single egocentric head RGB camera. All compared policies are adapted on the same demonstrations and output 30-step chunks at 10 Hz to a shared 50 Hz SONIC whole-body controller. The six tasks range from short tabletop pick-and-place to multi-stage loco-manipulation combining bipedal walking, torso bending, appliance articulation, and object placement. All videos below show autonomous closed-loop rollouts.

Across 10 trials per task from matched initial configurations, Z-sight reaches 73.3% average success, outperforming the strongest baseline Fast-WAM (53.3%) by 20.0 points while matching or exceeding every baseline on every task. The advantage is most pronounced on the four multi-stage tasks (Chair, Book, Trash, and Fruit), where action-only VLAs frequently stall after completing the initial subgoal.

Real-world results on the Unitree G1 over 10 trials per task. Solid bars and labels report full-task success rate, overlaid on lighter bars showing stage-weighted progress. π0.5, GR00T N1.7, and Ψ0 are VLAs, and Fast-WAM is a video-based world action model.

In simulation

On the SIMPLE whole-body humanoid loco-manipulation benchmark, all methods are adapted on the same 17-task dataset and evaluated across three domain-randomization levels on the 12 teleoperation tasks. Z-sight achieves 81.9% average success, outperforming GR00T N1.7 (72.3%), Fast-WAM (66.0%), and Ψ0 (65.8%). It leads at every difficulty level, maintaining 73.1% success at L2 under randomized target positions.

Results on SIMPLE. Left: success rates by difficulty level and overall average, with error bars showing ±1 standard deviation across three evaluation passes. Right: per-task success rates across the 12 teleoperation tasks (hover to inspect values).

The same mid-trained model also transfers competitively to single-arm (LIBERO) and bimanual (RoboTwin 2.0) tabletop benchmarks, with full comparisons reported in the paper.

Retaining online foresight adds modest compute at deployment. On a single RTX 4090, Z-sight generates a 30-step action chunk (3.0 s of execution at 10 Hz) in 109.5 ms, running faster than π0.5, Ψ0, and Fast-WAM even though Fast-WAM skips video generation at test time.

Median in-process action-chunk latency on one RTX 4090 over 100 CUDA-synchronized calls (480×640 RGB). Shorter is faster. All methods use 10 denoising steps except GR00T N1.7, which uses 4.

Does the policy use its foresight?

A policy trained with an auxiliary prediction loss could still ignore its predicted stream at deployment. Holding model weights fixed, we intervene at test time on both the content of the foresight stream and its declared noise schedule σ.

Closed-loop success rates (%) with fixed weights under test-time foresight interventions. RoboTwin 2.0 is evaluated under clean (C2C) and domain-randomized (C2R) conditions.

Injecting Gaussian noise while declaring the stream uncertain (σ = 1) preserves most of the policy's performance, because training with σ ≥ τ teaches the action expert to fall back on current observations when foresight is unreliable. Passing the same noise under the shared σ = τ schedule causes success to collapse, and feeding a valid prediction from another episode degrades control even further despite matching token statistics and the denoising schedule. Closed-loop control therefore relies on both declared uncertainty and scene-specific predicted content.

To verify that better future predictions directly improve sampled actions, we hold initial action noise fixed on held-out Unitree G1 trajectories and vary only the future tokens supplied to the action expert.

Normalized open-loop action-chunk MSE against held-out Unitree G1 demonstrations.

Substituting ground-truth future tokens lowers action MSE by 11.9% relative to the model's own prediction, whereas repeating clean present-frame tokens increases error by 28.3%.

Finally, we test whether alternative latent targets or extra model capacity explain the training gains, comparing controlled variants adapted from the same mid-trained checkpoint.

Success rates (%) across future-supervision designs and training ablations from the shared mid-trained checkpoint.

Generating an explicit spatial future stream outperforms FLARE-style implicit feature alignment on the same DINOv2-B targets, and predicting in the policy's co-evolving native token space yields a further jump (+13.5 points on RoboTwin C2R). Keeping the full 3.18B architecture while disabling the future-prediction loss drops success to 25.0% on SIMPLE Hard and 11.6% on RoboTwin C2R, confirming that the gains come from continued predictive supervision rather than extra parameters.

Citation

@misc{ye2026zsight,
  title  = {Z-sight: Unifying Vision-Language-Action and Latent World Modeling},
  author = {Ye, Junjie and Zhao, Yiqi and Xu, Muchen and Wang, Celina Shiyu and
            Dong, Hao and Ge, Ruohai and Xue, Rong and Kolligs, Benjamin J and
            Dutt, Aditya and Nagavalli, Sasanka and Samad, Nassim A. and
            Sukhatme, Gaurav S. and Deshmukh, Jyotirmoy V. and Wang, Yue and
            Guo, Yuliang},
  year   = {2026}
}