RealtimeWAM: How Fast Can I Run My World Action Model?

Huanan Liu; Ye Li; Kangye Ji; Xiaoyu Chen; Hanyun Cui; Yutian Shen; Yuan Meng; Chenglei Wu; Jingyan Jiang; Bo Li; Zhi Wang

WORLD ACTION MODELS · REAL-TIME INFERENCE

RealtimeWAM: How Fast Can I RunMy World Action Model?

Parallel execution and adaptive computation for responsive robot control. No additional model training.

Huanan Liu¹ Ye Li¹ Kangye Ji¹ Xiaoyu Chen^1,* Hanyun Cui¹ Yutian Shen¹ Yuan Meng¹ Chenglei Wu² Jingyan Jiang¹ Bo Li³ Zhi Wang^1,†

¹ Tsinghua University² YuanxingGuangnian Robotics³ Nanjing University

* Work done during an internship at Tsinghua University. † Corresponding author.

Read the paper ↗Code on GitHub ↗Watch the robot ↓

01 / REAL-WORLD MANIPULATIONThree shuttlecocks · moving conveyor

React as the scene changes.Conveyor speed: 1 m/min

8.90×

FastWAM speedup

24.09 ms average inference

10.67×

OpenWAM speedup

63.09 ms average inference

+37.2pp

OpenWAM real-world success

33.6% → 70.8% across five tasks

Simulation speedups relative to original Native execution, averaged across four settings. Real-world success is averaged separately over five tasks.

THE IDEA

Compute where it matters.
Overlap what can run together.

World action models predict actions—and sometimes future video—from the current observation. Their inference cost can leave a robot waiting while its environment continues to move.

RealtimeWAM coordinates three decisions: when each branch runs, which observation tokens need fresh features, and how many full Transformer evaluations a prediction receives.

We evaluate the framework on FastWAM and OpenWAM across RoboTwin, LIBERO, LIBERO-Plus, and five real-world tasks.

IN ACTION

Five tasks. Changing scenes.

Recorded task demonstrations.
Videos retain their source playback timing.

Carrot to plate

Pick and place from a moving conveyor.

Hang a mug

Place the handle onto a rack on a level tabletop.

Tennis ball to bowl

Transfer a ball from the conveyor into a bowl.

Intercept a rolling can

Catch and place a can on an inclined tabletop.

Each task was evaluated over 50 trials per backbone and inference configuration. These videos illustrate individual executions.

METHOD

One coordinated inference pipeline.

The execution schedule and the amount of fresh computation are designed together.

Method overview: parallel observation and prediction branches, selective token refresh, and adaptive two- or four-evaluation refinement. — RealtimeWAM combines parallel execution, observation-token reuse, and motion-adaptive predictive refinement.

Overlap execution

Release observation keys and values layer by layer, allowing prediction to overlap the remaining observation computation.

Refresh selected tokens

Use RGB and feature changes to select observation updates. Fill available execution capacity with informative tokens.

Adapt refinement

Decode a coarse action after two full evaluations. Predicted displacement determines whether to add two more.

Explore the execution schedule and token selection +

Native and parallel schedules for action-only and joint video-action architectures. — Parallel schedules for the two architecture families.

RGB-required tokens, feature selection, and the resulting refreshed tokens. — Observation changes guide which cached features are refreshed.

EXPERIMENTS

Success and latency, together.

FastWAMOpenWAM

FastWAM · success rate (%) and measured mean latency
Method	RoboTwin Clean	RoboTwin Random	LIBERO	Plus	Average success	Latency (ms)	Speedup
Native	91.88	91.78	97.60	49.80	82.77	214.33	1.00×
VLA-Cache	80.00	83.20	97.00	48.60	77.20	115.55	1.85×
TeaCache	81.42	90.86	97.05	49.20	79.63	49.28	4.35×
ToCa	91.35	91.24	94.55	49.15	81.57	102.66	2.09×
BAC	91.67	89.53	97.50	51.55	82.56	67.14	3.19×
RealtimeWAM (Ours)	92.04	90.50	97.25	51.20	82.75	24.09	8.90×

OpenWAM · success rate (%) and measured mean latency
Method	RoboTwin Clean	RoboTwin Random	LIBERO	Plus	Average success	Latency (ms)	Speedup
Native	93.64	94.06	99.30	64.75	87.94	672.83	1.00×
VLA-Cache	93.36	89.94	98.40	62.70	86.10	400.26	1.68×
TeaCache	90.42	94.12	98.60	63.80	86.74	245.99	2.74×
ToCa	93.42	93.98	98.10	63.70	87.30	440.42	1.53×
BAC	93.52	91.96	98.90	66.70	87.77	305.08	2.21×
RealtimeWAM (Ours)	92.78	94.02	99.00	63.85	87.41	63.09	10.67×

Averages weight the four settings equally. Caching baselines use compilation and CUDA Graph replay. Speedups use original Native latency as the reference. Hardware: a modified 48 GiB RTX 4090.

REAL-WORLD SUCCESS

Largest gains in dynamic tasks.

Across five tasks, average success improves by 17.2 percentage points on FastWAM and 37.2 points on OpenWAM.

NativeRealtimeWAM

FastWAM

35.6%

52.8%

OpenWAM

33.6%

70.8%

TASK-BY-TASK RESULTS

Success and inference latency on the robot.

FastWAM · five real-world tasks
Task	Native		RealtimeWAM
Task	Success (%) ↑	Latency (ms) ↓	Success (%) ↑	Latency (ms) ↓
Three shuttlecocks	24.0	210.36	40.0	24.33
Carrot to plate	46.0	209.74	74.0	21.86
Hang mug	72.0	211.08	70.0	25.07
Tennis ball to bowl	26.0	210.52	34.0	23.24
Rolling can to bowl	10.0	211.85	46.0	24.11
Average	35.6	210.71	52.8	23.72

OpenWAM · five real-world tasks
Task	Native		RealtimeWAM
Task	Success (%) ↑	Latency (ms) ↓	Success (%) ↑	Latency (ms) ↓
Three shuttlecocks	4.0	640.28	68.0	68.61
Carrot to plate	66.0	638.56	96.0	62.12
Hang mug	82.0	642.15	82.0	65.36
Tennis ball to bowl	16.0	639.47	76.0	64.85
Rolling can to bowl	0.0	643.89	32.0	65.43
Average	33.6	640.87	70.8	65.27

50 trials per task, backbone, and inference configuration. Latency is measured during local inference; averages weight the five tasks equally.

COMPONENT ABLATIONS

Where the gains come from.

OpenWAM on RoboTwin. Each variant changes the indicated component of the full method.

Success rate and measured latency
Configuration	Success (%) ↑	First forward (ms) ↓	Request (ms) ↓
Ours (full)	93.4	30.96	63.76
w/o token reuse	93.8	37.41	72.10
Random tokens	91.3	30.75	62.32
w/o capacity filling	92.1	30.76	61.78
Random gate	91.1	31.02	64.77
Fixed 2F	89.6	31.11	54.70
Fixed 4F	93.7	30.87	95.26

Request is the mean inference latency measured over complete rollouts. Fixed 2F and 4F retain the normal token policy; the random gate preserves the adaptive gate’s budget proportions.

A CLOSER LOOK

Computation changes throughout a rollout.

Observation updates and predictive refinement vary across successive action requests.

Fifty requests from an OpenWAM block-sorting rollout, with observation frames, predicted motion, evaluation counts, and token allocation. — A complete 50-request OpenWAM rollout showing both evaluation budgets and all three token capacities. The illustrated episode ends in task failure.

OPEN SOURCE

Run RealtimeWAM on OpenWAM.

All three components are available as a compact external runtime. Use the original OpenWAM model and weights without editing upstream source.

01 / DEFAULT

Parallel execution

Overlap independent branch computation, with compiled phases, fused operators and CUDA Graph replay.

02 / OPTIONAL

Observation FFN reuse

Refresh selected observation positions and reuse the remaining FFN outputs across requests. Enable with reuse_tokens=True.

03 / OPTIONAL

Adaptive 2F / 4F

Select two or four full Transformer evaluations while retaining ten Euler updates. Enable with adaptive_2f4f=True.

Parallel execution is enabled by default. Token reuse and adaptive refinement introduce approximation and are disabled by default. This release covers OpenWAM; FastWAM and the full closed-loop evaluation suite are outside its scope. Results above are from the paper.

View source ↗Installation & quick start ↗

CITATION

Building on this work?

Please cite the paper if you use RealtimeWAM in your research.

@misc{liu2026realtimewam,
  title         = {RealtimeWAM: How Fast Can I Run My World Action Model?},
  author        = {Huanan Liu and Ye Li and Kangye Ji
                   and Xiaoyu Chen and Hanyun Cui and Yutian Shen
                   and Yuan Meng and Chenglei Wu and Jingyan Jiang
                   and Bo Li and Zhi Wang},
  year          = {2026},
  eprint        = {2610.10079},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  doi           = {10.48550/arXiv.2610.10079},
  url           = {https://arxiv.org/abs/2610.10079}
}

Download BibTeX ↓

THE PAPER

RealtimeWAM

How Fast Can I Run My World Action Model?

Read the full paper ↗