Adapting VLAs to Environment Changes with a Latent World Model

Overview 潜在世界モデルを用いたVLAの環境変化への適応 (Adapting Vision Language Action Models to Environment Changes Using a Latent World Model) Presented as talk 2A1 03 on 3 September 2026 at the 44th Annual Conference of the Robotics Society of Japan (第44回日本ロボット学会学術講演会, RSJ 2026), Kanazawa University Kakuma Campus. Co authors: Satoshi Yamamori, Jun Morimoto (Graduate School of Informatics, Kyoto University). Motivation VLA policies lose a large amount of performance when the environment changes — on real hardware, a lighting change or a camera that has drifted a few centimetres is enough. Two families of methods try to fix this at inference time: Test Time Computing (TTC) — spend more compute, generate action candidates and let a verifier pick the best one (RoboMonkey, RoVer). The evaluation is thrown away after the pick; it never improves the policy. Test Time Adaptation (TTA) — update weights online from interaction (DSRL, EVOLVE VLA). The hard part is obtaining a learning signal from the environment at all. This work closes the loop between the two: the verifier's selection is itself the reward used to update a steering policy at test time, while the VLA stays frozen. Method At inference, per step: 1. A DSRL style steering policy outputs a noise action , added to the base noise . 2. The VLA's action expert takes that noise and generates action candidates . 3. A verifier scores the candidates; the best one is executed. 4. The selection is turned into a GRPO update of the steering policy — top 1 advantage (+1 for the argmax, otherwise), with clipped so the shift cannot run away, and updates…