25 June 2024

Perception Stitching:

Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies


Pingcheng Jian
Pingcheng Jian Duke University pingcheng-jian.github.io
Easop Lee
Easop Lee Duke University easoplee.github.io
Zachary Bell
Zachary Bell Air Force Research Laboratory
Michael M. Zavlanos
Michael M. Zavlanos Duke University www.michaelmzavlanos.org
Boyuan Chen
Boyuan Chen Duke University boyuanchen.com


Vision-based imitation learning has shown promising capabilities of endowing robots with various motion skills given visual observation. However, current visuomotor policies fail to adapt to drastic changes in their visual observations. We present Perception Stitching that enables strong zero-shot adaptation to large visual changes by directly stitching novel combinations of visual encoders. Our key idea is to enforce modularity of visual encoders by aligning the latent visual features among different visuomotor policies. Our method disentangles the perceptual knowledge with the downstream motion skills and allows the reuse of the visual encoders by directly stitching them to a policy network trained with partially different visual conditions. We evaluate our method in various simulated and real-world manipulation tasks. While baseline methods failed at all attempts, our method could achieve zero-shot success in real-world visuomotor tasks. Our quantitative and qualitative analysis of the learned features of the policy network provides more insights into the high performance of our proposed method.

Video (Click to YouTube)

Video Figure


Check out our paper linked here.


Check out our codebase at https://github.com/generalroboticslab/PerceptionStitching


  title={Perception Stitching: Zero-Shot Perception Encoder Transfer for Visuomotor Robot Policies},
  author={Pingcheng Jian and Easop Lee and Zachary Bell and Michael M. Zavlanos and Boyuan Chen},


This work is supported by ARL STRONG program under awards W911NF2320182 and W911NF2220113, by DARPA FoundSci program under award HR00112490372.


If you have any questions, please feel free to contact Pingcheng Jian.


Robot Learning Transfer Learning