Vision-language-action · Robotic endoscopy

EndoVLA

Dual-Phase Vision-Language-Action for Precise Autonomous Tracking in Endoscopy

From surgeon instruction to continuous endoscope motion.

Chi Kit Ng^1,* · Long Bai^1,2,* · Guankun Wang^1,* · Yupeng Wang¹ · Huxin Gao¹ · Kun Yuan^1,2 · Chenhan Jin¹ · Tieyong Zeng¹ · Hongliang Ren^1,†

¹The Chinese University of Hong Kong · ²Technical University of Munich · Conference on Robot Learning (CoRL) 2025

* Equal contribution (co-first authors) · † Corresponding author — hlren@ee.cuhk.edu.hk

CoRL 2025

Paper PDF Project video

EndoVLA tracking a polyp — Prompt-conditioned polyp tracking in a dynamic stomach phantom.

3Endoscopic tracking tasks

2Training phases

CoRL 2025PMLR volume 305

01 / Overview

Why EndoVLA

One policy for visually different tracking tasks.

Autonomous tracking can reduce the attention required to keep lesions and procedural landmarks in view. Conventional systems split perception, planning, and control into separately tuned components, making them difficult to adapt when anatomy, targets, or instructions change.

EndoVLA maps endoscopic images and surgeon-issued prompts directly to robot actions. The same policy handles polyp tracking, abnormal-mucosa following, and circular-marker tracking for circumferential cutting.

Language grounding

Surgeon prompts identify the target and desired tracking behavior.

Dual-phase training

Supervised learning establishes the policy; task-aware rewards refine it.

Transfer

The policy generalizes to new scenes and longer sequential tasks.

02 / Method

Training strategy

Learn the motion, then refine the behavior.

The first phase uses supervised fine-tuning on EndoVLA-Motion to learn the relationship among image observations, language prompts, and robot motion. The second phase applies reinforcement fine-tuning with task-specific rewards.

This division addresses two practical limitations of robotic endoscopy: limited demonstrations and domain shift between controlled training data and changing endoscopic scenes.

EndoVLA dual-phase fine-tuning architecture: Phase 1 supervised fine-tuning and Phase 2 reinforcement fine-tuning — Dual-phase fine-tuning (DFT) architecture. (a) The robotic endoscope: EndoVLA outputs a 2-DoF motor angle that drives the cable-actuated bending segment. (b) Phase 1 — supervised fine-tuning: a frozen vision encoder and a trainable MLP projector align image tokens with the language embedding space of the LLM, which is adapted by a Vision-SFT LoRA and decoded into a bounding box and a 2-DoF motor angle. (c) Phase 2 — reinforcement fine-tuning: the SFT model is refined by verifiable IoU, format, and motion-accuracy rewards. Figure 2 of the CoRL 2025 paper.

03 / Project video

CoRL presentation

The complete project in 81 seconds.

The narrated video introduces the tracking problem, training strategy, experimental platform, and representative results.

04 / Results

Evaluation

Tracking across targets and domains.

The experiments evaluate lesion tracking, mucosal-region following, and adherence to predefined circular markers. The paper reports improved endoscopic tracking and zero-shot transfer to general scenes and more demanding sequential tasks.

EndoVLA general-scene tracking — General-scene tracking illustrates transfer beyond the stomach-phantom training setup.

05 / Citation

Publication

EndoVLA at CoRL 2025.

If this work is useful, please cite the PMLR publication.

@inproceedings{kit2025endovla,
  title={EndoVLA: Dual-Phase Vision-Language-Action
         for Precise Autonomous Tracking in Endoscopy},
  booktitle={Conference on Robot Learning},
  year={2025}
}