Skip to content
AI NEWS JOURNAL
Hugging Face Official BlogAI NEWS / JOURNAL

NVIDIA releases Cosmos3-H-Dreams: the first real-time generative surgical simulator based on clinical videos of actual cholecystectomies.

NVIDIA has launched Cosmos3-H-Dreams, the first real-time generative surgical simulator trained on clinical cholecystectomy videos and robotic kinematics data. It supports interactive, low-latency visual simulation based on motions captured via the Versius surgical robot platform and runs on a single RTX PRO 6000 GPU.

NVIDIA officially launched Cosmos3-H-Dreams in October 2026, the world’s first real-time generative surgical simulator trained on synchronized data from actual clinical gallbladder removal videos and robotic kinematics. The system collects clinical procedure data on CMR Surgical’s Versius surgical robot platform, enabling users to drive the generation of continuous video frames via real-time motion commands for instruments such as the Cadiere grasping forceps and monopolar hook—covering 28 degrees of freedom including pose, gripping force, and energy output—while maintaining response latency that meets interactive requirements. It can run in real time on a single NVIDIA RTX PRO 6000 GPU.

Unlike its predecessor, Cosmos-H-Dreams—which focused on desktop suturing tasks—Cosmos3-H-Dreams directly addresses the complexities of the in‑body surgical environment: minute instrument movements give rise to strong coupled visual dynamics, such as tissue traction, structural occlusion, specular reflection shifts, and electrocautery smoke interference. This demands that the model achieve long‑term temporal coherence under motion conditions while preserving anatomical structure and geometric consistency with the instruments. To this end, it leverages Cosmos3-H-Embodied-Nano (16 billion parameters) as its base model, fine‑tunes it using gallbladder removal–specific data to construct a bidirectional teacher model, and then employs two‑stage distillation to produce a causal student model—the first version uses four denoising steps, while the second further compresses this to two steps, striking a balance between visual quality and inference speed.

Technically, the system integrates FlashDreams to accelerate streaming inference and introduces an enhanced FadeMem module that dynamically compresses the attention key-value cache, addressing memory bloat in long-sequence generation. The training strategy combines “causal teacher forcing” with “self-forcing,” enabling the student model to continuously calibrate itself against its own generated historical context and thereby suppress error accumulation. Demonstrations show that, for the same initial frame, different sequences of device motions can produce distinct, physically intuitive tissue deformations and device responses, validating its generalization capability to unseen trajectories.

NVIDIA explicitly emphasizes that Cosmos3-H-Dreams is a purely research‑oriented tool, not a clinically validated medical device, and is not intended for patient diagnosis or treatment. Its current capabilities are limited to specific gallbladder removal procedures and support only the dual‑instrument collaborative motions captured by the Versius platform. All model weights, training code, datasets (Open-H-Embodiment), adaptation guidelines, and the FlashDreams runtime library have been made open source, enabling researchers to integrate their own video‑action data and controllers for transfer learning—provided they strictly adhere to the original action representations and normalization specifications.

Future directions include: covering the entire workflow of cholecystectomy and defining quantifiable task boundaries and success criteria; integrating numerical simulations (such as explicit tissue mechanics modeling) to enhance biomechanical interpretability; conducting surgeon-led assessments of anatomical accuracy and procedural feedback; and extending to multi-instrument collaboration and closed-loop training of autonomous strategies. These remain open research topics, with no clinical deployment capabilities yet established.

Sources

Read original ↗

Related topics

← Continue to all updates