Gaussian Pancakes 🥞:
Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction

Sierra Bonilla, Shuai Zhang, Dimitrios Psychogyios, Danail Stoyanov, Francisco Vasconcelos, Sophia Bano
University College London
arXiv Video Code

3D Gaussian Splatting vs. Gaussian Pancakes

Abstract

Within colorectal cancer diagnostics, conventional colonoscopy techniques face critical limitations, including a limited field of view and a lack of depth information, which can impede the detection of pre-cancerous lesions. Current methods struggle to provide comprehensive and accurate 3D reconstructions of the colonic surface which can help minimize the missing regions and reinspection for pre-cancerous polyps. Addressing this, we introduce “Gaussian Pancakes”, a method that leverages 3D Gaussian Splatting (3D GS) combined with a Recurrent Neural Network-based Simultaneous Localization and Mapping (RNNSLAM) system. By introducing geometric and depth regularization into the 3D GS framework, our approach ensures more accurate alignment of Gaussians with the colon surface, resulting in smoother 3D reconstructions with novel viewing of detailed textures and structures. Evaluations across three diverse datasets show that Gaussian Pancakes enhances novel view synthesis quality, surpassing current leading methods with a 18% boost in PSNR and a 16% improvement in SSIM. It also delivers over 100× faster rendering and more than 10× shorter training times, making it a practical tool for real-time applications. Hence, this holds promise for achieving clinical translation for better detection and diagnosis of colorectal cancer.

Full Pipeline

xxxx

Proposed Gaussian Pancakes pipeline highlighting our contributions: A) RNNSLAM for mesh, camera poses, and depth maps; B) 3D GS initialization, and C) Geometric and depth regularizations, distinguishing our approach from traditional 3D GS.

Method

Visual Results

Contributions

  • Superior Rendering Quality: outperforms current models across all tested datasets.

  • Smoother Reconstructions: surpasses basic 3D GS method.

  • Fewer Artifacts: significantly less than 3D GS method.

  • Greatly Reduced Costs: uses 10x less resources, 100x faster rendering.

BibTeX

@article{Bonilla2024ARXIV,
  author    = {Sierra Bonilla and Shuai Zhang and Dimitrios Psychogyios and Danail Stoyanov and Francisco Vasconcelos and Sophia Bano},
  title     = {Gaussian Pancakes: Geometrically-Regularized 3D Gaussian Splatting for Realistic Endoscopic Reconstruction},
  journal   = {arXiv},
  volume    = {2404.06128},
  year      = {2024},
}