Modular Gaussian Splatting: Instance Decomposable Learning and Adaptive Rendering of 3D Scenes via Mixture of Experts

Anonymous1,
1Anonymous
Paper Link Coming Soon Code Coming Soon Data: Lerf-mask Data: Object-Nerf Data: Tanks&Temples + DeepBlending Data: Mip-NeRF360
(a) Original view
(b) Add figurines
(c) Add bear and toy
(d) Add scaled lego
(e) Rotate instances of desktop

Fig. 1. We introduce a method that decompose the 3D scene into numbers of instance representations via Mixture of Experts (MoEs) of 3D Gaussian Splatting within minutes on a single GPU, as shown in the top half of this figure. The instance representations can be edited, animated, composited, etc. with very realistic Gaussian Splatting rendering, offering new possibilities for Computer Graphics. Note for example that we captured different instance representations from multiple datasets, and make the composited scene in the bottom half of this figure. The project page provides more examples, including a video illustrating our results.

Abstract

This paper introduces Modular Gaussian Splatting (Modular-GS), a novel method that leverages 3D Gaussian Splatting and Mixture of Experts (MoEs) for decomposing and representing 3D scenes as a combination of Instance Gaussians.

Modular-GS achieves scene decomposition by inputting multi-view data and automatically generated masks, facilitating fine-grained modeling of individual objects. Our approach enables controllable editing and dynamic rendering of the scene by selectively combining different experts, supporting instance insertion across datasets.

Experimental results demonstrate that Modular-GS improves scene modeling quality and efficiency, offering new possibilities for Radiance Field Rendering. This work extends the boundaries of 3D scene representation and editing, advancing techniques for semantic understanding and real-time rendering.

Modular-GS enables both temporal and spatial dynamic adaptive rendering.

The Final Result of the Above Video.

BibTeX

@article{
      }