HyperDreamer: Hyper-Realistic 3D Content Generation and Editing from a Single Image

Overall framework


3D content creation from a single image is a long-standing yet highly desirable task. Recent advances introduce 2D diffusion priors, yielding reasonable results. However, existing methods are not hyper-realistic enough for post-generation usage, as users cannot view, render and edit the resulting 3D content from a full range. To address these challenges, we introduce Hyper-Dreamer with several key designs and appealing properties: 1) Full-range viewable: 360◦ mesh modeling with high-resolution textures enables the creation of visually compelling 3D models from a full range of observation points. 2) Full-range renderable: Fine-grained semantic segmentation and data-driven priors are incorporated as guidance to learn reasonable albedo, roughness, and specular properties of the materials, enabling semantic-aware arbitrary material estimation. 3) Full-range editable: For a generated model or their own data, users can interactively select any region via a few clicks and efficiently edit the texture with text-based guidance. Extensive experiments demonstrate the effectiveness of HyperDreamer in modeling region-aware materials with high-resolution textures and enabling user-friendly editing. We believe that HyperDreamer holds promise for adVancing 3D content creation and finding applications in various domains.

arXiv preprint
Tong WU 吴桐
Tong WU 吴桐
Ph.D. Student

My research interests include 3d vision, long-tailed recognition, and robustness.