HandDreamer：使用矫正性手形引导的零样本文本到 3D 手模型生成
HandDreamer: Zero-Shot Text to 3D Hand Model Generation using Corrective Hand Shape Guidance
针对 SDS 在关节物体上因文本概率地形多峰导致的视图不一致问题，提出用 MANO 初始化加手部骨架引导扩散提供结构先验，并设计矫正性手形引导损失把各视图拉回一致模态——这套「参数化模板先验 + 视图一致性正则」的思路可直接迁移到人体、动物等铰接物体的文本到 3D 生成。
原文依据 we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes
摘要译文
由模型翻译 · 术语统一 · 请对照原文核对关键结论虚拟现实的出现使得在虚拟世界交互中需要生成细节丰富且可定制的 3D 手模型。然而，当前的 3D 手模型生成方法既昂贵又繁琐，对用户提供的可定制性非常有限。尽管零样本文本到 3D 合成的最新进展已能借助分数蒸馏采样（SDS）生成多样且可定制的 3D 模型，但这些方法难以很好地泛化到 3D 手模型生成，会产生不自然的手部结构、视图不一致以及细节丢失。为解决这些局限，我们提出 HandDreamer，这是首个从文本提示进行零样本 3D 手模型生成的方法。我们的发现表明，SDS 中的视图不一致主要源于文本提示所描述的概率地形存在歧义，导致相似的视图收敛到分布的不同模态。由于手部在关节活动和姿态上变化很大，这一问题在手部上尤为严重。为缓解该问题，我们提出使用基于 MANO 手模型的初始化以及手部骨架引导的扩散过程，为手部结构提供强先验，并确保视图与姿态的一致性。此外，我们提出一种新的矫正性手形引导损失，确保 3D 手模型的所有视图收敛到视图一致的模态，同时不引入几何畸变。大量评估表明我们的方法优于最优方法，为 3D 手模型生成开辟了新的道路。
英文原文摘要 1616 字符
The emergence of virtual reality has necessitated the generation of detailed and customizable 3D hand models for interaction in the virtual world. However, the current methods for 3D hand model generation are both expensive and cumbersome, offering very little customizability to the users. While recent advancements in zero-shot text-to-3D synthesis have enabled the generation of diverse and customizable 3D models using Score Distillation Sampling (SDS), they do not generalize very well to 3D hand model generation, resulting in unnatural hand structures, view-inconsistencies and loss of details. To address these limitations, we introduce HandDreamer, the first method for zero-shot 3D hand model generation from text prompts. Our findings suggest that view-inconsistencies in SDS is primarily caused due to the ambiguity in the probability landscape described by the text prompt, resulting in similar views converging to different modes of the distribution. This is particularly aggravated for hands due to the large variations in articulations and poses. To alleviate this, we propose to use MANO hand model based initialization and a hand skeleton guided diffusion process to provide a strong prior for the hand structure and to ensure view and pose consistency. Further, we propose a novel corrective hand shape guidance loss to ensure that all the views of the 3D hand model converges to view-consistent modes, without leading to geometric distortions. Extensive evaluations demonstrate the superiority of our method over the state-of-the-art methods, paving a new way forward in 3D hand model generation.