Combining Neural Fields and Deformation Models for Non-Rigid 3D Motion Reconstruction from Partial Data

Aymen Merrouche, Stefanie Wuhrer, Edmond Boyer.

Arxiv Preprint

Figure: Sample Results.

Figure: Overview of our approach. Given Truncated Signed Distance Field grids representing partial observations of a moving 3D shape (leftmost), the method achieves detailed reconstructions with dense tracking (rightmost). During Feature-Fusion Based Completion (purple module), the TSDFs are encoded in a latent space where self-attention allows to fuse and complete the observed information. The fused latent features are decoded into coarse shapes and then refined where this coarse surface locates. During Inter-Frame Deformation Estimation (green module), the fusion is further constrained by fitting the reconstructions to a patch-wise near-rigid mesh deformation model that implements a near-isometric deformation assumption promoting their consistency.