Masked Generative Extractor for Synergistic Representation and 3D Generation of Point Clouds

• Main Authors: Zeng, Hongliang, Zhang, Ping, Li, Fang, Wang, Jiahua, Ye, Tingyu, Guo, Pengteng
• Format: Journal Article
• Language: English
• Published: 25-06-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computer Vision and Pattern Recognition
• DOI: 10.48550/arxiv.2406.17342

Representation and generative learning, as reconstruction-based methods, have
demonstrated their potential for mutual reinforcement across various domains.
In the field of point cloud processing, although existing studies have adopted
training strategies from generative models to enhance representational
capabilities, these methods are limited by their inability to genuinely
generate 3D shapes. To explore the benefits of deeply integrating 3D
representation learning and generative learning, we propose an innovative
framework called \textit{Point-MGE}. Specifically, this framework first
utilizes a vector quantized variational autoencoder to reconstruct a neural
field representation of 3D shapes, thereby learning discrete semantic features
of point patches. Subsequently, we design a sliding masking ratios to smooth
the transition from representation learning to generative learning. Moreover,
our method demonstrates strong generalization capability in learning
high-capacity models, achieving new state-of-the-art performance across
multiple downstream tasks. In shape classification, Point-MGE achieved an
accuracy of 94.2% (+1.0%) on the ModelNet40 dataset and 92.9% (+5.5%) on the
ScanObjectNN dataset. Experimental results also confirmed that Point-MGE can
generate high-quality 3D shapes in both unconditional and conditional settings.