Hyperbolic Timbre Embedding for Musical Instrument Sound Synthesis Based on Variational Autoencoders

• Published in: 2022 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) pp. 735 - 742
• Main Authors: Nakashima, Futa, Nakamura, Tomohiko, Takamune, Norihiro, Fukayama, Satoru, Saruwatari, Hiroshi
• Format: Conference Proceeding
• Language: English
• Published: Asia-Pacific of Signal and Information Processing Association (APSIPA) 07-11-2022
• Subjects: Aerospace electronics; Gaussian distribution; Information processing; Instruments; Timbre; Training
• ISSN: 2640-0103
• DOI: 10.23919/APSIPAASC55919.2022.9980158

In this paper, we propose a musical instrument sound synthesis (MISS) method based on a variational autoencoder (VAE) that has a hierarchy-inducing latent space for timbre. VAE-based MISS methods embed an input signal into a low-dimensional latent representation that captures the characteristics of the input. Adequately manipulating this representation leads to sound morphing and timbre replacement. Although most VAE-based MISS methods seek a disentangled representation of pitch and timbre, how to capture an underlying structure in timbre remains an open problem. To address this problem, we focus on the fact that musical instruments can be hierarchically classified on the basis of their physical mechanisms. Motivated by this hierarchy, we propose a VAE-based MISS method by introducing a hyperbolic space for timbre. The hyperbolic space can represent treelike data more efficiently than the Euclidean space owing to its exponential growth property. Results of experiments show that the proposed method provides an efficient latent representation of timbre compared with the method using the Euclidean space.