Real-time Percussive Technique Recognition and Embedding Learning for the Acoustic Guitar

• Main Authors: Martelloni, Andrea, McPherson, Andrew P, Barthet, Mathieu
• Format: Journal Article
• Language: English
• Published: 13-07-2023
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Learning; Computer Science - Sound
• DOI: 10.48550/arxiv.2307.07426

Real-time music information retrieval (RT-MIR) has much potential to augment
the capabilities of traditional acoustic instruments. We develop RT-MIR
techniques aimed at augmenting percussive fingerstyle, which blends acoustic
guitar playing with guitar body percussion. We formulate several design
objectives for RT-MIR systems for augmented instrument performance: (i) causal
constraint, (ii) perceptually negligible action-to-sound latency, (iii) control
intimacy support, (iv) synthesis control support. We present and evaluate
real-time guitar body percussion recognition and embedding learning techniques
based on convolutional neural networks (CNNs) and CNNs jointly trained with
variational autoencoders (VAEs). We introduce a taxonomy of guitar body
percussion based on hand part and location. We follow a cross-dataset
evaluation approach by collecting three datasets labelled according to the
taxonomy. The embedding quality of the models is assessed using KL-Divergence
across distributions corresponding to different taxonomic classes. Results
indicate that the networks are strong classifiers especially in a simplified
2-class recognition task, and the VAEs yield improved class separation compared
to CNNs as evidenced by increased KL-Divergence across distributions. We argue
that the VAE embedding quality could support control intimacy and rich
interaction when the latent space's parameters are used to control an external
synthesis engine. Further design challenges around generalisation to different
datasets have been identified.