AudioBERT: Audio Knowledge Augmented Language Model

• Main Authors: Ok, Hyunjong, Yoo, Suho, Lee, Jaeho
• Format: Journal Article
• Language: English
• Published: 12-09-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computation and Language; Computer Science - Sound
• DOI: 10.48550/arxiv.2409.08199

Recent studies have identified that language models, pretrained on text-only
datasets, often lack elementary visual knowledge, \textit{e.g.,} colors of
everyday objects. Motivated by this observation, we ask whether a similar
shortcoming exists in terms of the \textit{auditory} knowledge. To answer this
question, we construct a new dataset called AuditoryBench, which consists of
two novel tasks for evaluating auditory knowledge. Based on our analysis using
the benchmark, we find that language models also suffer from a severe lack of
auditory knowledge. To address this limitation, we propose AudioBERT, a novel
method to augment the auditory knowledge of BERT through a retrieval-based
approach. First, we detect auditory knowledge spans in prompts to query our
retrieval model efficiently. Then, we inject audio knowledge into BERT and
switch on low-rank adaptation for effective adaptation when audio knowledge is
required. Our experiments demonstrate that AudioBERT is quite effective,
achieving superior performance on the AuditoryBench. The dataset and code are
available at \bulurl{https://github.com/HJ-Ok/AudioBERT}.