Spoken language understanding using long short-term memory neural networks

• Published in: 2014 IEEE Spoken Language Technology Workshop (SLT) pp. 189 - 194
• Main Authors: Kaisheng Yao, Baolin Peng, Yu Zhang, Dong Yu, Zweig, Geoffrey, Yangyang Shi
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-12-2014
• Subjects: language understanding; Logic gates; long short-term memory; Recurrent neural networks; Semantics; Speech; Training; Vectors
• DOI: 10.1109/SLT.2014.7078572

Neural network based approaches have recently produced record-setting performances in natural language understanding tasks such as word labeling. In the word labeling task, a tagger is used to assign a label to each word in an input sequence. Specifically, simple recurrent neural networks (RNNs) and convolutional neural networks (CNNs) have shown to significantly outperform the previous state-of-the-art - conditional random fields (CRFs). This paper investigates using long short-term memory (LSTM) neural networks, which contain input, output and forgetting gates and are more advanced than simple RNN, for the word labeling task. To explicitly model output-label dependence, we propose a regression model on top of the LSTM un-normalized scores. We also propose to apply deep LSTM to the task. We investigated the relative importance of each gate in the LSTM by setting other gates to a constant and only learning particular gates. Experiments on the ATIS dataset validated the effectiveness of the proposed models.