Toward Practical Usage of the Attention Mechanism as a Tool for Interpretability

• Published in: IEEE access Vol. 10; pp. 47011 - 47030
• Main Authors: Tutek, Martin, Snajder, Jan
• Format: Journal Article
• Language: English
• Published: Piscataway IEEE 2022; The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
• Subjects: Artificial intelligence; Decision making; explainable AI; GRU; interpretability; LSTM; Machine translation; Natural language; Natural language processing; Neural networks; Predictive models; recurrent neural network; Recurrent neural networks; Regularization; Regularization methods; Task analysis; Training data
• ISSN: 2169-3536; 2169-3536
• DOI: 10.1109/ACCESS.2022.3169772

Natural language processing (NLP) has been one of the subfields of artificial intelligence much affected by the recent neural revolution. Architectures such as recurrent neural networks (RNNs) and attention-based transformers helped propel the state of the art across various NLP tasks, such as sequence classification, machine translation, and natural language inference. However, if neural models are to be used in high-stakes decision making scenarios, the explainability of their decisions becomes a paramount issue. The attention mechanism has offered some transparency in the workings of otherwise black-box RNN models: attention weights (scalar values assigned input words) invite to be interpreted as the importance of that word, providing a simple method of interpretability. Recent work, however, has questioned the faithfulness of this practice. Subsequent experiments have shown that faithfulness of attention weights may still be achieved by incorporating word-level objectives in the training process of neural networks. In this article, we present a study that extends the techniques for improving faithfulness of attention based on regularization methods that promote retention of word-level information. We perform extensive experiments on a wide array of recurrent neural architectures and analyze to what extent the explanations provided by inspecting attention weights are correlated with the human notion of importance. We find that incorporating tying regularization consistently improves both the faithfulness (−0.14 F1, +0.07 Brier, on average) and plausibility (<inline-formula> <tex-math notation="LaTeX">+53.6\% </tex-math></inline-formula> attention mass on salient tokens) of explanations obtained through inspecting attention weights across analyzed datasets and models.