iSEA: An Interactive Pipeline for Semantic Error Analysis of NLP Models

• Main Authors: Yuan, Jun, Vig, Jesse, Rajani, Nazneen
• Format: Journal Article
• Language: English
• Published: 08-03-2022
• Subjects: Computer Science - Computation and Language; Computer Science - Human-Computer Interaction
• DOI: 10.48550/arxiv.2203.04408

Error analysis in NLP models is essential to successful model development and
deployment. One common approach for diagnosing errors is to identify
subpopulations in the dataset where the model produces the most errors.
However, existing approaches typically define subpopulations based on
pre-defined features, which requires users to form hypotheses of errors in
advance. To complement these approaches, we propose iSEA, an Interactive
Pipeline for Semantic Error Analysis in NLP Models, which automatically
discovers semantically-grounded subpopulations with high error rates in the
context of a human-in-the-loop interactive system. iSEA enables model
developers to learn more about their model errors through discovered
subpopulations, validate the sources of errors through interactive analysis on
the discovered subpopulations, and test hypotheses about model errors by
defining custom subpopulations. The tool supports semantic descriptions of
error-prone subpopulations at the token and concept level, as well as
pre-defined higher-level features. Through use cases and expert interviews, we
demonstrate how iSEA can assist error understanding and analysis.