You Shall Know a Tool by the Traces it Leaves: The Predictability of Sentiment Analysis Tools

• Main Authors: Baumartz, Daniel, Bagci, Mevlüt, Henlein, Alexander, Konca, Maxim, Lücking, Andy, Mehler, Alexander
• Format: Journal Article
• Language: English
• Published: 18-10-2024
• Subjects: Computer Science - Computation and Language
• DOI: 10.48550/arxiv.2410.14626

If sentiment analysis tools were valid classifiers, one would expect them to
provide comparable results for sentiment classification on different kinds of
corpora and for different languages. In line with results of previous studies
we show that sentiment analysis tools disagree on the same dataset. Going
beyond previous studies we show that the sentiment tool used for sentiment
annotation can even be predicted from its outcome, revealing an algorithmic
bias of sentiment analysis. Based on Twitter, Wikipedia and different news
corpora from the English, German and French languages, our classifiers separate
sentiment tools with an averaged F1-score of 0.89 (for the English corpora). We
therefore warn against taking sentiment annotations as face value and argue for
the need of more and systematic NLP evaluation studies.