Learning to rank for why-question answering

• Published in: Information retrieval (Boston) Vol. 14; no. 2; pp. 107 - 132
• Main Authors: Verberne, Suzan, van Halteren, Hans, Theijssen, Daphne, Raaijmakers, Stephan, Boves, Lou
• Format: Journal Article
• Language: English
• Published: Dordrecht Springer Netherlands 01-04-2011; Springer; Springer Nature B.V
• Subjects: Artificial intelligence; Computer Science; Data Mining and Knowledge Discovery; Data Structures and Information Theory; Datasets; Exact sciences and technology; Experiments; Genetic algorithms; Information and communication sciences; Information processing and retrieval; Information retrieval; Information retrieval. Man machine relationship; Information science. Documentation; Information Storage and Retrieval; Learning; Machine learning; Natural Language Processing (NLP); Pattern Recognition; Questions; Ranking; Ranking systems; Ratings & rankings; Regression; Regression analysis; Relevance; Research process. Evaluation; Sciences and techniques of general use; Studies; Support vector machines; Tasks; Tuning; questions; Learning to rank; Question answering; Why-questions; Information retrieval; Query processing; Machine learning
• ISSN: 1386-4564; 1573-7659
• DOI: 10.1007/s10791-010-9136-6

In this paper, we evaluate a number of machine learning techniques for the task of ranking answers to
why
-questions. We use TF-IDF together with a set of 36 linguistically motivated features that characterize questions and answers. We experiment with a number of machine learning techniques (among which several classifiers and regression techniques, Ranking SVM and
SVM
map
) in various settings. The purpose of the experiments is to assess how the different machine learning approaches can cope with our highly imbalanced binary relevance data, with and without hyperparameter tuning. We find that with all machine learning techniques, we can obtain an MRR score that is significantly above the TF-IDF baseline of 0.25 and not significantly lower than the best score of 0.35. We provide an in-depth analysis of the effect of data imbalance and hyperparameter tuning, and we relate our findings to previous research on learning to rank for Information Retrieval.