Different Horses for Different Courses: Comparing Bias Mitigation Algorithms in ML

• Main Authors: Ganesh, Prakhar, Gohar, Usman, Cheng, Lu, Farnadi, Golnoosh
• Format: Journal Article
• Language: English
• Published: 17-11-2024
• Subjects: Computer Science - Artificial Intelligence; Computer Science - Computers and Society; Computer Science - Learning
• DOI: 10.48550/arxiv.2411.11101

With fairness concerns gaining significant attention in Machine Learning
(ML), several bias mitigation techniques have been proposed, often compared
against each other to find the best method. These benchmarking efforts tend to
use a common setup for evaluation under the assumption that providing a uniform
environment ensures a fair comparison. However, bias mitigation techniques are
sensitive to hyperparameter choices, random seeds, feature selection, etc.,
meaning that comparison on just one setting can unfairly favour certain
algorithms. In this work, we show significant variance in fairness achieved by
several algorithms and the influence of the learning pipeline on fairness
scores. We highlight that most bias mitigation techniques can achieve
comparable performance, given the freedom to perform hyperparameter
optimization, suggesting that the choice of the evaluation parameters-rather
than the mitigation technique itself-can sometimes create the perceived
superiority of one method over another. We hope our work encourages future
research on how various choices in the lifecycle of developing an algorithm
impact fairness, and trends that guide the selection of appropriate algorithms.