A new Bayesian procedure for the estimation of the referees' decision thresholds and agreement. The case of professional handball refereeing

• Published in: Psychology of sport and exercise Vol. 65; p. 102347
• Main Authors: Bruno, Giovanni, Vicovaro, Michele, Nucci, Massimo, Cropanise, Giovanni, Fabbian, Valter, Mondin, Mauro, Prastaro, Sebastiano, Borellini, Valter, Spoto, Andrea
• Format: Journal Article
• Language: English
• Published: Elsevier Ltd 01-03-2023
• Subjects: Bayesian statistics; Decision threshold; Handball; Inter-rater agreement; Refereeing; Refereeing; Decision threshold; Handball; Bayesian statistics; Inter-rater agreement
• ISSN: 1469-0292; 1878-5476
• DOI: 10.1016/j.psychsport.2022.102347

In double refereeing, agreement between referees is fundamental for the achievement of a flawless and smooth refereeing activity. Nonetheless, the agreeement can be affected by several external and internal factors, with a negative impact on the consistency of the refereeing and the fluidity of the game. The referee’s Decision Threshold (DT) is the limit above which the evidence of a foul results in the call of an infraction by the referee. It represents the individual refereeing style, and it is one of the factors that can contribute to low agreement between the referees. We present an application of a new Bayesian procedure to estimate referee’s DT in a typical refereeing task. To this end, 56 italian professional handball (FIGH) referees were asked to evaluate 96 potential foul plays, belonging to four different infraction types: 7-m throw, passive play, offensive foul, disciplinary sanctions. The proposed method provides information about: (i) referee’s individual DT; (ii) the agreement between pairs of referees (i.e., one-to-one); (iii) the agreement between each referee and all the other referees (i.e., one-to-total); (iv) the agreement between each referee and a reference referee (i.e., one-to-expert); (v) the proportion of agreement between the referees and a reference referee for each potential foul play. Sport federations would profit by this procedure in different ways: by using the information about the DT to train referees, by detecting referees with a low agreement with their colleagues or with the reference referee, by focusing training on specific plays to improve the arbitration class internal consistency.
•Agreement among referees is crucial for the quality of refereeing.•The definition of a foul play depends on a referee-specific decision threshold (DT).•Discrepancies between referees’ DTs may lead to spurious agreements and disagreements.•We present a model quantifying agreement among referees, net of the influence of DTs.•This model improves the evaluation of refereeing consistency.