Probability of superiority for comparing two groups of clusters

• Published in: Behavior research methods Vol. 55; no. 2; pp. 646 - 656
• Main Author: Uanhoro, James O.
• Format: Journal Article
• Language: English
• Published: New York Springer US 01-02-2023; Springer Nature B.V
• Subjects: Behavioral Science and Psychology; Cluster Analysis; Cognitive Psychology; Computer Simulation; Educational Status; Humans; Literature reviews; Monte Carlo Method; Monte Carlo simulation; Probability; Psychology; Research Design; Software; Clustered data; Two group comparisons; Probability of superiority; Effect sizes
• ISSN: 1554-3528; 1554-3528
• DOI: 10.3758/s13428-022-01815-6

The probability of superiority (PS) has been recommended as a simple-to-interpret effect size for comparing two independent samples—there are several methods for computing the PS for this particular study design. However, educational and psychological interventions increasingly occur in clustered data contexts; and a review of the literature returned only one method for computing the PS in such contexts. In this paper, we propose a method for estimating the PS in clustered data contexts. Specifically, the proposal addresses study designs that compare two groups and group membership is determined at the cluster level. A cluster may be: (i) a group of cases with each case measured once, or (ii) a single case with each case measured multiple times, resulting in longitudinal data. The proposal relies on nonparametric point estimates of the PS coupled with cluster-robust variance estimation, such that the proposed approach should remain adequate regardless of the distribution of the response data. Using Monte Carlo simulation, we show the approach to be unbiased for continuous and binary outcomes, while maintaining adequate frequentist properties. Moreover, our proposal performs better than the single extant method we found in the literature. The proposal is simple to implement in commonplace statistical software and we provide accompanying R code. Hence, it is our hope that the method we present helps applied researchers better estimate group differences when comparing two groups and group membership is determined at the cluster level.