Quantifying the reliability of image replication studies: the image intra-class correlation coefficient (I2C2)

• Published in: Cognitive, affective, & behavioral neuroscience Vol. 13; no. 4; pp. 714 - 724
• Main Authors: Shou, H., Eloyan, A., Lee, S., Zipunnikov, V., Crainiceanu, A.N., Nebel, M.B., Caffo, B., Lindquist, M.A., Crainiceanu, C.M.
• Format: Journal Article
• Language: English
• Published: 01-12-2013
• ISSN: 1530-7026; 1531-135X
• DOI: 10.3758/s13415-013-0196-0

This manuscript proposes the image intra-class correlation (I2C2) coefficient as a global measure of reliability for imaging studies. The I2C2 generalizes the classic intra-class correlation (ICC) coefficient to the case when the data of interest are images, thereby providing a measure that is both intuitive and convenient. Drawing a connection with classical measurement error models for replication experiments, the I2C2 can be computed quickly, even in high-dimensional imaging studies. A nonparametric bootstrap procedure is introduced to quantify the variability of the I2C2 estimator. Furthermore, a Monte Carlo permutation is utilized to test reproducibility versus a zero I2C2, representing complete lack of reproducibility. Methodologies are applied to three replication studies arising from different brain imaging modalities and settings: Regional Analysis of VolumEs in Normalized Space (RAVENS) imaging for characterizing brain morphology, seed-voxel brain activation maps based on resting state functional MRI (fMRI), and fractional anisotropy (FA) in an area surrounding the corpus callosum via diffusion tensor imaging (DTI). Software and data are provided to ensure rapid dissemination of methods. Resting state functional MRI (fMRI) brain activation maps are found to have low reliability ranging between 0.2 to 0.4.