Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis

•A new platform (DataExplained) helps analysts justify preferred and rejected analytical paths in real time.•Independent analysts used DataExplained to test two hypotheses on the same dataset in a crowdsourced initiative.•Analysts conducted radically different analyses and reported dispersed effect...

Full description

Saved in:

Bibliographic Details
Published in:	Organizational behavior and human decision processes Vol. 165; pp. 228 - 249
Main Authors:	van Assen, Marcel A.L.M., Liu, Yang, Heer, Jeffrey, Mohamed, Zainab, Amireh, Hashem, Venkatesh Prasad, Vaishali, Bernstein, Abraham, Snellman, Kaisa, Madan, Nikhil, Silberzahn, Raphael, Murase, Toshio, Mandl, Benjamin, Kelchtermans, Stijn, Naseeb, Chan, Richard Chan, C.S., Adie, Prestone, Alspaugh, Sara, Alstott, Jeff, Ariño de la Rubia, Eduardo, Arzi, Adbi, Bahník, Štěpán, Baik, Jason, Banker, Sachin, Barros-Rivera, Brenda, Briers, Robert A., Castrillo, Laura, Catlett, Timothy, Chen, Olivia, Cohn, Brent, Cugueró-Escofet, Natàlia, Cyrus-Lai, Wilson, Danielsson, Henrik, Russo, Rosaria de F.S.M., de Silva, Niko, Dondelinger, Frank, Duarte de Souza, Carolina, Tyson Dube, B., Dubova, Marina, Mark Dunn, Ben, Fox, Nick, Gnambs, Timo, Gong, Yuanyuan, Greenawalt, Brandon, Han, Dan, Hong, Antony B., Huang, Lilian, Hui, Kent N., Hultman, Keith A., Kamdar, Jash, Kappler, Gregor, Kaszubowski, Erikson, Kleinberg, Bennett, Lavbič, Dejan, Liverani, Silvia, Loh, Bianca, MacDonald, Kyle, Madan, Christopher R., Hjorth Madsen, Lasse, Maimone, Christina, Marshall, Adrienne, Ester Matskewich, Helena, Mavon, Kimia, McLain, Katherine L., Mertens, Ulf, Moore, Ben, Moore, Andrew, Nantz, Eric, Nasrullah, Ziauddin, Nejkovic, Valentina, Nell, Colleen S, Arthur Nelson, Andrew, Nilsonne, Gustav, Nolan, Rory, O'Brien, Christopher E., O'Shea, Kieran, Palsetia, Diana, Protzko, John, Riddle, Travis, Rosenberg, Joshua M., Schulte-Mecklenbeck, Michael, Sharma, Nirek, Shotwell, Gordon, Stedden, William, Stodden, Victoria, Stoltzman, Scott, Subbaiah, Subashini, Tatman, Rachael, Thibodeau, Paul H., Tomkins, Sabina, Duncan Wadsworth, W., Wanders, Florian, Watts, Krista, Whelpley, Christopher E., Won, Andy, Wu, Lawrence, Yip, Arthur, Youngflesh, Casey, Zhang, Leilei, Zibman, Chava, Luis Uhlmann, Eric
Format:	Journal Article
Language:	English
Published:	Elsevier Inc 2021
Subjects:	Analysis-contingent results Crowdsourcing data analysis Medicin och hälsovetenskap Psychology psykologi Research reliability Researcher degrees of freedom Scientific robustness Scientific transparency Scientific robustness Researcher degrees of freedom Analysis-contingent results Crowdsourcing data analysis Research reliability Scientific transparency
Online Access:	Get full text
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	•A new platform (DataExplained) helps analysts justify preferred and rejected analytical paths in real time.•Independent analysts used DataExplained to test two hypotheses on the same dataset in a crowdsourced initiative.•Analysts conducted radically different analyses and reported dispersed effect sizes, including significant effects in the opposite direction for the same hypothesis.•A BOBA multiverse analysis highlights the importance of variable operationalizations beyond statististical choices. In this crowdsourced initiative, independent analysts used the same dataset to test two hypotheses regarding the effects of scientists’ gender and professional status on verbosity during group meetings. Not only the analytic approach but also the operationalizations of key variables were left unconstrained and up to individual analysts. For instance, analysts could choose to operationalize status as job title, institutional ranking, citation counts, or some combination. To maximize transparency regarding the process by which analytic choices are made, the analysts used a platform we developed called DataExplained to justify both preferred and rejected analytic paths in real time. Analyses lacking sufficient detail, reproducible code, or with statistical errors were excluded, resulting in 29 analyses in the final sample. Researchers reported radically different analyses and dispersed empirical outcomes, in a number of cases obtaining significant effects in opposite directions for the same research question. A Boba multiverse analysis demonstrates that decisions about how to operationalize variables explain variability in outcomes above and beyond statistical choices (e.g., covariates). Subjective researcher decisions play a critical role in driving the reported empirical results, underscoring the need for open data, systematic robustness checks, and transparency regarding both analytic paths taken and not taken. Implications for organizations and leaders, whose decision making relies in part on scientific findings, consulting reports, and internal analyses by data scientists, are discussed.
ISSN:	0749-5978 1095-9920 1095-9920
DOI:	10.1016/j.obhdp.2021.02.003