A Comparison Between Responses From a Propensity-Weighted Web Survey and an Identical RDD Survey

• Published in: Social science computer review Vol. 22; no. 1; pp. 128 - 138
• Main Authors: Schonlau, Matthias, Zapert, Kinga, Simon, Lisa Payne, Sanstad, Katherine Haynes, Marcus, Sue M., Adams, John, Spranca, Mark, Kan, Hongjun, Turner, Rachel, Berry, Sandra H.
• Format: Journal Article
• Language: English
• Published: Thousand Oaks, CA SAGE Publications 01-04-2004; Sage; SAGE PUBLICATIONS, INC
• Subjects: Adjustment; Bias; Categories; Comparative studies; Computer Assisted Research; Digits; Health; Health care; History, theory and methodology; Internet; Methodology; Methodology (Data Collection); Polls & surveys; Propensity; Responses; Scoring; Selfselection; Sociology; Surveys; Telephone Surveys; Telephones; Web sites; Weighting; poststratification; web survey; propensity weights; weighting; phone survey; Harris Interactive; self-selection; Response; Telephone Survey; Survey; Comparison; Bias; Selection; Internet; Sampling
• ISSN: 0894-4393; 1552-8286
• DOI: 10.1177/0894439303256551

The authors conducted a large-scale survey about health care twice, once as a web and once as a random digit dialing (RDD) phone survey. The web survey used a statistical technique, propensity scoring, to adjust for selection bias. Comparing the weighted responses from both surveys, there were no significant response differences in 8 of 37 questions. Web survey responses were significantly more likely to agree with RDD responses when the question asked about the respondent’s personal health (9 times more likely), was a factual question (9 times more likely), and only had two as opposed to multiple response categories (17 times more likely). For three questions, significant differences turned insignificant when adjacent categories of multicategory questions were combined. Factual questions tended to also be questions with two rather than multiple response categories. More study is needed to isolate the effects of these two factors more clearly.