the efficacy of propensity score-based weights in a web marketing survey that asks questions on hygiene products and attitude towards local banks in a northern European country. They find that estimates from the web survey and the phone surveys (which they use as a comparison) are quite different, and various weighting schemes did not make a difference regarding this. Some studies find that the type of measure matters. Schonlau et al (2004) found among propensity score adjusted web-survey estimates, only 8 of 37 estimates were not significantly different from phone survey estimates. Web survey estimates were significantly more likely to agree with RDD phone survey estimates for found factual questions, and those regarding personal health, and when the questions involved two as opposed to multiple categories of responses. Lee and Valliant (2009) show that weighting adjustment that combines propensity score adjustment and calibration adjustment has the potential to reduce bias for volunteer panel web survey estimates that are contaminated by sample selection bias. However, their results are for model variables used for propensity score adjustment or in the calibration exercise, and not for non-model variables.

Malhotra and Krosnick (2007) compare the 2000 and 2004 American National Election Study (ANES) data collected via using probability sampling with simultaneous Internet surveys of volunteer samples. The non-probabilistic samples were weighted using the raking method using the US CPS so as to match sample proportions with the population proportions on age and education. A comparison of the results yielded many differences in the distributions of variables and in the associations between variables. Applying the weights did almost nothing to reduce the differences between modes/sampling methods in the distributions of the political variables. Chang and Krosnick (2009) simultaneously fielded a probabilistic telephone survey, a probabilistic Internet survey, and a non-probabilistic internet survey. The non-probabilistic Internet survey used a weighting procedure to adjust for variable propensity of individuals to have regular access to email and the Internet. The comparisons showed that th <mark>e probability samples were more representative than the nonprobability sample in terms of demographics and electoral participation, even after re-weighting.</mark> <mark>Brüggen et al. (2016) compare the accuracy of results obtained from 18 opt-in online “panels” participating in the Dutch Online Panel Comparison Study (NOPVO) with data from a probability sample collected from the Internet (the LISS panel; Scherpenzeel 2009), and two probability samples collected via computer assisted personal interviewing, and compared all of them with the Dutch government’s registry of all 16 million residents of the country, the Municipal Basic Administration (MBA). The non-probability sample surveys were weighted using General Regression (GREG) and Horvitz-Thompson Estimators. Results indicate that the nonprobability samples yielded less accurate estimates of proportions than the probability samples, and that weighting does not reduce selection bias in the level estimates. Dutwin and Buskirk (2017) compare RDD surveys and nonprobability Internet panel surveys to a highquality in-person survey. The surveys were weighted using propensity weighting, raking, and sample matching. Their results showed that nonprobability samples attained the greatest estimated bias, and the in-person sample, the lowest. The weighting techniques were not able to improve the measures. Maccinnis et al. (2018) compare data across a variety of probability and nonprobability sampling methods in the United States, using a set of 50 measures of 40 benchmark variables. The probability samples interviewed by telephone or the Internet were the most accurate. Internet surveys of a probability sample combined with an opt-in sample were less accurate; least accurate were internet surveys of opt-in panel samples. These results were not altered by implementing poststratification weights provided by survey companies.</mark><sup>12</sup>

> 12 Cornesse et al. 2020 provides an exhaustive literature review of studies comparing non-probability and probability samples.

8