Seminar Series - Experiment Aversion: A Challenge for learning health systems

Описание: Pragmatic randomized controlled trials—known in the social sciences as field experiments, and in the tech industry as A/B tests—are important tools in a Learning Health System (LHS), or for evidence-based practice in any domain. They can rigorously compare the effectiveness of existing practices (e.g., approved drugs, procedures, healthcare delivery programs) and/or practice innovations, helping to identify wasteful, ineffectual, or even harmful practices, efficiently allocate limited resources, and inform iterative improvements in safety and quality. The alternative to an A/B test is often maintaining or implementing one or more untested standards of care without knowing their effects on patients and other stakeholders. Yet several A/B tests in healthcare and other domains have drawn strong public disapproval.

Vignette experiments (N=10,737) exploring scenarios in medicine, public health, and non-health domains (e.g., autonomous vehicle safety, poverty alleviation, retirement savings) have found that both laypeople and clinicians can be averse to A/B tests, preferring that a decision-maker unilaterally implement option A or B for everyone instead of conducting an A/B test and making the superior intervention the standard of care going forward (Meyer et al., PNAS 2019; Heck et al., PNAS 2020; Vogt et al., BMJ Open 2024).

However, in these previous studies, descriptions of the proposed A/B tests were silent about the presence or absence of consent, front door notice, and ethics oversight—three ethics policies that might reduce or eliminate experiment aversion. This talk will review this previous work, and then present some results from new vignette experiments (N=7,197) which investigate the effect of these policies on experiment aversion. It will also draw on qualitative work to discuss some reasons for experiment aversion and the implications of this work for learning health systems and other aspiring evidence-based organizations.