Incentives for Sleep and Exercise:
This experiment will estimate enrollment bias for randomized clinical trials offering to incentivize behavior change. In this experiment, we will test whether those who are most motivated to change behavior are also most likely to enroll in a (hypothetical) RCT when offering financial or social incentives for behavior change.
We hypothesize that those most likely to enroll are already motivated to change their behavior prior to enrollment, which could be bias trials towards the null. We will test this hypothesis by estimating if motivation to change a behavior predicts interest in joining RCT targeting that behavior. We will also test if baseline behavior predicts interest in joining these RCT.
We will conduct this experiment using mock invitations to learn about and potentially join a RCT. The study outcome will be responses to this invitation. We will not offer invitations to an actual trial, but the stimuli (mock invitations to a "ghost" trial) and task (response to the invitation) fundamentally resemble a trial's counterparts. The invitations will specify an opportunity to earn financial or social incentives for improving a healthy behavior.
We will invite participants to earn financial or non-financial incentives for increasing their sleep or exercise. In this study, the primary outcome will measure if they were "not interested," "slightly interested," or "very interested" in participating.
Prior to receiving their invitation, they will complete an online questionnaire measuring their motivation to increase each behavior, plus their recent behavior and socio-demographics.
Separate analyses will be conducted for financial and social incentives and for sleep vs exercise trial invitations. We will report point estimates and 95% confidence intervals (CI) for behavior and motivation.
The analyses will examine if their interest in joining the RCT is predicted by 1) their baseline behavior (i.e., amount of sleep or exercise), or 2) their motivation to change the specific behavior. As noted above, we hypothesize the their level of motivation to change a specific behavior will predict interest in a trial targeting that behavior.
Testing Relative Large and Small Vaccination Incentives:
Using a separate sample, this experiment will test whether policies offering large or small financial incentives are likely to strengthen COVID-19 vaccine intention. This experiment will randomize individuals to one of four study arms that include 1) a control condition, 2) an educational message, 3) a message about the relatively large financial incentive, or 4) a message about the relatively small financial incentive. The goal of this study is to estimate if either type of incentive policy is likely to have negative effects on vaccine intention, as some experts have warned.
When analyzing the effects of relatively large and small financial incentives on vaccination intentions, we will report point estimates and 95% CIs for the overall sample and demographic sub-groups. We will also report summary statistics for all the overall sample and sub-populations. We will test whether, compared to a control condition, either of the financial incentives increase, decrease, or have no effect on the percentage who want to vaccinate. In a fourth study arm, subjects will receive an educational message that will also be compared to the control condition.
Testing Vaccine Incentives Plus Different Measures of Vaccine Intention:
In a related experiment, we will separately test the effects of 10 experimental conditions, with a counter-balanced experimental manipulation using an FDA approval message, plus a control condition. The goal of this study is to compare the effects of a wider variety of vaccine interventions that experts have proposed, including incentives and mandates.
In addition, we will also randomize individuals to questionnaires using different methods of measuring vaccine intention, the study outcome.
Comparing different methods of estimating vaccine intention: Embedded within the experiment testing different proposed vaccine policies, we will test if methodological differences in the response option for the primary outcome effect the percent reporting "yes". To do this, we will test 2 (Yes and No) vs 3 (Yes, No and Unsure) level response options and randomly order both sets.
This methodological experiment will examine whether the proportion responding "yes" to the same question (about whether they want to vaccinate soon) varies depending on the order of response options and whether they include a maybe/unsure option. We will run cross tabs and chi-square tests for the 2 vs 3 response levels and the order. The instrumentation tests will be conducted for COVID-19 vaccination boosters, the initial shots, plus vaccination against influenza.
When testing the effects of potential vaccine policies, the control group, with no vaccine policy presented will be compared to: cash incentives for $1000, $200, or $100; a $1,000 tax credit; lotteries for $100,000, $200,000, or $1 million; $1,000 tax on the unvaccinated; and mandates by employers or airlines, bars, and restaurants. The main outcome is whether they would want to get vaccinated soon given the hypothetical vaccine policy.
(Those assigned to the employer mandate condition will be excluded from analyses if they report being unlikely to have an employer.)
OLS specification will be our main result and the other measures are provided as robustness checks.
The OLS model can exclude all the demographic controls and run the binary dependent variable on the treatment variables. (Note that this approach is legitimate because the treatments are being randomized across respondents.) The treatments include the financial policies (incentives and penalties of different amounts and types) and mandates (of different types) being noted in a message.
Type of model: We will perform pairwise t tests of percent of respondents answering "Yes" comparing those treated with an incentive to the control group. We will perform these pairwise tests on subsets by race, gender, income, education, and other socio-demographics.
Additionally, we will conduct these pairwise tests on by type of treatment. Comparing lottery to cash incentive, comparing positive incentive vs. penalty, comparing size of incentive, and comparing employer mandates against the control.
We will also conduct regression analyses on the pooled dataset where the left-hand side observations are individual responses where those answering "Yes" will be coded as 1 and those answering, "No" or "Unsure" will be coded zero. We will include a set of controls (race, gender, income, education, etc) as well as an indicator variable reflecting whether the respondent received a treatment. Regression models will include ordinary least squares, probit, nearest neighbor matching, and propensity score matching. We will also run these regressions where the treatment variable is split up into several indicator variables reflecting the type of treatment provided as well as an indicator for FDA approval.
We will estimate a model-alternatively using ordinary least squares and logistic regression-with a binary-outcome dependent variable (equal to one if the respondent wanted to be vaccinated, and otherwise equal to zero). For explanatory variables, we include dummy variables for each of the ten treatment arms.
Criteria for statistical significance: We will use .05 as our threshold for statistical significance.
Sample size calculation for the survey experiment comparing 10 different vaccine policies: We estimate that if the final sample sizes for each condition include at least 300 subjects, we can detect about 5% or larger difference. We plan to double the allocation for the control and $1000 conditions to allow for planned comparisons.
All experiments: Each subject will be randomized to a condition. Participants will be randomized to reduce the chance that observed effects are due to unmeasured factors. In addition, all study procedures were automated, which improves the control over how the experiment is conducted, allowing all procedures to be consistently standardized.
The studies will enroll national, theory-based samples recruited through MTurk and/or Prolific platforms. To reduce enrollment bias, recruitment and enrollment materials will describe the research in vague terms (e.g., "we are interested in learning your opinions and preferences related to health). Each experiment will also measure socio-demographic variables for descriptive purposes.
Recommended data cleaning procedures for each experiment: Attention checks can identify those who should be excluded from the main analyses. (Regardless of performance on the attention check, all participants will be compensated for their time.) Analyses will exclude those with duplicate IDs or a high fraud score, We will conduct analyses that include and exclude those who finished the fastest (fastest 5%).
Replication studies will include the same study design and procedures.
