Partisan differences in the effects of economic evidence and local data on legislator engagement with dissemination materials about behavioral health: a dissemination trial
Implementation Science volume 17, Article number: 38 (2022)
State legislators make policy decisions that influence children’s exposure to adverse childhood experiences (ACEs), such as child maltreatment, and their effects on behavioral health. Effective dissemination of scientific research can increase the likelihood that legislators’ decisions are aligned with evidence to prevent ACEs and their consequences, and effective dissemination requires legislators to engage with dissemination materials. Informed by the elaboration likelihood model of persuasive communication and Brownson’s Model of Dissemination Research, we tested the hypothesis that inclusion of economic evidence and local data would increase legislator engagement with dissemination materials about evidence-supported policies related to ACEs and behavioral health.
A three-arm randomized dissemination trial was conducted. A university researcher e-mailed dissemination materials which contained evidence about ACEs and behavioral health problems to state legislators (two e-mails sent 2 weeks apart, 12,662 e-mails delivered to 6509 legislators). The e-mail subject lines, text, and policy brief content were manipulated across the study arms. The intervention condition received state-tailored data about rates of ACEs and state-tailored economic evidence about the costs of ACEs for public systems, the enhanced control condition received state-tailored data and not economic evidence, and the control condition received national data and not economic evidence. Outcomes were rates of e-mail views, policy brief link clicks, requests for researcher consultation, and mentions of child maltreatment terms in legislators’ social media posts.
For the first e-mail, the e-mail view rate was 42.6% higher in the intervention than in the enhanced control condition (22.8% vs. 14.8%) and 20.8% higher than in the control condition (22.8% vs. 18.5%) (both p < .0001). Similar results were observed for the second e-mail. These differences remained significant after adjustment for demographic differences across study conditions in individual-level models, but not multilevel models. There was a significant interaction between the experimental condition and political party (p < .0001) in which the intervention increased e-mail view rates among Democrats but not Republicans. The intervention had no effect on policy brief link clicks or requests for consultation and a mixed effect on social media posts.
Inclusion of state-tailored economic evidence in dissemination materials can increase engagement with research evidence among Democrat, but not Republican, legislators. Dissemination strategies tailored for legislators’ political party affiliation may be needed.
Dissemination research—defined by the US National Institutes of Health as “the scientific study of targeted distribution of information and intervention materials to a specific public health or clinical practice audience” —is underdeveloped in the field of implementation science [2, 3]. Policy-focused research is also an underdeveloped area of implementation science, particularly in the USA [4,5,6]. Thus, at the intersection of these two areas, it is not surprising that little empirical guidance exists to inform decisions about the dissemination of research evidence to policymakers in the USA. For example, a 2020 systematic review of strategies to disseminate research to US policymakers, published in Implementation Science, concluded that very little quantitative research has evaluated the effects of policymaker-targeted dissemination strategies . As a consequence of this knowledge gap, dissemination decisions (e.g., decisions about which data to include in a policy brief or emphasize in e-mails to elected officials and their staff) are typically based on anecdote instead of evidence and have a suboptimal influence on policymaking [2, 8]. While capacity- and relationship-building strategies have demonstrated the ability to improve evidence-informed policymaking [9, 10], there is also a need to strengthen the empirical foundation for dissemination strategies that “push” evidence to policymakers because these strategies are far less resource intensive and are regularly used by researchers, advocates, and other intermediary organizations.
Building on descriptive research with state legislators about mental health and substance use (i.e., behavioral health) issues [11,12,13,14,15,16,17,18], the current study experimentally tests the effects of including state-tailored economic evidence and data in behavioral health dissemination materials on legislators’ engagement with dissemination materials. We also examine whether political party affiliation moderates the effects of dissemination materials and explore associations between political party affiliation and legislator engagement with dissemination materials. The study’s results provide concrete guidance to inform dissemination practice and the study’s design offers a model for future dissemination experiments with policymakers. The study also advances the field of implementation science by assessing the role of political identity in moderating dissemination outcomes, which has received little attention in prior work.
Study context: evidence about adverse childhood experiences, behavioral health, and evidence-supported policy strategies
The current study focuses on disseminating evidence about adverse childhood experiences (ACEs) as risk factors for behavioral health problems and evidence-supported policy strategies to address ACEs. ACEs include incidents of child maltreatment—such as physical and sexual abuse and neglect—as well as other adverse experiences before age 18, such as witnessing domestic and community violence [19,20,21,22]. ACEs are well-established risk factors for behavioral health problems [22,23,24]. ACE exposure is common in the USA, with an estimated 16% of US adults having experienced ≥4 ACEs and 38% having experienced ≥2 ACEs .
State legislators (there are 7383 in the USA) are an important audience to target with the dissemination of evidence about ACEs because they make policy decisions that can reduce exposure to ACEs and mitigate their behavioral health consequences [26,27,28]. Reviews have identified state-level policies that have demonstrated effectiveness in achieving these outcomes [29,30,31]. These include policies that increase economic security and reduce caregiver stress [32,33,34] and policies that increase the reach of evidence-based public programs, such as nurse family partnerships [35, 36]. Organizations such as the American Academy of Pediatrics  and Harvard Center on the Developing Child  have emphasized the importance of disseminating evidence about ACEs to state legislators.
Despite the potential of state legislators to address ACEs, the legislative response to ACEs and legislators’ knowledge about ACEs are suboptimal. A recent review found that only 20.5% of bills introduced in state legislatures that mentioned ACEs also identified evidence-based or evidence-supported interventions related to ACEs . This analysis also found that Democrats were significantly more likely than Republicans to introduce, co-sponsor, and vote in favor of legislation to address ACEs. A 2017 survey of 475 state legislators found that only one-third had heard of ACEs and that most were not knowledgeable about the extent to which ACEs were risk factors for behavioral health problems . The survey also found that Republican legislators had significantly lower levels of knowledge about ACEs than Democrats. A nationally representative 2020 public opinion survey of US adults found similar results related to political party affiliation . Public opinion surveys and experiments have also found, however, that evidence about the behavioral health consequences of ACEs and their economic impacts for public systems  could increase support for evidence-supported policies to address ACEs among both Republicans and Democrats [41, 42]. Taken together, this body of research underscores the importance of effectively disseminating evidence about ACEs to state legislators and provides some indication of messaging strategies that could be successfully used in practice.
Prior dissemination research with state legislators and knowledge gap
Prior research conducted with state legislators provides additional guidance about how to develop dissemination materials about ACEs for state legislators. The aforementioned 2017 survey of legislators found that the local relevance of data and the inclusion of economic evidence were perceived as extremely important attributes of behavioral health dissemination materials, especially among Republicans [11, 13, 15]. The perceived importance of economic evidence and local data is consistent with a larger body of research about attributes of dissemination materials that are important to policymakers [7, 43,44,45]. Experiments with state and local legislators in the USA have also tested the effects of manipulating aspects of dissemination materials (e.g., framing of issues in e-mail subject lines [46,47,48], relational context [49, 50], inclusion of maps and narratives [51, 52]).
While prior work sheds light on how to format dissemination materials, it sheds little light on what types of evidence to include. This knowledge gap is important because only a finite amount of evidence can be included in concise dissemination materials, and research demonstrates that shorter presentations of evidence are preferred by legislators [2, 11]. Very little research has also explored the moderating effect of policymakers’ political party affiliation on the effects of dissemination materials . This knowledge gap warrants attention given increasing political polarization among both state policymakers and the public in the USA [54, 55], the impact this polarization has on policymaking, and results from communication experiments with the general public which often find that political party identification moderates responses to different types of health messages [56, 57].
Study aims and hypotheses
The primary aims of the study were to test the pre-registered hypotheses that:
Inclusion of state-tailored economic evidence about ACEs and state-tailored data about rates of ACEs in dissemination materials (intervention) increases legislator engagement with dissemination materials compared with inclusion of national data about rates of ACEs and no economic evidence (control)
Inclusion of state-tailored economic evidence about ACEs and state-tailored data about rates of ACEs in dissemination materials (intervention) increases engagement with dissemination materials compared with inclusion of state-tailored data about rates of ACEs and no economic evidence (enhanced control)
Inclusion of state-tailored data about rates of ACEs and no economic evidence (enhanced control) increases engagement with dissemination materials compared with inclusion of national data about rates of ACEs and no economic evidence (control)
The secondary, exploratory aims were to determine whether the effects of dissemination materials were moderated by legislators’ political party affiliation and to assess the extent to which engagement with dissemination materials is independently associated with political party affiliation.
We focus on measures of engagement with dissemination materials as our outcome for three reasons. First, engagement is considered an important dissemination outcome [2, 58]. Measures of engagement with dissemination materials have been used in dissemination research conducted with policymakers [46,47,48] as well as clinicians  and are often treated as primary outcomes in experimental marketing research—a field that has similar goals as dissemination research [2, 60, 61]. Second, engagement can be assessed unobtrusively (i.e., without surveys or other research interaction) and this is important because it is increasingly challenging to obtain sufficiently high survey response rates from state legislators. Third, engagement with dissemination materials is a prerequisite for the materials to affect other dissemination outcomes such as knowledge, attitudes, and behavior change.
The elaboration likelihood model (ELM) of persuasive communication provides the theoretical basis for our hypotheses [62,63,64]. ELM has been identified as a theory that has utility for dissemination research  and has been previously used in dissemination research with state legislators  and other practice audiences [65, 66]. ELM posits that the extent to which a message is perceived as relevant by audiences influences the extent to which audiences engage with and cognitively process the message. Based on prior research with state legislators [11, 13, 15], we believe that the inclusion of state-tailored economic evidence and data will increase the perceived relevance of dissemination materials and subsequent engagement with the materials. We further expand this model to consider the audience’s predisposing characteristics. Theories of motivated reasoning suggest that message acceptance and engagement is also dependent upon the extent to which the message resonates with an audience’s predisposing attitudes (i.e., attitudes about the relevance of or trust in scientific evidence) . Given evidence of widening differences by partisanship in confidence in science, we examine legislator partisanship as a moderator .
The overarching structure of our study is also informed by Brownson and colleagues’ Model of Dissemination Research , which was used to guide the aforementioned systematic review of dissemination strategies targeting US policymakers . To apply the Model to our study, the message source is a university researcher, the message is evidence about ACEs and behavioral health (which is experimentally manipulated), the channel used to deliver the message is e-mail, and the audience is US state legislators and their staff.
Study design, population, and randomization
We conducted a pre-registered, clustered randomized dissemination field experiment (Center for Open Science pre-registration: https://osf.io/cgh64; Supplemental file 1 CONSORT and TIDieR checklists). The study population was 6964 US state legislators (94% of those in the USA) who had an e-mail address in the KnowWho database as of January 25, 2021. KnowWho is a service that compiles and maintains up-to-date contact information and biographical data of government officials. We limit our sample and analyses to legislators for whom e-mail addresses were available because dissemination materials were delivered exclusively via e-mail. We refer to “legislators” throughout the article because they were our target audience, but it should be noted that we technically collected outcomes data from legislative offices because we do not know whether legislators or their staff engaged with dissemination materials.
Clustered, stratified randomization was used in which all fifty US states were randomized to one of the three conditions (detailed below). Randomization was at the state level to avoid contamination, wherein a legislator assigned to one condition would share dissemination materials with a legislator assigned to a different condition in the same state. Randomization was stratified to approximate balance in the number of legislators in each condition and the percentage of legislators in each condition that were Republican. We stratified on these two variables because the number of legislators in each state varies dramatically (i.e., range = 49 to 393) and prior research has shown that legislators’ opinions about ACEs vary significantly by political party .
To carry out stratified randomization, we classified each state according to whether the size of its legislature was less than the national median of 139 and whether the percentage of the legislature that was Republican was less than the national mean of 53.3%. This resulted in four strata of states defined by the cross-classification of legislature size and percentage of Republican legislators. We then randomized the states within each stratum for a balanced design, assigning all legislators within each state to the same study condition .
Study conditions and dissemination materials
Legislators in the intervention condition were sent state-tailored data about rates of ACEs and state-tailored economic evidence about the costs of ACEs for public systems’ legislators in the enhanced control condition were sent state-tailored data about rates of ACEs and no economic evidence, and legislators in the control condition were sent national data about rates of ACEs and no economic evidence. Three elements of dissemination materials were manipulated in each condition: e-mail subject line, e-mail body test, and policy brief content.
The syntaxes of the e-mail subject lines were identical with the exception of elements that were modified to emphasize core aspects of each study condition. We used the language “Child Maltreatment,” as opposed to “Adverse Childhood Experiences,” in e-mail subject lines and language of “child maltreatment and adverse childhood experiences” in the e-mail body text because we assumed that more legislators would be familiar with the term “child maltreatment” than ACEs . The subject lines for the three conditions are shown in Supplemental file 3.
The body text of the e-mails was modified similarly to summarize evidence contained in the policy brief (see Supplemental file 3 for e-mail text for all study conditions). Consistent with recommended practices to enhance e-mail engagement [61, 70], all e-mails were personalized to include the name and title of the legislator. Across all study conditions, the e-mails concluded with an invitation for consultation from the project principal investigator (PI) about evidence-supported policy approaches to addressing ACEs. This invitation was intended to provide relational context, which has been shown experimentally to increase policymaker engagement with research evidence [49, 50]. Furthermore, building relationships between researchers and policymakers is frequently identified as one of the most effective ways to facilitate more evidence use by policymakers [9, 44, 71, 72].
Policy briefs were accessible via a link in the e-mails that directed to a webpage where the policy brief was displayed in PDF format (see Supplemental file 4 for example policy briefs). Based on prior research demonstrating that state legislators have a strong preference for research evidence being concise , policy briefs were a single PDF page. Policy briefs across all three study conditions contained six identical elements:
Descriptive information about types of experiences that are considered ACEs 
A statement about the potential of positive childhood experience to prevent the consequences of ACEs 
A list of evidence-supported state policies to address ACEs from the National Conference of State Legislators [29, 30], a professional association which is largely perceived as a credible source of research evidence among Democrat and Republican legislators 
References for evidence cited
The policy briefs also included visual cues that corresponded with the study conditions . The intervention condition policy brief included an image of a bank bag with a “$” to emphasize that economic evidence was included. The intervention and enhanced control condition policy briefs included an image of the legislator’s state to signal that state-tailored information was included. The control condition policy brief included an image of the USA.
The intervention and enhanced control policy briefs included state-tailored data about the prevalence of children in the state with ≥ 1 ACE and ≥ 2 ACEs, using data from the 2018 National Survey of Children’s Health, as well as data about the past-year incidence of reported cases of child maltreatment of any type, physical abuse, sexual abuse, and neglect—using data from the 2017 US National Child Abuse and Neglect Data System . These types of child maltreatment are all considered ACEs [19,20,21,22]. In the control condition, the same data were presented at the US national level.
The intervention policy briefs included state-tailored economic evidence about the costs of some ACEs for state public systems. These estimates were generated by multiplying the number of reported incidents of non-fatal child maltreatment in each state in 2017 by an estimate of the lifetime cost of a single case of non-fatal child maltreatment—reported in a cost study by Peterson et al.  We calculated and included state-specific cost estimates from child welfare, special education, and criminal justice system perspectives (adjusted to be expressed in 2019 dollars) and presented this evidence in the policy briefs. We focused on costs from public system perspectives because prior research indicates that state legislators have a strong preference for economic evidence from this perspective .
Two e-mails were sent from the project PI’s university e-mail account using Qualtrics over the course of 3 weeks (first e-mail on March 22, 2021, second e-mail on April 5, 2021). Legislators for whom e-mails bounced back as undeliverable in the first e-mail (n = 492) were removed from the second e-mail.
Outcomes and measures
The primary outcomes were legislator e-mail and policy briefs views. These were the primary outcomes because they were highly proximal indicators of engagement with dissemination materials. These outcomes were assessed using standard web-based marketing practices used in prior research [46,47,48, 59]. E-mail views were assessed by a 1-pixel translucent image that was embedded at the top of each e-mail. When an e-mail was viewed, the pixel was automatically downloaded, and the download was tracked in a de-identified database maintained by SSRS (a survey research firm). E-mail views were operationalized as a separate dichotomous outcome (yes/no) for each of the two e-mails. Policy brief views were assessed by monitoring link clicks to view the policy briefs. When a link was clicked, the legislator was automatically logged in to view the policy brief, and logins were tracked in the de-identified database maintained by SSRS. Policy brief views were also operationalized as a dichotomous outcome (yes/no) for each e-mail. Every legislator had a unique pixel ID and policy brief link for each e-mail, which allowed for these outcomes to be tracked at the individual level for the two separate e-mails.
Secondary outcomes were legislator requests for consultation about ACEs and mentions of topics related to ACEs in social media posts and newsletters to constituents. These were the secondary outcomes because they were more distal indicators of engagement with dissemination materials. E-mail replies and phone calls requesting consultation were tracked by the project PI and operationalized as a dichotomous outcome (yes/no) for each of the two e-mails. Mentions of topics related to ACEs in legislators’ Facebook and Twitter posts and newsletters to constituents were assessed using Quorum, a database that compiles this information in real time and has been previously used to characterize legislators’ public communication about health issues [81,82,83]. We searched Quorum for mentions of words from two sets of key terms in the 2-month period after when the first dissemination e-mail was sent (between March 22, 2021, and May 22, 2021). One set of terms was broadly related to the child maltreatment and the dissemination material content (i.e., “child abuse” and/or “physical abuse” and/or “child sexual abuse” and/or “child neglect” and/or “child maltreatment”). The other set of terms was explicitly related to ACEs (i.e., “adverse childhood experiences” and/or “ACEs”), the terms used in the aforementioned review of ACEs legislation .
Information about the demographic characteristics of legislators was obtained from the KnowWho database and served as covariates. Political party affiliation was operationalized as Republican or non-Republican (with non-Republicans hereafter referred to as Democrats because registered Democrats comprise over 98% of legislators in this category and consistent with how political party has been operationalized in prior work [15, 51]), gender was operationalized as male or female because this is a binary variable in the KnowWho database, and race/ethnicity was operationalized as non-Hispanic White (yes/no) to maximize statistical power. We also used KnowWho data to characterize each legislator according to whether they had a professional background in healthcare (yes/no).
Analysis was conducted at the individual legislator level because the e-mail dissemination intervention and outcome assessment were at this level. Legislators for whom e-mails were undeliverable (n = 492 in the first e-mail analysis, n = 356 in the second e-mail analysis) were excluded from the analysis. Data from the first and second dissemination e-mails were analyzed separately. Chi-square tests were used in randomization checks to assess differences in demographic characteristics of legislators across the three study conditions in each dissemination e-mail.
We use results from both unadjusted and adjusted analyses to determine the effects of dissemination materials because there is a lack of consensus about whether it is appropriate to adjust for demographic differences across conditions in randomized trials [84, 85]. We used chi-square tests to assess effects in unadjusted analyses and multivariable logistic regression models that adjusted for differences in demographic characteristics statistically significant at a threshold of p < .05. Legislators with missing data on covariates were excluded from regression analyses (n = 237 for the first e-mail, n = 213 for the second e-mail). We ran single as well as multilevel (legislator, state) random-intercept models to account for correlated outcomes between legislators in the same state (i.e., clustering) . The intraclass correlation coefficient (ICC) ranged from 0.22 to 0.35 across the six main effect models (mean ICC across the models = 0.27), indicating that the data were suitable for multilevel analysis . Our primary analytic approach was unadjusted because the sample size of our experiment was constrained by the number of state legislators in the USA and thus underpowered for multivariable models. We assessed whether the effects of dissemination materials were moderated by political party affiliation by stratifying results by political party within each study condition and by assessing the significance of a political party*study condition interaction term in regression models.
The first dissemination e-mail was successfully delivered to 6509 legislators and the second e-mail was delivered to 6153 legislators. CONSORT diagrams are included as Supplemental file 2. Table 1 shows the demographic characteristics of legislators to whom dissemination e-mails were successfully delivered, stratified across the three study conditions and two separate e-mails. There were small but statistically significant differences across the conditions in terms of political party affiliation, gender, and race/ethnicity.
Effects of dissemination materials on e-mail views
E-mail views were registered for 1220 (18.7%) legislators in the first dissemination e-mail and 1089 (17.7%) in the second e-mail. In the first e-mail, the view rate was 42.6% higher in the intervention than enhanced control condition (22.8% vs. 14.9%) and 20.8% higher in the intervention than control condition (22.8% vs. 18.5%) (both p < .0001, Table 2). Similar effects were observed in the second dissemination e-mail, in which the view rate was 25.9% higher in the intervention than enhanced control condition (20.9% vs. 16.1%) and 31.6% higher in the intervention than control condition (20.9% vs. 15.2%) (both p<.0001).
The effect of the intervention treatment on e-mail views generally remained significant after adjustment for demographics in individual-level logistic regression models (Table 3). For example, compared to legislators in the control condition, the adjusted odds of a legislator in the intervention condition viewing the e-mail were 24% higher [adjusted odds ratio (AOR)=1.24; p= .008] in the first e-mail and 43% higher in the second e-mail (AOR= 1.43; p <.0001). The point estimates of the intervention effect were nearly identical in multilevel models that accounted for state clustering, but confidence intervals were wider and results were not statistically significant. The enhanced control treatment did not have a statistically significant effect on e-mail views compared to the control in unadjusted or adjusted analyses.
The effects of the intervention treatment on e-mail views were moderated by political party affiliation. The political party*study condition interaction term was statistically significant in the individual-level model (intervention vs. control: p=. 002 and enhanced control vs. control: p= .0008). Among Democrats in the first e-mail, the e-mail view rate was 38.2% higher in the intervention than control condition (28.4% vs. 19.3%, p < .001), while the view rate was nearly identical among Republicans across these two conditions (17.4% vs. 17.8%) (Fig. 1). Similar results were observed in the second e-mail. Counter to our hypothesis, the e-mail view rate was significantly (p < .001) lower in the enhanced control than control group among Republicans in the first e-mail (11.3% vs. 17.8%) as well as the second e-mail (13.2% vs. 15.1%). This was not observed among Democrats.
Intervention effects on policy brief link clicks
Policy brief link clicks were registered for 208 (3.2%) legislators in the first dissemination e-mail and 142 (2.3%) in the second e-mail. There were no significant differences in link click rates across the study conditions in unadjusted or adjusted analyses. There was no significant interaction between political party affiliation and study condition for link clicks.
Intervention effects on requests for consultation
Following the first dissemination e-mail, 16 (0.3%) legislators requested consultation and 22 (0.4%) requested consultation following the second e-mail. There were no significant differences in the frequency of requests for consultation across the study conditions in unadjusted or adjusted analyses. There was also no significant interaction between political party affiliation and study condition for the consultation outcome.
Intervention effects on mentions of terms related to ACEs in social media posts and newsletters to constituents
In the 2-month period following the date when the first dissemination e-mail was sent, 238 (3.9%) of legislators posted or sent ≥1 messages that mentioned any terms related to child maltreatment broadly or ACEs explicitly. The intervention condition did not impact the odds of a legislator mentioning the terms when mentions of the two terms were analyzed together (Table 4). Supplemental file 5 shows results from the separate analysis of mentions of terms broadly related to child maltreatment and terms explicitly related to ACEs. In these analyses, the intervention treatment did have the effect of significantly increasing the odds of a legislator mentioning a term broadly related to child maltreatment compared to the enhanced control group (AOR= 1.44, p = .04). However, the intervention treatment significantly reduced the odds of a legislator explicitly mentioning ACEs compared to the enhanced control (AOR= 0.10, p = .002) and control (AOR = 0.14, p = .0002) conditions.
Adjusted associations between political party affiliation and engagement with dissemination materials
After adjustment for study experimental condition, gender, and race/ethnicity, and state-level clustering, Democrats remained significantly more likely than Republicans to view the dissemination e-mails, view the policy briefs, and request consultation. For example, in the multilevel model for the first dissemination e-mail, Democrats had 49% higher odds of viewing the e-mail (AOR= 1.49 p < .0001), two times higher odds of clicking the policy brief link (AOR= 2.13, p < .0001), and over eight times higher odds of requesting consultation (AOR = 8.59, p= .007) than did Republicans. Similar results were observed in the second e-mail.
We experimentally tested the effects of including state-tailored economic evidence and state-tailored incidence/prevalence data on state legislator engagement with dissemination materials and explored associations between political party affiliation and engagement with dissemination materials. We found that the inclusion of economic evidence significantly increases engagement with the dissemination e-mails, but that this effect is entirely driven by Democrats. We did not find that the inclusion of economic evidence increased rates of clicking a link to view a policy brief or requesting expert consultation related to the dissemination materials. We also did not find that inclusion of state-tailored incidence/prevalence data increased engagement with dissemination materials compared to inclusion of national data. After adjustment for study condition and demographic covariates, we found that Democrats were significantly more likely to engage with dissemination materials than Republicans.
The significant effect of the intervention on e-mail view rates can be attributed to the mention of “economic impact” in the e-mail subject lines because viewing a subject line precedes viewing the body text of an e-mail. This finding is consistent with prior experiments which have found the e-mail subject lines influence e-mail view rates [46,47,48, 60, 61]. However, the magnitudes of the effects of subject lines on e-mail views in our experiment (e.g., 42.6% higher in the intervention than enhanced control group in the first e-mail) are larger than those observed in prior work. For example, a series of experiments with state legislators in 2020 which tested disparity vs. non-disparity frames in e-mail subject lines found effects on e-mail views that were no larger than 21% between study conditions . A 2018 marketing experiment found that adding an e-mail recipient’s first name to the subject line increased the probability of viewing the e-mail by 20% . According to the elaboration likelihood model of persuasive communication, the larger magnitudes of effects in our study may reflect economic evidence being perceived as highly relevant to state legislations, which is consistent with prior descriptive research with state legislators . We did not find, however, that higher e-mail view rates in the intervention conditions translated into higher policy brief link click rates. These findings have at least two important implications for dissemination practice. First, results confirm that e-mail subject lines are an important element of dissemination materials and should be developed based on theory. Second, results highlight the importance of including the most important messages about evidence in the body text of dissemination e-mails because the majority of recipients who view an e-mail will not click a link to engage with additional evidence.
The finding that inclusion of state-tailored incidence/prevalence data did not increase engagement with dissemination materials compared to inclusion of national data is inconsistent with our hypothesis and some prior dissemination experiments with policymakers in the USA and the UK [88, 89]. Among Republicans, counter to our hypothesis, e-mail view rates were actually significantly lower among legislators who were sent state as opposed to national prevalence data. This finding warrants future investigation.
We found the inclusion of economic evidence had no effect on Republican legislators’ engagement with dissemination materials. This finding is inconsistent with our theoretically informed hypothesis and prior research which found that Republican legislators placed relatively more value on economic evidence related to behavioral health issues than Democrats [13, 15]. A recent survey-based experiment also observed results among Republican legislators that were counter to theory and prior research . The experiment found that inclusion of narratives (i.e., fictional stories about people affected by the issue) decreased Republican legislators’ support for evidence-based childcare policies, while the narratives increased support among Democrats. In addition to the null effect of the intervention condition on Republican engagement with dissemination materials in our study, we found that Republican political party affiliation was significantly and inversely associated with all four of our engagement outcomes. Taken together, these findings signal a need for future research that tests the effects of tailoring messages in dissemination materials for Republican legislators and also potentially the sources from which they are delivered.
Study key strengths and limitations
Key strengths of our study include the cluster randomized design (which enhances internal validity), assessment of outcomes via unobtrusive measures for the entire study population as opposed to surveys with a potentially non-representative sample of the population (which enhances external validity), assessment of multiple measures of engagement with dissemination materials ranging from the very proximal (i.e., e-mail views, policy brief link clicks) to more distal (i.e., requests for consultation, posts on social media), assessment outcomes at the individual level (e.g., tracking individual policy brief link clicks as opposed to aggregate policy brief website traffic), and the multilevel analysis approach.
A limitation of our study relates to the imprecise measurement of e-mail views and policy brief link clicks. Although these outcomes were assessed using marketing industry best practices that are standard in dissemination research, they are inherently imprecise because some e-mail servers and account settings block the automatic pixel downloads and link click logins used to register engagement outcomes. However, while this error may have affected study power, it should theoretically have been nondifferential with respect to study condition because of the study’s randomized design . Our study was underpowered to detect significant effects in multilevel models. A post hoc power analysis revealed that, with 80% power for our primary outcomes, the minimally detectable odds would be 6.42 for the multilevel models. This cannot be addressed, however, because the number of state legislators and US states is fixed and nearly all (94%) were included in our study.
It should be emphasized that the e-mails were sent from a university researcher. While both Republican and Democrat legislators have identified university researchers as a credible source of research evidence in prior work , other studies have observed mistrust in researchers among legislators . Our findings may have differed if dissemination materials were sent from another source. It should also be noted that there is no consensus, or even discussion, in the field in terms of what constitutes a “minimally important difference” in outcomes of engagement with dissemination materials. This is an important issue for building the field of dissemination research. Finally, our results are not necessarily generalizable beyond the US context. While some findings—such as the importance of e-mail subject lines—likely extend to different national contexts, the USA is unique in terms of its two-party political system and large number of states that have wide variation in staffing structure. E-mail is also a primary means of disseminating evidence to policymakers in the USA and this may not be the case in many low- and middle-income countries, limiting the generalizability of results to these contexts.
Inclusion of state-tailored economic evidence in dissemination materials can increase engagement with research evidence among Democrat, but not Republican, legislators. Inclusion of state, as opposed to national, incidence/prevalence data does not have this effect, and national data might actually be more effective at prompting engagement with dissemination materials among Republican legislators. After adjustment for covariates, Republican legislators engaged with evidence disseminated from a university researcher at significantly lower rates than did Democrats. Taken together, the study findings support the inclusion of economic evidence in legislator-targeted dissemination materials but also highlight a need for future research that tests the effects of tailoring dissemination materials for legislators with different political party affiliations.
Availability of data and materials
Data are however available from the authors upon reasonable request.
National Institutes of Health. PAR-19-274. https://grants.nih.gov/grants/guide/pa-files/PAR-19-274.html, Accessed 1 Nov 2021.
Purtle J, Marzalik JS, Halfond RW, Bufka LF, Teachman BA, Aarons GA. Toward the data-driven dissemination of findings from psychological science. Am Psychol. 2020;75(8):1052.
Brownson RC, Eyler AA, Harris JK, Moore JB, Tabak RG. Getting the word out. J Public Health Manag Pract. 2018;24(2):102–11.
Purtle J, Peters R, Brownson RC. A review of policy dissemination and implementation research funded by the National Institutes of Health, 2007–2014. Implement Sci. 2015;11(1):1–8.
Hoagwood KE, Purtle J, Spandorfer J, Peth-Pierce R, Horwitz SM. Aligning dissemination and implementation science with health policies to improve children’s mental health. Am Psychol. 2020;75(8):1130.
Emmons KM, Chambers DA. Policy implementation science–an unexplored strategy to address social determinants of health. Ethn Dis. 2021;31(1):133.
Ashcraft LE, Quinn DA, Brownson RC. Strategies for effective dissemination of research to United States policymakers: a systematic review. Implement Sci. 2020;15(1):1–17.
Petkovic J, Welch V, Jacob MH, et al. The effectiveness of evidence summaries on health policymakers and health system managers use of evidence from systematic reviews: a systematic review. Implement Sci. 2016;11(1):1–14.
Crowley DM, Scott JT, Long EC, et al. Lawmakers’ use of scientific evidence can be improved. Proc Natl Acad Sci. 2021;118(9):e2012955118.
Williamson A, Barker D, Green S, et al. Increasing the capacity of policy agencies to use research findings: a stepped-wedge trial. Health Res Policy Syst. 2019;17(1):1–16.
Purtle J, Nelson KL, Bruns EJ, Hoagwood KE. Dissemination strategies to accelerate the policy impact of children’s mental health services research. Psychiatr Serv. 2020;71(11):1170–8.
Purtle J, Dodson EA, Brownson RC. Uses of research evidence by State legislators who prioritize behavioral health issues. Psychiatr Serv. 2016;67(12):1355–61.
Purtle J, Dodson EA, Nelson K, Meisel ZF, Brownson RC. Legislators’ sources of behavioral health research and preferences for dissemination: variations by political party. Psychiatr Serv. 2018;69(10):1105–8. https://doi.org/10.1176/appi.ps.201800153.
Purtle J, Lê-Scherban F, Wang X, Brown E, Chilton M. State legislators’ opinions about adverse childhood experiences as risk factors for adult behavioral health conditions. Psychiatr Serv. 2019;70(10):894–900.
Purtle J, Lê-Scherban F, Wang X, Shattuck PT, Proctor EK, Brownson RC. Audience segmentation to disseminate behavioral health evidence to legislators: an empirical clustering analysis. Implement Sci. 2018;13(1):121.
Purtle J, Lê-Scherban F, Wang X, Shattuck PT, Proctor EK, Brownson RC. State legislators’ support for behavioral health parity laws: the influence of mutable and fixed factors at multiple levels. Milbank Q. 2019;97(4):1200–32.
Meisel ZF, Mitchell J, Polsky D, et al. Strengthening partnerships between substance use researchers and policy makers to take advantage of a window of opportunity. Subst Abuse Treat Prev Policy. 2019;14(1):12.
Srivastav A, Spencer M, Thrasher JF, Strompolis M, Crouch E, Davis RE. Addressing health and well-being through state policy: understanding barriers and opportunities for policy-making to prevent adverse childhood experiences (ACEs) in South Carolina. Am J Health Promot. 2020;34(2):189–97.
Ellis WR, Dietz WH. A new framework for addressing adverse childhood and community experiences: the building community resilience model. Acad Pediatr. 2017;17(7):S86–93.
Wade R, Shea JA, Rubin D, Wood J. Adverse childhood experiences of low-income urban youth. Pediatrics. 2014;134(1):e13–20.
McEwen CA, Gregerson SF. A critical assessment of the adverse childhood experiences study at 20 years. Am J Prev Med. 2019;56(6):790–4.
Felitti VJ, Anda RF, Nordenberg D, et al. Relationship of childhood abuse and household dysfunction to many of the leading causes of death in adults: the Adverse Childhood Experiences (ACE) Study. Am J Prev Med. 1998;14(4):245–58.
Hughes K, Bellis MA, Hardcastle KA, et al. The effect of multiple adverse childhood experiences on health: a systematic review and meta-analysis. Lancet Public Health. 2017;2(8):e356–66.
Afifi TO, Enns MW, Cox BJ, Asmundson GJ, Stein MB, Sareen J. Population attributable fractions of psychiatric disorders and suicide ideation and attempts associated with adverse childhood experiences. Am J Public Health. 2008;98(5):946–52.
Merrick MT, Ford DC, Ports KA, Guinn AS. Prevalence of adverse childhood experiences from the 2011-2014 Behavioral Risk Factor Surveillance System in 23 states. JAMA Pediatr. 2018;172(11):1038–44.
Shonkoff JP, Bales SN. Science does not speak for itself: translating child development research for the public and its policymakers. Child Dev. 2011;82(1):17–32.
Bethell CD, Solloway MR, Guinosso S, et al. Prioritizing possibilities for child and family health: an agenda to address adverse childhood experiences and foster the social and emotional roots of well-being in pediatrics. Acad Pediatr. 2017;17(7):S36–50.
Shern DL, Blanch AK, Steverman SM. Toxic stress, behavioral health, and the next major era in public health. Am J Orthop. 2016;86(2):109.
National Conference of State Legislatures. Preventing and mitigating the effects of adverse childhood experiences. 2018. http://www.ncsl.org/Portals/1/HTML_LargeReports/ACEs_2018_32691.pdf. Accessed 1 Nov 2021.
Bradford K. Reducing the effects of adverse childhood experiences: National Conference of State Legislatures; 2020. https://www.ncsl.org/research/health/reducing-the-effects-of-adverse-childhood-experiences.aspx. Accessed 1 Nov 2021
Centers for Disease Control and Prevention Preventing Adverse Childhood Experiences (ACEs): Leveraging the best available evidence. 2019. https://www.cdc.gov/violenceprevention/pdf/preventingACES.pdf. Accessed 1 Nov 2021.
Raissian KM, Bullinger LR. Money matters: does the minimum wage affect child maltreatment rates? Child Youth Serv Rev. 2017;72:60–70.
Klevens J, Schmidt B, Luo F, Xu L, Ports KA, Lee RD. Effect of the earned income tax credit on hospital admissions for pediatric abusive head trauma, 1995-2013. Public Health Rep. 2017;132(4):505–11.
Walsh D, McCartney G, Smith M, Armour G. Relationship between childhood socioeconomic position and adverse childhood experiences (ACEs): a systematic review. J Epidemiol Community Health. 2019;73(12):1087–93.
MacMillan HL, Wathen CN, Barlow J, Fergusson DM, Leventhal JM, Taussig HN. Interventions to prevent child maltreatment and associated impairment. Lancet. 2009;373(9659):250–66.
Miller TR. Projected outcomes of nurse-family partnership home visitation during 1996–2013, USA. Prev Sci. 2015;16(6):765–77.
Garner AS, Shonkoff JP, Siegel BS, et al. Early childhood adversity, toxic stress, and the role of the pediatrician: translating developmental science into lifelong health. Pediatrics. 2012;129(1):e224–31.
Crowley DM, Connell CM, Noll J, Green L, Scott T, Giray C. Legislating to prevent adverse childhood experiences: growth and opportunities for evidence-based policymaking and prevention. Prev Sci. 2022;23(2):181-91. https://doi.org/10.1007/s11121-021-01292-x.
Purtle J, Nelson KL, Gollust SE. Public opinion about adverse childhood experiences: social stigma, attribution of blame, and government intervention. Child Maltreat. 2021:10775595211004783. https://doi.org/10.1177/10775595211004783.
Peterson C, Florence C, Klevens J. The economic burden of child maltreatment in the United States, 2015. Child Abuse Negl. 2018;86:178–83.
Purtle J, Nelson KL, Srivastav A, Gollust SE. Perceived persuasiveness of evidence about adverse childhood experiences: Results from a national survey. Academic pediatrics. 2021;21(3):529-33.
Gollust SE, Nelson KL, Purtle J. Selecting evidence to frame the consequences of adverse childhood experiences: testing effects on public support for policy action, multi-sector responsibility, and stigma. Prev Med. 2022;154:106912.
Brownson RC, Eyler AA, Harris JK, Moore JB, Tabak RG. Research full report: getting the word out: new approaches for disseminating public health science. J Public Health Manag Pract. 2018;24(2):102.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. A systematic review of barriers to and facilitators of the use of evidence by policymakers. BMC Health Serv Res. 2014;14(1):1–12.
Oliver K, Innvar S, Lorenc T, Woodman J, Thomas J. Barriers and facilitators of the use of evidence by policy makers: an updated systematic review. BMC Health Serv Res. 2014;14(2);1-12.
Long B, Scott T, Anderson A, Pugel J, Crowley M. Getting science into the hands of state and federal lawmakers: lessons learned in messaging. In: APHA’s 2020 VIRTUAL Annual Meeting and Expo (Oct. 24-28). Washington: American Public Health Association; 2020.
Fernandes M, Long E, Pugel J, et al. Rapport building communications for optimizing science dissemination. In: Paper presented at: SPR Virtual 29th Annual Meeting; 2021.
Long EC, Pugel J, Scott JT, et al. Rapid-cycle experimentation with state and federal policymakers for optimizing the reach of racial equity research. Am J Public Health. 2021;111(10):1768-71. https://doi.org/10.2105/AJPH.2021.306404.
Levine AS. Single conversations expand practitioners’ use of research: evidence from a field experiment. PS: Polit Sci Polit. 2021;54(3):432-7.
Levine AS. Why do practitioners want to connect with researchers? Evidence from a field experiment. PS. Polit Sci Polit. 2020;53(4):712–7.
Niederdeppe J, Winett LB, Xu Y, Fowler EF, Gollust SE. Evidence-based message strategies to increase public support for state investment in early childhood education: results from a longitudinal panel experiment. Milbank Q. 2021;99(4):1088–131.
Niederdeppe J, Roh S, Dreisbach C. How narrative focus and a statistical map shape health policy support among state legislators. Health Commun. 2016;31(2):242–55.
Winett LB, Niederdeppe J, Xu Y, Gollust SE, Fowler EF. When “Tried and True” advocacy strategies backfire: narrative messages can undermine state legislator support for early childcare policies. J Public Interest Commun. 2021;5(1):45.
Shor B, Thurber JA, Yoshinaka A. Polarization in American state legislatures. In: American gridlock: the sources, character, and impact of political polarization; 2015. p. 203–21.
Iyengar S, Lelkes Y, Levendusky M, Malhotra N, Westwood SJ. The origins and consequences of affective polarization in the United States. Annu Rev Polit Sci. 2019;22:129–46.
Fowler EF, Baum LM, Barry CL, Niederdeppe J, Gollust SE. Media messages and perceptions of the Affordable Care Act during the early phase of implementation. J Health Polit Policy Law. 2017;42(1):167–95.
Gollust SE, Barry CL, Niederdeppe J. Partisan responses to public health messages: motivated reasoning and sugary drink taxes. J Health Polit Policy Law. 2017;42(6):1005–37.
Pugel J, Long EC, Fernandes MA, et al. Who is listening? Profiles of policymaker engagement with scientific communication. Policy Internet. 2022;14(1):186–201.
Meisel ZF, Metlay JP, Sinnenberg L, et al. A randomized trial testing the effect of narrative vignettes versus guideline summaries on provider response to a professional organization clinical policy for safe opioid prescribing. Ann Emerg Med. 2016;68(6):719–28.
Calfano B. Power lines: unobtrusive assessment of e-mail subject line impact on organization website use. J Political Mark. 2019;18(3):179–95.
Sahni NS, Wheeler SC, Chintagunta P. Personalization in email marketing: the role of noninformative advertising content. Mark Sci. 2018;37(2):236–58.
Petty RE, Cacioppo JT. The elaboration likelihood model of persuasion. In: Communication and persuasion. New York: Springer; 1986. p. 1–24.
Petty RE, Cacioppo JT. Attitudes and persuasion: classic and contemporary approaches. Boulder: Westview Press; 1996.
Petty RE, Haugtvedt CP, Smith SM. Elaboration as a determinant of attitude strength: creating attitudes that are persistent, resistant, and predictive of behavior. Attitude Strength: Antecedents and consequences. 1995;4:93-130.
Liang L, Bernhardsson S, Vernooij RW, et al. Use of theory to plan or evaluate guideline implementation among physicians: a scoping review. Implement Sci. 2017;12(1):26.
Vogt F, Hall S, Hankins M, Marteau TM. Evaluating three theory-based interventions to increase physicians’ recommendations of smoking cessation services. Health Psychol. 2009;28(2):174.
Taber CS, Lodge M. Motivated skepticism in the evaluation of political beliefs. Am J Polit Sci. 2006;50(3):755–69.
Lee JJ. Party polarization and trust in science: what about democrats? Socius. 2021;7:23780231211010101.
Cochran WG. Sampling techniques. New York: Wiley; 1977.
Dijkstra A. The psychology of tailoring-ingredients in computer-tailored persuasion. Soc Personal Psychol Compass. 2008;2(2):765–84.
Dobbins M, Rosenbaum P, Plews N, Law M, Fysh A. Information transfer: what do decision makers want and need from researchers? Implement Sci. 2007;2(1):1–12.
Mitton C, Adair CE, McKenzie E, Patten SB, Perry BW. Knowledge transfer and exchange: review and synthesis of the literature. Milbank Q. 2007;85(4):729–68.
Merrick MT, Ford DC, Ports KA, et al. Vital signs: estimated proportion of adult health problems attributable to adverse childhood experiences and implications for prevention—25 states, 2015–2017. Morb Mortal Wkly Rep. 2019;68(44):999.
Bethell C, Jones J, Gombojav N, Linkenbach J, Sege R. Positive childhood experiences and adult mental and relational health in a statewide sample: associations across adverse childhood experiences levels. JAMA Pediatr. 2019;173(11):e193007.
Bryant DJ, Oo M, Damian AJ. The rise of adverse childhood experiences during the COVID-19 pandemic. Psychol Trauma Theory Res Pract Policy. 2020;12(S1):S193.
Cuartas J. Heightened risk of child maltreatment amid the COVID-19 pandemic can exacerbate mental health problems for the next generation. Psychol Trauma Theory Res Pract Policy. 2020;12(S1):S195.
Holland KM, Jones C, Vivolo-Kantor AM, et al. Trends in US emergency department visits for mental health, overdose, and violence outcomes before and during the CoViD-19 pandemic. JAMA Psychiatry. 2021;78(4):372–9.
Gollust SE, Seymour JW, Pany MJ, Goss A, Meisel ZF, Grande D. Mutual distrust: perspectives from researchers and policy makers on the research to policy gap in 2013 and recommendations for the future. Inquiry. 2017;54:0046958017705465.
Liu-Thompkins Y. A decade of online advertising research: what we learned and what we need to know. J Advert. 2019;48(1):1–13.
U.S. Department of Health & Human Services, Administration for Children and Families, Administration on Children, Youth and Families, Children’s Bureau. (2019). Child maltreatment 2017. https://www.acf.hhs.gov/sites/default/files/documents/cb/cm2017.pdf. Accessed 1 Nov 2021.
Engel-Rebitzer E, Stokes DC, Buttenheim A, Purtle J, Meisel ZF. Changes in legislator vaccine-engagement on Twitter before and after the arrival of the COVID-19 pandemic. Hum Vaccin Immunother. 2021;17(9):2868-72. https://doi.org/10.1080/21645515.2021.1911216.
Guntuku SC, Purtle J, Meisel ZF, Merchant RM, Agarwal A. Partisan differences in Twitter language among US legislators during the COVID-19 pandemic: cross-sectional study. J Med Internet Res. 2021;23(6):e27300.
Stokes DC, Purtle J, Meisel ZF, Agarwal AK. State legislators’ divergent social media response to the opioid epidemic from 2014 to 2019: longitudinal topic modeling analysis. J Gen Intern Med. 2021;36(11):3373-82. https://doi.org/10.1007/s11606-021-06678-9.
Mutz DC, Pemantle R, Pham P. The perils of balance testing in experimental design: messy analyses of clean data. Am Stat. 2019;73(1):32–42.
De Boer MR, Waterlander WE, Kuijper LD, Steenhuis IH, Twisk JW. Testing for baseline differences in randomized controlled trials: an unhealthy research behavior that is hard to eradicate. Int J Behav Nutr Phys Act. 2015;12(1):1–8.
Turner EL, Prague M, Gallis JA, Li F, Murray DM. Review of recent methodological developments in group-randomized trials: part 2—analysis. Am J Public Health. 2017;107(7):1078–86.
Snijders TA, Bosker RJ. Multilevel analysis: an introduction to basic and advanced multilevel modeling. Thousand Oaks: Sage; 2011.
Brownson RC, Dodson EA, Stamatakis KA, et al. Communicating evidence-based information on cancer prevention to state-level policy makers. J Natl Cancer Inst. 2011;103(4):306–16.
Lyons RA, Kendrick D, Towner EM, et al. The advocacy for pedestrian safety study: cluster randomised trial evaluating a political advocacy approach to reduce pedestrian injuries in deprived communities. PLoS One. 2013;8(4):e60158.
Armstrong BG. Effect of measurement error on epidemiological studies of environmental and occupational exposures. Occup Environ Med. 1998;55(10):651–6.
Access to Quorum was provided by the Center for Health Economics of Treatment Interventions for Substance Use Disorder, HCV, and HIV (National Institute on Drug Abuse grant 1P30DA040500-01) and in partnership with the Research-to-Policy Collaboration, affiliated with The Pennsylvania State University’s Edna Bennett Pierce Prevention Research Center.
The study was funded by the Robert Wood Johnson Foundation (76069).
Ethics approval and consent to participate
The study was approved by the Drexel University IRB (1912007529).
Consent for publication
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
CONSORT and TIDieR checklists.
Example Policy Briefs.
Unadjusted and Adjusted Associations between Study Conditions, Demographics, and Mentions of Terms Related to Dissemination Materials in U.S. State Legislators’ Social Media Posts and Newsletters to Constituents, 2021.
About this article
Cite this article
Purtle, J., Nelson, K.L., Gebrekristos, L. et al. Partisan differences in the effects of economic evidence and local data on legislator engagement with dissemination materials about behavioral health: a dissemination trial. Implementation Sci 17, 38 (2022). https://doi.org/10.1186/s13012-022-01214-7
- Adverse childhood experiences
- Child maltreatment
- Behavioral health
- United States