Development of a framework and coding system for modifications and adaptations of evidence-based interventions

Evidence-based programs and interventions are frequently modified during the implementation process to address differences between the context in which the intervention was originally designed and tested, and the one into which it is ultimately implemented [1]. Descriptions of modifications in the research literature range broadly from slight changes in terminology or delivery in different languages, to removal of core components or integration with other interventions. Modifications can include adaptations, which are planned or purposeful changes to the design or delivery of an intervention, but they can also include unintentional deviations from the interventions as originally designed. That is, some modifications occur with the intention to retain fidelity to the fundamental elements or spirit of the intervention, whereas others may be unplanned changes made in reaction to a specific circumstance. Some may be relatively minor, while others might represent a significant change. Such variation in the nature of modifications can have very different implications for outcomes of interest. While some modifications might facilitate implementation and sustainability by improving the fit between the intervention and the target population or the context into which it is introduced, modifications may also erode treatment integrity. However, little research has been conducted to determine whether modifications of different natures compromise or enhance the desired benefits of interventions. Assessing fidelity, which comprises adherence to the intervention components, competence or skill with which the intervention is delivered, and differentiation from other treatments [2], by itself may fail to capture certain types of modifications (e.g., minor changes to terminology or language). While the recommended considerations for fidelity include unique and essential elements, necessary but not unique, acceptable but not necessary, and proscribed elements [3], most fidelity instruments do not contain an exhaustive listing of acceptable and proscribed behaviors. Thus, fidelity monitoring alone will not facilitate an understanding of whether different types of modifications are detrimental, non-detrimental or enhancements [4, 5]. Without a better understanding of the nature and impact of modification, and the levels of fidelity necessary to promote desired outcomes, it is difficult to determine the best course of action with respect to the implementation of complex interventions in different contexts. This manuscript presents a comprehensive framework and model for classifying a broad range of modifications that may be made to evidence-based interventions. Such a framework, by quantifying the specific types and levels of modifications, can allow for more precise determination of the effects of such modifications on clinical or implementation outcomes of interest.

While the case has been made both for strict fidelity to interventions and for modifying interventions as necessary [4, 6–10], few studies have examined the impact of modifications to treatments on health-related behaviors or outcomes. Among those studies, results have not been consistent. Levitt and colleagues compared outcomes of an intervention for post-traumatic stress disorder (PTSD) that included the option to use certain prescribed modifications, such as repeating or skipping modules, with clinical outcomes from a randomized controlled trial [11]. In this study, levels of fidelity to core intervention components remained high when the intervention was delivered with modifications, and PTSD symptom outcomes were comparable to those in a controlled clinical trial [11]. Galovski and colleagues also found positive outcomes when a highly specified set of adaptations were used in a different PTSD treatment [12]. Other studies have demonstrated similar or improved outcomes after modifications were made to fit the needs of the local audience and expand the target population beyond the original intervention. For example, an enhanced outcome was demonstrated after modifying a brief HIV risk-reduction video intervention to match presenter and participant ethnicity and sex [13]; effectiveness was also retained after modifying an HIV risk-reduction intervention to meet the needs of five different communities [14]. However, in other studies, modifications to enhance local acceptance appeared to compromise effectiveness. For example, Stanton and colleagues modified a sexual risk reduction intervention that had originally been designed for urban populations to address the preferences and needs of a more rural population, but found that the modified intervention was less effective than the original, unmodified version [15]. Similarly, in another study, cultural modifications that reduced dosage or eliminated core components of the Strengthening Families Program increased retention but reduced positive outcomes [16].

A challenge to a more complete understanding of the impact of specific types of modifications is a lack of attention to their classification. Some descriptions of intervention modifications and adaptations have been published (c.f. [17–19]), but there have been relatively few efforts to systematically categorize them. Researchers identified modifications made to evidence-based interventions such as substance use disorder treatments [1] and prevention programs [20] through interviews with facilitators in different settings. Others have described the process of adaptation (e.g., [21, 22]). For example, Devieux and colleagues [23] described a process of operationalizing the adaptation process based on Bauman and colleagues’ framework for adaptation [8], which includes efforts to retain the integrity of an intervention’s causal/conceptual model. Other researchers [24–26] have also made recommendations regarding specific processes for adapting mental health interventions to address individual or population-level needs while preserving fidelity. Some work has been done to characterize and examine the impact of modifications made at the individual and population level. For example, Castro, Barrera and Martinez presented a program adaptation framework that described two basic forms of cultural adaptation: the modification of program content and modification of program delivery, and made distinctions between tailored and individualized interventions [27]. A description of person-centered interventions similarly differentiates between tailored, personalized, targeted and individualized interventions, all of which may actually lie on a continuum in terms of their complexity and comprehensiveness [27].

While these existing recommendations and models for adaptation provide critical guidance regarding the process of adaptation, particularly to improve the cultural relevance and individual-level ‘fit’ of interventions, adaptations may also occur to address provider needs or constraints in the intervention or program setting or healthcare system. Relatively little research has been conducted to empirically classify the nature of the full range of modifications made to interventions in routine care settings. Recognizing the benefits to identifying common types of modifications to evidence-based interventions, Hill and colleagues identified the types of modifications made for a single evidence-based prevention program in a statewide implementation [20]. Although they suggested that their study was a starting point in the development of a set of modification categories that generalize across interventions, the modifications that they identified were highly specific to the intervention that they examined. A more general and developed taxonomy can facilitate efforts to understand the nature of modifications that are made, whether by design or happenstance, as well as the impact of different types of modifications on implementation and health-related outcomes of interest. As a next step toward this understanding, it is necessary to identify the types of modifications that may occur when implementing interventions under a broad set of circumstances and in a variety of settings.

The purpose of this study was to develop a coding scheme to characterize modifications made to evidence-based interventions when they are implemented in contexts or with populations that differ from that in which they were originally developed or tested. Such a system can facilitate more systematic study of the types of modifications that are most commonly made across different contexts, populations and interventions. Additionally, it can provide a way to study the impact of different types of modifications on outcomes of interest. Hill and colleagues described a theory that a few types of modifications are likely to comprise the majority of all modifications that occur in practice [20]. Identifying what those modifications are, and what their impacts are for different interventions, can assist intervention developers and those who implement the interventions in determining and facilitating the range of modifications that are acceptable—and in preventing those that are not. We therefore sought to identify examples of a variety of modifications that practitioners, treatment developers, and other stakeholders made to a diverse set of interventions and programs. We intended this framework to apply in particular to three types of programs and interventions outlined by Scheirer [28]: those implemented by individual providers; programs requiring coordination among multiple staff; and new procedures aimed at targeting individual behaviors or behavioral health conditions.

In developing the coding system, we searched the literature for articles published or in press before June, 2012, that assessed or described modifications to interventions implemented in routine service settings. We searched the following databases: Medline, ISI, PsycInfo, Academic Search Premier, Health Source, ERIC, Pubmed and Google Scholar, using the terms ‘modif’ or ‘adapt’ and ‘evidence based treatment’ or ‘evidence-based intervention.’ We also employed a snowballing strategy, in which we searched the reference sections of articles that we identified as well as theoretical papers on implementation that discussed modification and adaptation. Two authors (SWS, AC) reviewed abstracts and full text articles when necessary to determine their eligibility for this project and discussed one difference of opinion regarding inclusion with the rest of the study team. We included articles that provided sufficient detail about one or more modifications to facilitate coding. Modifications could be either adaptations (intentional, planned changes that typically included an effort to preserve fidelity) or changes that were made without premeditation during the delivery of the intervention. Articles were excluded if they assessed fidelity but did not describe modifications; if the interventions were developed, as opposed to being modified, for the purpose of the study or implementation project; or if they only provided recommendations for adaptation of an intervention without describing at least one specific modification that was made during the course of a research or implementation effort (Figure 1).

Once identified, articles were examined and segmented into discrete units for coding. One team member (SWS) segmented the articles into separate descriptions of modifications, and the remaining team members reviewed the entire articles and provided feedback regarding the accuracy of the segmentation process. Due to the dearth of previous work that has been done to actually classify a broad range of modifications, we used an analytic approach that was rooted in grounded theory [29, 30] to code the data from these articles. Using an iterative process, the study team examined the subset of identified segments that described modifications to identify emergent themes or categories. We then applied these themes to additional segments, allowing additional categories to emerge until theoretical saturation was achieved and a comprehensive coding scheme was developed. We specified characteristics of each category of modifications and ensured that the categories were mutually exclusive and exhaustive by combining or revising redundant codes. We also provided a draft of the framework and codebook to seven implementation researchers who were interested in the subject of modification and requested feedback. No additional codes or alternative classification structures were suggested, although some suggested combining some similar constructs into a single code and identified aspects of the codebook that could be clarified. After this feedback was incorporated, two team members (SWS, KT) then re-examined the article data, such that all segments identified in each article were coded with the finalized codebook. The two raters overlapped on 20% of the identified segments, and we computed Cohen’s kappa coefficients [31] to determine interrater reliability for each rating category.

This study represents an effort to systematically characterize the types of modifications that are made to interventions when they are implemented in real world settings. On a high level, two of the major categories of coding mapped onto Castro and colleagues’ distinction between modifications of program content, and modification of the form of delivery (e.g., location of delivery, delivery person, or channel of delivery) [27]. In a sample of studies described in peer-reviewed articles, which represent a variety of interventions and contexts, we found that contextual modifications were occasionally reported, but that content modifications were reported much more frequently. Tailoring the intervention to address language, cultural differences, literacy, or situational constraints was the most commonly identified content modification, followed by the addition or removal of elements and changes to the length or pacing of the intervention.

Other modifications identified in our coding process, such as drift and loosening of structure, occurred relatively rarely within the articles that we reviewed. This low frequency in the current sample is not surprising, as such behaviors are unlikely to occur in a planned manner, and may be less likely to be emphasized when describing an evidence-based intervention in a peer-reviewed article. Furthermore, relatively few of the articles that were sampled employed the type of observation or stakeholder interviews through which such behaviors may be identified. While drift might also be considered a discontinuation of the intervention entirely or a lack of fidelity rather than a modification, it also seems important to capture it in a system designed to classify deviations from and modifications to a protocol in order to better measure its impact on outcomes of interest. For example, the impact of the option to occasionally or strategically drift on clinician or client satisfaction may be important to explore, in addition to the impact of drift on clinical effectiveness.

In contrast to the findings in Hill and colleagues’ study [20], most of the articles that we found in our search process described modifications that were made proactively in recognition of key differences between the implementation setting and the original intervention. In another report, we describe findings that emerged when we applied this framework to interview data from a sample of community-based mental health service providers who were trained in an EBP [53]. Several of the lower-frequency modifications identified in the current study were endorsed much more frequently in that study, suggesting that modifications made proactively may differ from those made once implementation is underway. Thus, at this stage of development, we determined that it is important to represent a more exhaustive set of possible modifications in the classification system.

As the discussion above indicates, some modifications may signify decreases in fidelity, while others may be consistent with the design of the intervention. The tension between modification and fidelity is a critical issue in implementation science [4, 54, 55]. Many recognize that modifications will occur throughout the course of an implementation effort, but the type and extent of modifications that can occur without compromising effectiveness or degrading fidelity to an unacceptable degree has not been sufficiently explored. In theory, it is possible to make some types of modifications without compromising effectiveness or removing the key elements of an intervention. However, for some interventions, the core elements have not yet been determined empirically, and very little is known about the impact of behaviors such as integrating other interventions or selectively implementing particular aspects of a treatment. Fidelity measures that emphasize competence or the spirit of an intervention over adherence may not adequately capture some potentially important types of modification, and those that emphasize adherence may not capture modifications such as tailoring. Thus, when observation or reliable self-report is possible, the use of a fidelity measure along with this modification framework can guide decisions regarding the extent to which a particular modification represents a departure from core elements of an intervention. Used alone or as a complement to fidelity measures, this measure may also be useful in determining whether particular elements can be removed, re-ordered, integrated or substituted without compromising effectiveness.

Despite the breadth of the coding system we developed, interrater agreement for the subset of independently-coded articles was quite high, reaching standards of ‘substantial agreement’ and ‘almost perfect’ agreement for the level and nature of modifications, respectively [52]. Within our research group, this level of reliability was achieved after a brief series of hour-long weekly coding meetings, suggesting that our coding scheme can be used to reliably classify modifications described in research articles without overly burdensome training.

We note several potential limitations to the study and framework. First, our search process was not intended to identify every article that described modifications to evidence-based interventions, particularly if adaptation or modification was not a major topic addressed in the article. Instead, we sought to identify articles describing modifications that occurred across a variety of different interventions and contexts and to achieve theoretical saturation. In the development of the coding system, we did in fact reach a point at which additional modifications were not identified, and the implementation experts who reviewed our coding system also did not identify any new concepts. Thus, it is unlikely that additional articles would have resulted in significant additions or changes to the system.

In our development of this framework, we made a number of decisions regarding codes and levels of coding that should be included. We considered including codes for planned vs. unplanned modifications, major vs. minor modifications (or degree of modification), codes for changes to the entire intervention vs. changes to specific components, and codes for reasons for modifications. We wished to minimize the number of levels of coding in order to allow the coding scheme to be used in quantitative analyses. Thus, we did not include the above constructs, or constructs such as dosage or intensity, which are frequently included in frameworks and measures for assessing fidelity [56]. Additionally, we intend the framework to be used for multiple types of data sources, including observation, interviews and descriptions, and we considered how easily some codes might be applied to information derived from each source. Some data sources, such as observations, might not allow coders to discern reasons for modification or make distinctions between planned and unplanned modifications, and thus we limited the framework to characterizations of modifications themselves rather than how or why they were made. However, sometimes, codes in the existing coding scheme implied additional information such as reasons for modifying. For example, the numerous findings regarding tailoring interventions for specific populations indicate that adaptations to address differences in culture, language or literacy were common. Aarons and colleagues offer a distinction of consumer-driven, provider-driven, and organization-driven adaptations that might be useful for researchers who wish to include additional information regarding how or why particular changes were made [35]. While major and minor modifications may be easier to distinguish by consulting the intervention’s manual, we also decided against including a code for this distinction. Some interventions have not empirically established which particular processes are critical, and we hope that this framework might ultimately allow an empirical exploration of which modifications should be considered major (e.g., having a significant impact on outcomes of interest) for specific interventions. Furthermore, our effort to develop an exhaustive set of codes meant that some of the types of modifications, or individuals who made the modifications, appeared at fairly low frequencies in our sample, and thus, their reliability and utility require further study. As it is applied to different interventions or sources of data, additional assessment of reliability and further refinement to the coding system may be warranted.

An additional limitation to the current study is that our ability to confidently rate modifications was impacted by the quality of the descriptions provided in the articles that we reviewed. At times, it was necessary to make some assumptions about how things were actually modified, or the level at which the modifications occurred. The level of detail available in records, clinical notes, or other qualitative data that may be utilized to investigate modifications may similarly impact future investigations. We attempted to address this limitation by making decision rules about the level of detail and clarity required to assign codes and by documenting these rules in detail in our coding manual. The level of rater agreement that we achieved suggests that our process was reasonably successful, despite occasional ambiguities in the descriptions. In future efforts to utilize this system, two strategies can minimize the likelihood that insufficient data are available to assign codes. Whenever possible, observation by raters knowledgeable about the intervention and its core components should be used to identify modifications. This may be especially important in differentiating minor modifications (which might be coded as ‘tailoring/tweaking/refining’) from more intensive modifications (which, for example, might be coded as ‘removing elements’); ultimately, making these distinctions requires a thorough knowledge of the intervention itself. When interviews are conducted in lieu of observation or in addition to review of existing records, we recommend asking very specific follow-up questions regarding modifications that are made. Familiarity with both the intervention and the coding system when interviewing can increase the likelihood that sufficient information is obtained to make an appropriate judgment. Despite these measures, interrater reliability may vary across different data sources, although additional work by our research group suggests that reliability remains high when the coding scheme is applied to interview data [53]. We are currently examining reliability when the coding scheme is used for observation using audio recordings of psychotherapy sessions as well, and we recommend that when using this framework, researchers assess reliability.

We believe that the framework that we present can be used flexibly depending on the goals of the research and the type of data collection that occurs. For example, researchers may wish only to code exclusively for content or context-level modifications if they are interested in determining the impact of specific types of modifications on health outcomes. Similarly, the code for the decision maker may not be necessary if researchers are studying modifications made by one particular group or evaluating adaptations that were pre-specified by a single decision-maker before implementation began. However, this code might be very informative if the researchers wish to understand the impact of the nature and process of modification on outcomes such as stakeholder engagement or fidelity to core program or intervention elements.

This coding system may be used to advance research that is designed with the goal of understanding the impact of changes made to interventions in particular contexts. Ultimately, such an understanding will require simultaneous use of this coding scheme and treatment outcome assessments, in order to help researchers and clinicians determine what specific types of modifications are most useful in increasing the effectiveness of interventions. Such an understanding will allow stakeholders to make more informed decisions about whether and how to modify the interventions when implementing them in contexts that differ from those in which they were originally developed and tested. Additionally, when used in the context of fidelity monitoring, this system can provide more useful information about what actually occurs when lower levels of adherence are identified, as well as the types of modifications that can occur within acceptable levels of fidelity. Baumann and colleagues suggested that there is a range of feasible fidelity, as well as a point of ‘dramatic mutation,’ at which the intervention is no longer recognizable or effective [8]. This system of characterizing modifications may be useful in determining these ranges and boundaries with greater specificity. By understanding the types of modifications that can be made while keeping the intervention out of the range of dramatic mutation, stakeholders may ultimately find it easier to adapt interventions as needed while attending to an intervention’s most critical components. Investigations of the impact of particular types of modifications on clinical outcomes can further inform efforts to implement evidence-based interventions while preserving desired levels of effectiveness. Finally, another potential area of investigation using this framework is on the impact of specific modifications on implementation outcomes such as adoption and sustainability. Additional knowledge about these critical issues in implementation science will yield important guidance for those wishing to advance the implementation of evidence-based programs and interventions.