Skip to main content

Implementation of the CALM intervention for anxiety disorders: a qualitative study



Investigators recently tested the effectiveness of a collaborative-care intervention for anxiety disorders: Coordinated Anxiety Learning and Management(CALM) []) in 17 primary care clinics around the United States. Investigators also conducted a qualitative process evaluation. Key research questions were as follows: (1) What were the facilitators/barriers to implementing CALM? (2) What were the facilitators/barriers to sustaining CALM after the study was completed?


Key informant interviews were conducted with 47 clinic staff members (18 primary care providers, 13 nurses, 8 clinic administrators, and 8 clinic staff) and 14 study-trained anxiety clinical specialists (ACSs) who coordinated the collaborative care and provided cognitive behavioral therapy. The interviews were semistructured and conducted by phone. Data were content analyzed with line-by-line analyses leading to the development and refinement of themes.


Similar themes emerged across stakeholders. Important facilitators to implementation included the perception of "low burden" to implement, provider satisfaction with the intervention, and frequent provider interaction with ACSs. Barriers to implementation included variable provider interest in mental health, high rates of part-time providers in clinics, and high social stressors of lower socioeconomic-status patients interfering with adherence. Key sustainability facilitators were if a clinic had already incorporated collaborative care for another disorder and presence of onsite mental health staff. The main barrier to sustainability was funding for the ACS.


The CALM intervention was relatively easy to incorporate during the effectiveness trial, and satisfaction was generally high. Numerous implementation and sustainability barriers could limit the reach and impact of widespread adoption. Findings should be interpreted with the knowledge that the ACSs in this study were provided and trained by the study. Future research should explore uptake of CALM and similar interventions without the aid of an effectiveness trial.

Peer Review reports


Anxiety disorders are highly prevalent. In the United States, population surveys estimate that close to 29% have experienced an anxiety disorder in their lifetime [1], while about 18% experience an anxiety disorder each year [2]. Individuals with anxiety disorders suffer significant impairment and reduced quality of life [35], and they are costly to healthcare systems and economies in general due to high use of healthcare and reduced productivity [68]. Despite the high prevalence and disease burden, only about one-third of persons with anxiety disorders receive treatment in a given year [9]. Most of the treatment sought for anxiety is in primary care, and the most frequent treatment is medication [3, 9]. While treatment guidelines recommend selective serotonin reuptake inhibitor medication and cognitive behavioral therapy (CBT) as first-line treatments [10, 11], CBT is under-utilized among anxiety patients [3], and the quality of medication provided is suboptimal [9, 12].

As primary care is the most frequent care location for patients with anxiety, it is not surprising that clinical research has focused on interventions for that setting. Many of the empirically supported interventions to date can be characterized as "collaborative care," as they center around the activities of a care manager, usually a nurse or social worker, who maintains relatively frequent contact with the patient; performs a range of psychoeducational, therapeutic, and monitoring functions; and works as a liaison between the primary care provider and a consulting mental health professional, usually a psychiatrist. While most of the research and implementation work in the field thus far has concerned itself with collaborative care for depression [1315], a growing literature supports collaborative-care interventions for panic disorder [16, 17] and for both panic disorder and generalized anxiety disorder [18]. Authors of the current manuscript [19] conducted a large randomized controlled trial of a collaborative-care intervention in 17 community primary clinics for four anxiety disorders (generalized anxiety disorder, panic disorder, posttraumatic stress disorder, and/or social anxiety disorder) and recently published data showing it is effective, with small to medium effects sizes, in comparison to treatment as usual [20].

Given the effectiveness of these interventions, a next generation of work will likely pursue implementation research and the development and testing of implementation interventions and strategies to support widespread adoption of collaborative care for anxiety disorders. This work will draw upon current models of implementation (e.g., Greenhalgh et al., 2004; Fixsen, 2005) [21, 22] and the current implementation research supporting current "roll-out" efforts for collaborative care for depression (and other chronic conditions) in large/national healthcare systems (e.g., the Department of Veterans Affairs in the United States [15, 23] and several countries in Europe [24]). However, the anxiety-specific collaborative-care interventions might have unique features and could present their own barriers and facilitators to translation into routine care, whether they are adopted on their own or in conjunction with other collaborative-care interventions for other mental health disorders such as depression or alcohol use disorders. To date, no implementation trials have been conducted for anxiety collaborative-care interventions, and little is known about the presence or absence of specific barriers and facilitators to implementing and sustaining such an intervention.

The purpose of this paper is to present qualitative data from a process evaluation of the Coordinated Anxiety Learning and Management (CALM) intervention, as part of a randomized controlled effectiveness trial of a collaborative-care for anxiety disorders. As noted above, results from this study have been recently published demonstrating CALM's clinical effectiveness [20]. Alongside the effectiveness trial, investigators conducted a qualitative process evaluation focused on the implementation of the CALM intervention in the participating clinics. In the field of implementation science, qualitative methods are frequently used to improve understanding of healthcare contexts and to shed light on key implementation challenges [2527]. These methods are commonly part of what Stetler et al. [28] refer to as "developmental formative evaluation", that is, context-specific "diagnostic" evaluation of barriers and facilitators to implementation, which leverages the experiences and lessons learned in effectiveness trials to enhance the development of wider implementation and dissemination interventions. The qualitative study described here is just such a developmental formative evaluation. The key research questions were as follows: (1) What were the facilitators/barriers to implementing CALM? (2) What were the facilitators/barriers to sustaining CALM after the study was completed?

It should be noted here that the study investigated implementation of the CALM intervention during a clinical effectiveness trial-the clinics were not explicitly attempting to adopt the CALM intervention as part of their routine care. The collaborative-care interventionists were employees of the research study (though some were previously employed by these same clinics), and the providers were referring to them within the context of a research study. However, many of the challenges and successes associated with implementing research interventions within community clinics are illustrative and informative to future implementation efforts [29, 30]. Further, in the current study, the investigators asked questions regarding the prospects of retaining the intervention after the research grant was completed, and those data are directly applicable to the issue of sustainability.


Context of the process evaluation: The CALM effectiveness study

A total of 1004 adult primary care patients with one of four anxiety disorders (generalized anxiety disorder, panic disorder, posttraumatic stress disorder, or social anxiety disorder) were recruited from 17 clinics within four national sites (Seattle, Los Angeles, San Diego, and Little Rock). The clinics were purposively selected based on a number of considerations, including clinician interest, space availability, size and diversity of patient population, and insurance mix. Anxiety clinical specialists (ACSs) delivered education, self-activation in the context of promoting medication adherence, and CBT to intervention patients and monitored their symptoms using a web-based system in which they recorded anxiety and depression ratings at each contact. Intervention patients chose CBT (34%), anti-anxiety medications (9%), or both (57%) in a "stepped care" treatment that varied according to clinical need. ACSs were located on-site in each clinic and conducted face-to-face CBT with the assistance of a computerized program. ACSs interacted with an off-site study psychiatrist as needed and communicated clinical recommendations about medications between the study psychiatrist and the patients' primary care providers. Control patients received usual care from their primary care clinician. Anxiety symptoms, functioning, satisfaction with care, and healthcare utilization were assessed at six-month intervals. The salaries of the ACSs were covered by the study, and most facilities received some additional assistance/incentives, such as fees, to cover the use of space and/or small amounts of salary coverage for clinic liaisons/champions who helped facilitate the study.

CALM's innovations included (1) the flexibility to treat any one of four anxiety disorders, co-occurring depression, and/or alcohol abuse; (2) using on-site clinicians (ACSs) to conduct initial assessments; (3) a computer-assisted psychotherapy delivery system; and (4) a web-based system customized for anxiety status tracking. CALM was designed for easy dissemination in a variety of primary care settings.

Study locations

The 17 clinics can be categorized as members of large health maintenance organizations (HMOs; n = 4), federally qualified community healthcare centers (n = 4), university-affiliated clinics (n = 4), or private clinics, either free-standing or part of a hospital group (n = 5). Some of the private clinics and federally qualified centers, however, had university affiliations to some extent, though they were not located on university campuses. About two-thirds of the clinics were internal medicine, with the remaining being family practice. Less than half of the clinics had an in-house mental health provider or providers, and those professionals were usually master's-trained clinicians (e.g., social workers, counseling psychologists). About three-fourths of the clinics served some uninsured patients.


The total number of participants in this qualitative key-informant interview study was 61, including 14 ACSs hired and trained by the study to conduct the intervention, 18 primary care physicians (PCPs), 13 primary care nurses, and 16 primary care clinic administrators or other staff members. The ACSs came from nursing or social work backgrounds almost exclusively, and all but two were female. Eleven of the PCPs were internal medicine physicians, and the remaining seven were family practice physicians. Twelve of the physicians were female, and six were male. The clinic nurses were a relatively even mixture of registered nurses, licensed practical nurses, and licensed vocational nurses. All but two were female. The clinic administrators interviewed (n = 8) were the administrative leads for their clinics, and all but three were female. The other "clinic staff" informants were a mixture of front desk clerks, scheduling and administrative assistants, and project coordinators. All but one was female. No other personal or demographic information was collected from the participants. While a substantial majority of the participants were female, we feel that the sample generally reflects the characteristics of the work environment and the gender make-up of the majority of the professions sampled, namely, nursing, clinic administration, and social work.

In terms of recruitment, all of the ACSs were asked to participate in the key informant interviews. Most of the ACSs (N = 9) were interviewed twice, at a midpoint of the intervention's implementation and at the conclusion of the study. Four ACSs were interviewed only at the midpoint (two had moved on to another position by the endpoint of the study). One was interviewed only at the conclusion. Two ACSs refused to participate. The majority of ACSs worked in one clinic at their respective sites, but several worked in more than one (part-time) across the span of the study. The PCPs, nurses, and clinic administrators/staff with moderate-to-strong involvement in the CALM study (as rated by the study coordinators and/or ACSs) were targeted and "oversampled" for participation. The rationale for this was that because the aim of the qualitative implementation study was to uncover barriers and facilitators to implementation and sustainability of the intervention, it was necessary to interview mostly those clinicians and staff with at least moderate knowledge about the implementation, firsthand, in their clinics. However, clinicians and staff less involved with the CALM intervention (approximately 20% of the sample) were also interviewed to provide balance. The implications of this sampling strategy, both positive and negative, are discussed later in the manuscript. The clinician and staff participants were all interviewed during the final year of the intervention. We did not track approaches to and refusal rates of clinicians and other staff. Some of the potential participants were approached individually, based on their level of participation with the CALM intervention, while others were collectively approached in open calls for participation in meetings or via email. We did not predefine a desired number of participants per category or site; however, we attempted to get at least two employees per clinic to participate, including at least one clinician.

Data collection

The study protocol, consent forms, and interview guides were reviewed and approved by the Institutional Review Boards at the University of Washington, the University of California at Los Angeles, the University of California at San Diego, and the University of Arkansas for Medical Sciences. After the study was described to potential participants, written informed consent was obtained. Data were collected via key informant interviews by phone. Interview guides were used for each provider/staff group, including ACSs. While each guide elicited some information specific to the provider/staff type, a common core of questions was included in all of the interview guides (see Table 1). The questions were decided upon by the study investigators collectively, and the team revised them several times, including after the interviews began.

Table 1 Core questions for all qualitative interviews

The lead author interviewed the ACS sample at the study's close. One trained interviewer per host site (Seattle, Los Angeles, San Diego, and Little Rock) interviewed their clinic's clinicians and staff. Interviews were recorded and transcribed verbatim. In two instances, the audio recorder malfunctioned and notes from the interviewer were used during coding.

Data analyses

Transcripts were content analyzed through a combination of manual coding of printed transcripts and electronic coding using ATLAS.ti software (ATLAS.ti Scientific Software Development GmBH, Berlin, Germany). Content analysis is a common research method for the subjective interpretation of text through a systematic process of classifying text into categories or themes that represent similar meanings [31]. In the current study, the interview protocol was designed to support both "conventional" and inductive and "directed" a priori content analyses [32, 33]. The protocol included a mixture of "grand-tour" type questions that are common opening questions for inductive analyses (e.g., "Tell me about your role and involvement in the CALM project" and "How was CALM implemented in your clinic and how did it go?") and more specific questions/probes informed by existing conceptual models of implementation [21, 34], which focused on the organizational context (norms and attitudes, routine care procedure, resources), the process of implementation (stage of adoption, variation in implementation), and mechanisms of diffusion (influence of peers/leaders, change agents, incentives). With conventional content analyses, investigators focus on descriptions of phenomena to identify themes and concepts that emerge from reading the interview transcript text without being constricted to a specific theory or conceptual model of behavior (i.e., keeping an open-as opposed to an empty-mind). With directed content analysis, investigators are guided by existing theory or research findings and explore predetermined themes or concepts in their coding. We employed both types of content analysis, with an emphasis on the former.

In terms of coding, the lead author coded the ACS interviews himself, while he and another investigator co-coded the provider/staff interviews. When two coders were involved, they independently coded identical sections of text and compared coding and interpretations. Each subsample of participants' transcripts was coded as a group, in this sequence: ACSs, physicians, nurses, administrators, and staff. This method allowed the coders to focus on that data and emergent themes of one group at a time (as opposed to coding random transcripts), and the sequence allowed the coders to investigate the stakeholders most involved with the CALM intervention first (ACSs), then in descending order of overall stakeholder involvement in implementation.

Informed by the aims of the study and guided by previous examples of similar implementation analyses [2527], the transcripts were first coded using the "top-level" codes (or "macro-themes") of barriers/facilitators to implementation and barriers/facilitators to sustainability. Organizing the analyses around these top-level codes [35] improved the "usability" of the information and allowed the data to be blended easily with other reports documenting implementation barriers/facilitators for major mental health initiatives. Top-level codes were used to broadly categorize the data and represent informants' beliefs about which factors hindered or facilitated implementation and/or sustainability of the CALM intervention and how the intervention could be improved. Further subcoding of categories within each top-level code came next. The subcoding step assigned new codes that described the content of the barriers/facilitators reported by the participants (e.g., "provider interest in mental health issues" as a facilitator of implementation). A third coding step classified the individually categorized barriers/facilitators into types, such as "provider attitudes/behaviors as facilitators" and "clinic structure-related barriers." This last step was interpretive, representing the views of the coders. The final list of codes consisted of behaviors, attitudes, personal characteristics, contexts, processes, and policies the informants believed to be associated with the implementation and sustainability of the CALM intervention.


While numerous barriers and facilitators were recognized and reported by the informants, we chose to highlight here those reported most often and those referred to by the participants as the most salient from their perspective. Additional information about the full range of barriers and facilitators is available from the authors. Below we present barriers and facilitators to implementation and sustainability, across stakeholder and clinic types, organized by the following subcategories: provider attitudes/behavior, clinic structure, intervention characteristics, and patient characteristics (See also Table 2). Barriers/facilitators are noted in italicized text. Exemplar quotations are contained in Additional file 1.

Table 2 Major Themes

Barriers to implementation

Provider attitudes/behaviors

The most often cited barrier to implementation across stakeholders was uneven physician "buy-in" or support for the CALM intervention within clinics. A common pattern emerged: some physicians were highly motivated to participate in the intervention, some were marginally so, and some were not at all motivated to participate. The highly motivated physicians were usually less than half of the total number of physicians in a clinic. Further, participants across stakeholder groups recognized that it was challenging to increase physician interest. Many participants speculated that the less enthusiastic physicians were not comfortable with treating mental health. Others noted that low enthusiasm could also be linked to providers (both physicians and nurses) feeling that the prevalence of anxiety was low in their clinic. Also, many physicians reported that enthusiasm for the intervention could wane without supportive attention from "champions" or "opinion leaders."

Clinic structure

Two very commonly noted barriers to implementation across stakeholder group and clinic types were high prevalence of part-time primary care providers and space concerns for the ACS. The ACS and physician stakeholder groups especially reported concerns about part-time physicians (including residents) being harder to reach with information about CALM than full-time providers, and thus it was harder to engage with them and facilitate their involvement. University-affiliated clinics with large numbers of residents seemed most impacted by the barrier. Informants from all stakeholder groups and clinic types noted barriers associated with finding adequate space for the ACSs to do their work. If an ACS's work space was not proximal to the providers, their relationships with them suffered. Further, if an ACS had no permanent or reliable place to work, a common issue in many clinics, this also hindered communication with providers and negatively impacted referrals.

Intervention characteristics

An often cited barrier to implementation within this category was ACSs who worked part-time in a clinic. Referrals are hindered when an ACS is on-site only certain days per week to receive them. Also, less "face time" in the clinic means less communication and fewer opportunities for building rapport with clinic providers and staff. Some physicians discussed communication with ACSs being unsatisfactory. Interestingly, the nature of the dissatisfaction was not uniform. In fact, some physicians wanted more frequent and extensive communication with the ACS, while others felt there was too much. A goal of the CALM intervention was to customize communications with providers to meet their requests/needs for monitoring patients, and it appears that this was not optimized during the study. In addition, some nurses reported feeling "out of the loop" between their PCPs, the ACSs, and the off-site psychiatrist, and they felt that they could not act as "champions" for the intervention when they were out of the loop. This barrier could also fall under the category of clinic structure if its genesis had more to do with pre-existing clinic culture. Some nurses also reported only hearing about the intervention from their PCP, indicating that in some clinics the nurses were not consistent targets for education and marketing about the intervention.

Patient characteristics

Some physicians noted that when patients drop out or "no show" for their ACS appointments, this can weaken their enthusiasm for the intervention. Perhaps related to this, a patient-related barrier noted among ACSs working in clinics with high numbers of uninsured was that the high overall disease burden and social stressors among lower socioeconomic status (SES) patients seemed to make it harder for those patients to engage in the intervention. The pressures of unstable employment, housing, and transportation appeared to negatively affect how some patients prioritized their anxiety treatment. Two ACSs who worked in clinics that saw many Hispanic patients noted that in general, Hispanic patients were reluctant to engage in the CALM intervention. They speculated that culturally bound beliefs about mental illness were contributing to males being resistant to admitting a problem and females not getting necessary support from their families to participate in the sessions.

Facilitators to implementation

Provider attitudes/behaviors

Implementation seemed to go smoothly and referrals to the ACS were most plentiful when physicians (especially) and nurses had enthusiastically "bought in" to the intervention. The factor most linked to strong buy-in was a belief that mental health concerns should be a priority. This is the inverse of the buy-in barrier noted above. It is unclear to what extent education and outreach efforts of the study investigators and the ACSs influenced this priority setting. When the ACSs were asked to speculate on this point, they reported that most of the enthusiastic providers were that way "from the get go," indicating that many providers were positively predisposed to a mental health intervention. Approximately half of the providers interviewed self-identified as champions of the intervention. Some ACSs made the point that a highly "bought in" nurse could act as a champion and facilitate numerous contacts and referrals, even in the case where his/her physician was not as enthusiastic. Informants from all of the stakeholder groups noted that when positive outcomes of patients were communicated, this increased provider enthusiasm and referral activity. Additionally, when reduced somatic complaints were observed in some patients, provider enthusiasm increased. In addition, among those with a favorable view of CALM, providers did not need to see dramatic improvements in their patients to maintain a positive attitude towards the intervention. As well, most provider informants said they appreciated the additional referral source.

Clinic structure

Two related facilitators in this category were pre-existing presence of a mental health provider and pre-existing presence of collaborative-care services for another disorder. If the providers were in the habit of referring patients to another mental health specialist within their clinic, or were in the habit of using a collaborative-care coordinator of some kind, those factors facilitated the use of the ACS. The large HMO clinics were the most likely to be already employing a collaborative-care coordinator for another disorder (e.g., diabetes commonly and, in one case, depression). As noted above, the in-house mental health professionals were usually master's-level clinicians. The interviews with the ACSs did not uncover much information about their relationship with these providers, though some found them to be "good referrers."

Intervention characteristics

Perhaps the most universal facilitator across clinic stakeholders was that the CALM intervention was not overly burdensome. The physicians and nurses reported little to no increased burden in their workload, and many noted reductions in their workload as a result of reduced somatic complaints. The vast majority found the referral processes very easy and that referrals worked best when the ACSs were in the clinic full time. While it was usual for providers to leave a paper-and-pencil or electronic referral note for the ACSs to follow up on, some providers liked to have the ACS join their encounter with the patient for a face-to-face "hand-off." These face-to-face hand-offs were certainly easier when the ACSs were in the clinic full-time, but the other kinds of referrals were also more quickly acted upon when the ACS was in the clinic full-time. Further, the ACSs found that the more they had "face time" with providers to discuss CALM and establish rapport, the better the implementation went. Face time could occur in the hallways, at lunchtime, or during staff meetings. From the clinic stakeholder point of view, when ACSs were seen as warm, engaging, and visible, they received high marks. And most of the informants who commented specifically about their ACS did so very positively. Most of the providers found the distribution of flyers and information about the study in the patient rooms and/or waiting rooms to be helpful, with this resulting in many patients presenting their concerns about anxiety and asking about the intervention without needing to be screened or prompted.

Patient characteristics

The main patient-level facilitator seemed to be that many expressed their preference to these informants (especially the nurses) that they prefer coming to primary care for their mental health issues. This preference seemed to be rooted in both the ease of coming there ("one-stop shopping") and that stigma was reduced by coming to primary care and seeing a mental health provider there.

Barriers to sustainability

Clinic structure

The two main barriers expressed by the vast majority of stakeholders were paying for the ACS services and space for the ACS. Most, but not all, of the stakeholders expressed doubt that the intervention would be maintained and that the main "culprit" would be the difficulty in paying for the ACS. A payor or payors would have to decide to reimburse for the services, an external source would have to provide the service for free (e.g., a university psychiatry department), or the clinic would have to pay for the employee themselves. In the latter case, the informants indicated that a strong "business case" would have to be made in favor of it and that this seemed unlikely at the present time. Further, many participants noted their difficulty in finding appropriate space for the ACS during the intervention, and they suspected that this would continue if they tried to sustain the intervention.

Facilitators to sustainability

Provider attitudes/behavior

Numerous providers and administrators reported a positive opinion about the CALM intervention and expressed that they would like to see CALM continue in their clinic. Many informants reported that, overall, the intervention increased their clinic's awareness of anxiety and that increased their desire to continue treating anxiety.

Clinic structure

The clinics that already have a mental health provider or a collaborative-care program for another disorder felt more confident that sustaining CALM was possible or even probable in a minority of cases.


Overall results and key findings

This qualitative process evaluation found, in general, that the CALM collaborative-care intervention for multiple anxiety disorders was not overly burdensome to providers and staff, and it was relatively easy to incorporate into the clinics' routine. Satisfaction with the intervention among these respondents was generally high. Primary care providers appreciated the additional referral source and the feedback they received from the ACS and psychiatrist. The majority of informants reported seeing moderate improvements in enough patients for them to find value in the intervention. A majority of informants stated, without being prompted, they would like to see the intervention continue after the clinical effectiveness study was over. We did not see much evidence of outright "resistance" to the CALM intervention, rather more a lack of motivation/buy-in from a number of providers and some not seeing it as solving key concerns of the clinics, which served more as a challenge to sustainability than uptake during the clinical trial.

There were many important facilitators to implementation, perhaps the most important being positive attitudes about the intervention among providers (buy-in). Providers who held a pre-existing belief in the importance of recognizing and treating mental health problems in primary care, who found the intervention nonburdensome, who perceived the ACS as visible and well-liked, who valued the feedback from the clinical team, and who observed positive patient outcomes (especially reduced somatic complaints) were those who most enthusiastically supported the intervention. Other facilitators were a reliable and proximate location of the ACS's workspace, having the ACS work full-time in the clinic, "face time" for ACSs to interact frequently with providers, and the perception of a relatively high prevalence of anxiety among clinic patients. Those clinics with previous experience with an on-site mental health provider and/or collaborative-care interventionist appeared to more readily implement the intervention. It is also possible that clinics with pre-existing mental health providers attracted a greater number of patients with anxiety disorders, and therefore, those clinics might have recognized a greater benefit of the intervention.

Numerous barriers to implementation were also found. First and foremost, it was clear that not all providers bought in to the intervention. Some were infrequent users of the intervention, and some never used it at all. For some, this relative lack of use appeared to be tied to their impression that anxiety prevalence was low in their clinics. For others, the lack of use appeared to be linked to a general lack of comfort with treating mental illness or a belief that mental health should not be treated in primary care. Further, these providers did not seem to respond to (or perhaps did not attend) traditional educational sessions by experts presenting evidence on anxiety prevalence and effectiveness of collaborative-care interventions. It is possible that additional marketing of the intervention might have improved provider buy-in. However, many of the providers' attitudes might have been relatively easily fixed. Future research should test the effect of various marketing strategies in primary care clinics.

In addition, it was clear that nurses in some clinics served as key advocates and sources of referral. Yet, the study did not specifically market to clinic nurses or provide educational opportunities for them. This was an important oversight. Future studies should consider active promotion of primary care interventions with clinic nursing staff. Other key barriers to implementation were having large numbers of part-time primary care providers within a clinic, using a part-time ACS, a lack of dedicated space for ACSs, unsatisfactory communication with ACSs, and engagement challenges related to low SES and Hispanic patients. Important barriers to sustainability of the intervention were the cost of the ACS (a "deal-breaking" barrier) and space concerns.

There were not many dramatic differences in reaction to the intervention across stakeholder groups or clinic types, but a few differences did emerge. Clinic administrators/managers, who were usually those charged with finding ACS work space and overseeing clinic operations, reported the most initial skepticism about the intervention and reported experiencing the most burden caused by the intervention. The burden, however, was still mild to moderate, and administrators were generally quite positive about the intervention. In terms of clinic types, the federally qualified community health centers were perhaps at some disadvantage because more of their patients were low SES, and those patients seemed to struggle more with engagement. At the same time, providers in these clinics seemed to especially appreciate the additional referral source (a free mental health intervention for their patients). Offering CBT was also a key draw for participation as some of the providers reported very little access to CBT, even when a patient was insured. Two clinics administered by a large, regional HMO and one university-affiliated clinic had experience with collaborative-care interventions, and providers in these clinics were more likely to think that sustainability would be feasible.

Findings in context of the literature

Most of the barriers and facilitators found in this study are consistent with leading conceptual models of dissemination and implementation in healthcare and the empirical literatures supporting them. For example, numerous barriers and facilitators to implementation and sustainability reported by informants had to do with provider beliefs, attitudes, motivation, and norms, and these are central determinants of implementation in the models of Mendel et al. [34], Greenhalgh et al. [21], and Damschroder et al. [36]. These "personal" determinants certainly interact with the characteristics of the innovation. Most implementation models in healthcare place a good deal of emphasis on predisposing characteristics of the innovation for implementation success. For example, Greenhalgh et al.'s [21] model, based on empirical findings, posits that an innovation has a better chance of successful adoption when it has demonstrated clinical advantage (via research evidence), compatibility with existing practices, observability (of results), low complexity, and potential for local tailoring. These same general attributes are also reflected in the diffusion and implementation models of Rogers [37] and Damschroder et al. [36]. Innovation characteristics interact not only with provider/staff-level attributes but also with clinic-level attributes such as culture (norms and practices of the system) and climate (worker's perceptions of, and reaction to, the characteristics of the work environment) [38]. We found that if the clinic culture had previous experience with collaborative care and/or making internal referrals for mental health conditions, this promoted both successful implementation and increased the perception of sustainability. This study also identified barriers and facilitators associated with how the intervention was supported "on the ground" by study and clinic staff. Central to the Promoting Action on ResearchImplementation in Health Services (PARIHS) implementation framework is the notion of facilitation, defined as "making things easier for others" [39]. This model suggests that implementation success is maximized when there are coordinated efforts to encourage participation, promote action, create supportive systems, and monitor and feedback progress. This study suggests that clinics in which ACSs operated not only as clinicians but also as facilitators in conjunction with local champions were more successful. Lastly, this study found the major barrier to sustainability to be financial (i.e., paying for the key interventionist). Most implementation models recognize and emphasize the cost and payor factor, along with others making up the "outer context" of implementation (e.g., the socio-political climate, mandates, and incentives) [21, 34, 36, 40].

It was apparent from the analyses that at least two "meta-themes" emerged-the first being associated with communication. Many of the barriers and facilitators we found appear related to positive communication among the ACS, primary care staff, and study-provided psychiatrist, for example, satisfaction with communication with the ACS, nurses being in or out of "the loop," positive clinical outcomes being communicated to the providers, ACSs being part-time or full-time, face-to-face hand-offs of patients, and space (proximity to ACSs). These communication issues are certainly linked to rapport and trust, and together they form a necessary foundation for a working collaborative relationship. It is clear from these analyses that for a collaborative-care intervention to be successfully implemented, there need to be positive lines of communication created that foster rapport and trust. The second meta-theme relates to the lack of anxiety-specific barriers and facilitators observed. Most of the barriers and facilitators were similar to those noted for depression collaborative care and perhaps other collaborative-care interventions in other non-mental health conditions [15, 41]. The one anxiety-specific barrier mentioned was a perceived low prevalence of anxiety in some clinics. This could mean that interventions such as CALM for anxiety disorders would be amenable to being combined with other collaborative-care interventions, at least for mental health conditions, as seems to be the current thrust in some health systems [42]. Such combinations could be more cost effective and improve the "business case." The CALM intervention already combined multiple anxiety disorders and had a companion module for comorbid depression.

Implications and future directions

The next step in the research agenda for CALM investigators will be to develop and test an implementation strategy to facilitate adoption under more naturalistic circumstances, that is, without research funding covering the ACS salaries and other cost offsets. This research provides important data to inform the development of such an implementation strategy. The data presented here indicate that the implementation strategy should, at a minimum, include the following elements:

  1. 1)

    A marketing plan designed to engage those who are not predisposed to identifying and treating mental health problems in primary care [15] and includes physicians, nurses, and administrators.

  2. 2)

    Encouragement of the clinic participants to tailor the intervention to their needs [43], with special attention paid to the process of communication between providers and the collaborative-care team (ACS and psychiatrist) and space for the ACSs. Roles and responsibilities of champions and opinion leaders should also be made explicit [27, 44], with some consideration given to training champions and/or opinion leaders in evidence-based facilitation strategies [44].

  3. 3)

    A process for delivering "easily digestible" outcomes data elements to providers and staff to reinforce uptake [45].

  4. 4)

    Additional features to support engagement of lower SES patients Hispanic patients.

  5. 5)

    A focus on financing. Bachman and colleagues [46] have outlined an array of possible funding mechanisms for depression care management that also apply to CALM, for example, practice-based care management on a fee-for-service basis, practice-based care management under contract to health plans, group-model HMO internal funding, and third-party-based care management under contract to health plans. Any implementation strategy would need to include efforts at securing "coverage" of the ACS.

Study limitations and strengths

The study has several limitations that deserve mention. As noted above, this study investigated barriers and facilitators to the implementation of the CALM intervention during a clinical effectiveness trial. The implications derived here, while contributing to accumulating findings on collaborative-care interventions, still need to be tested in future implementation trials and observational studies of real-world adoption in order to be generalizable. Further, the timings of the clinic provider and staff interventions were not uniform. They occurred during the final year of a two-year intervention. Therefore, "exposure" to the intervention and its outcomes varied considerably, and this difference could have impacted the results. Perhaps most importantly, the sampling strategy likely introduced a positive bias in that most of the participants were at least moderate users of CALM, and such users were more likely to be positively predisposed to the intervention. The investigators decided that the majority of respondents should have at least moderate experience with the intervention to be able to provide the richest feedback. To counteract this bias, we did interview providers and staff with less or no direct involvement with the CALM intervention. Also, the ACS interviews asked specific questions about provider and staff buy-in from the perspective of the clinic as a whole, which also contributed balance.

The study has several strengths that also deserve mention. Very few clinical effectiveness studies devote the time and resources to an implementation-focused process evaluation. By doing so, the investigators were able to gather useful data to help them plan for future implementation efforts and reduce the time between development of research findings and adoption in routine care settings. Further, the process evaluation was large in scope, encompassing 61 key informants from multiple perspectives across 17 clinics. The use of semistructured interview guides contributed consistency and reliability to the data collection but did not limit the flow of conversation or discovery of themes. Also, "saturation" was reached, indicating that the number of interviews per stakeholder group was sufficient to fully explore the phenomena under study.


This study demonstrated that the CALM intervention is implementable in community settings within a clinical research context and supported by appropriate facilitation provided by research investigators and staff. Future research will develop and test implementation interventions.


  1. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE: Lifetime prevalence and age-of-onset distributions of DSM-IV disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005, 62: 593-768. 10.1001/archpsyc.62.6.593.

    Article  PubMed  Google Scholar 

  2. Kessler RC, Chui WT, Demler O, Walters EE: Prevalence, severity, and comorbidity of 12-month dsm-iv disorders in the National Comorbidity Survey Replication. Arch Gen Psychiatry. 2005, 62: 617-709. 10.1001/archpsyc.62.6.617.

    Article  PubMed  PubMed Central  Google Scholar 

  3. Kroenke K, Sptzer RL, Williams JB, Monahan POLowe B: Anxiety disorders in primary care: prevalence, impairment, comorbidity, and detection. Ann Intern Med. 2007, 146: 317-325.

    Article  PubMed  Google Scholar 

  4. Cramer V, Torgersen S, Kringlen E: Quality of life and anxiety disorders: a population study. J Nerv Ment Dis. 2005, 193: 196-202. 10.1097/01.nmd.0000154836.22687.13.

    Article  PubMed  Google Scholar 

  5. Sherbourne CD, Sullivan G, Craske M, Roy-Byrne P, Golinelli D, Rose R, Chavira D, Bystritsky A, Stein M: Functioning and disability levels in primary care out-patients with one or more anxiety disorders. Psychol Med. 2010, doi:10.1017/S0033291710000176. Published Online by Cambridge University Press

    Google Scholar 

  6. Strine TW, Mokdad AH, Balluz LS, Gonzalez O, Crider R, Berry JT, Kroenke K: Depression and anxiety in the United States: finding from the 2006 behavioral risk factor surveillance system. Psychiatr Serv. 2008, 59: 1383-1390. 10.1176/

    Article  PubMed  Google Scholar 

  7. Kessler RC, Keller MMB, Wittchen HU: The epidemiology of generalized anxiety disorder. Psychiatr Clin of North Am. 2001, 1: 19-39.

    Article  Google Scholar 

  8. Greenberg PE, Sisitsky T, Kessler RC, Finkelstein SN, Berndt ER, Davidson JR, Ballenger JC, Fyer AJ: The economic burden of anxiety disorders in the 1990s. J Clin Psychiatry. 1999, 60: 427-435. 10.4088/JCP.v60n0702.

    CAS  Article  PubMed  Google Scholar 

  9. Wang PS, Lane M, Olfson M, Pincus HA, Wells KB, Kessler RC: Twelve- month use of mental health services in the United States: results from the National Comorbidity Survey Republication. Arch Gen Psychol. 2005, 62: 629-640. 10.1001/archpsyc.62.6.629.

    Article  Google Scholar 

  10. National Institute for Clinical Excellence: Anxiety: management of anxiety in adults in primary, secondary, and community care. Clinical Guideline 22. 2005, London: National Institute for Clinical Excellence

    Google Scholar 

  11. Terluin B, va Heest FB, van der Meer K: Guideline Anxiety Disorders Dutch College of General Practitioners (first version). Huisarts en Wetenschap. 2004, 47: 26-37.

    Article  Google Scholar 

  12. Roy-Byrne PP, Russo J, Dugdale DC, Lessler D, Cowley DS, Katon W: Undertreatment of Panic Disorder in Primary Care: role of patient and physician characteristics. J Am Board of Family Practice. 2002, 15: 443-450.

    Google Scholar 

  13. Simon G: Collaborative care for mood disorders. Curr Opin Psychiatry. 2008, 22: 37-41.

    Article  Google Scholar 

  14. Gillbody S, Bower P, Fletcher J, Richards D, Sutton AJ: Collaborative Care for depression: a cumulative meta-analysis and review of longer-term outcomes. Arch Intern Med. 2006, 166: 2314-2321. 10.1001/archinte.166.21.2314.

    Article  Google Scholar 

  15. Smith JL, Williams JW, Owen RR, Rubenstein LV, Chaney E: Developing a national dissemination plan for collaborative care for depression: QUERI series. Implement Sci. 2008, 3: 59-10.1186/1748-5908-3-59.

    Article  PubMed  PubMed Central  Google Scholar 

  16. Roy-Byrne PP, Craske MG, Stein MB, Sullivan G, Bystritsky A, Kation W, Golinelli D, Sherbourne CD: A randomized effectiveness trial of cognitive-behavioral therapy and medication for primary care panic disorder. Arch Gen Psychiatry. 2005, 62: 290-298. 10.1001/archpsyc.62.3.290.

    Article  PubMed  PubMed Central  Google Scholar 

  17. Roy-Byrne PP, Katon W, Cowley DS, Russo J: A randomized effectiveness trial of collaborative care for patients with panic disorder. Arch Gen Psychiatry. 2001, 58: 869-876. 10.1001/archpsyc.58.9.869.

    CAS  Article  PubMed  Google Scholar 

  18. Rollman BL, Herbeck BB, Reynolds CF, Schulberg HC, Shear MK: A contemporary protocol to assist primary care physicians in the treatment of panic and generalized anxiety disorders. Gen Hosp Psychiatry. 2003, 25: 74-82. 10.1016/S0163-8343(03)00004-5.

    Article  PubMed  Google Scholar 

  19. Sullivan G, Craske MG, Sherbourne C, Edlund MJ, Rose RD, Golinelli D, Chavira DA, Bystritsky A, Stein MB, Roy-Byrne PP: Design of the coordinated anxiety learning and management (CALM) study: innovations in collaborative care for anxiety disorders. Gen Hosp Psychiatry. 2007, 29: 370-387.

    Article  Google Scholar 

  20. Roy-Byrne PP, Craske MG, Sullivan G, Rose RD, Edlund MJ, Lang AI, Bystritsky A, Welch SS, Chavira DA, Golinelli D, Campbell-Sills L, Sherbourne CD, Stein MB: Delivery of evidence-based treatment for multiple anxiety disorders in primary care: a randomized controlled trial. JAMA. 2010, 303: 1921-1928. 10.1001/jama.2010.608.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  21. Greenhalgh T, Robert G, Macfarlane F, Bate P, Kyriakidou O: Diffusion of innovations on service organizations: systematic review and recommendations. Milbank Q. 2004, 82: 581-629. 10.1111/j.0887-378X.2004.00325.x.

    Article  PubMed  PubMed Central  Google Scholar 

  22. Fixsen DL, Naoom SF, Blasé KA, Friedman RM, Wallace F: Implementation Research: A Synthesis of the Literature. The National Implementation Research Network (FMHI Publication #23). 2005, University of South Florida, Louis de la Parta Florida Mental Health Institute

    Google Scholar 

  23. Luck J, Hagigi F, Parker LE, Yano EM, Rubenstein LV, Kirchner JE: A social marketing approach to implementing evidence-based practice in VHA QUERI: the TIDES depression collaborative care model. Implement Sci. 2009, 4: 64-10.1186/1748-5908-4-64.

    Article  PubMed  PubMed Central  Google Scholar 

  24. De Jong J, Van Steenbergen-Weijenburg KM, Huijbregts KM, Vlasveld MC, Van Marwijk HW, Beekman AT, Van Der Feltz-Cornelis CM: The depression initiative. Description of a collaborative care model for depression and of the factors influencing its implementation in the primary care setting in the Netherlands. Int J Inter Care. 2009, 9: e81-

    Google Scholar 

  25. Palinkas LA, Schoenwald SK, Hoaqwood K, Landsverk J, Chorpita BF, Weisz JR: An ethnographic study of implementation of evidence-based treatments in child mental health: first steps. Psychiatr Serv. 2008, 59: 738-746. 10.1176/

    Article  PubMed  Google Scholar 

  26. Brunette MF, Asher D, Whitley R, Lutz WJ, Weider BL, Jones AM, McHugo GJ: Implementation of integrated dual disorders treatment: a qualitative Ananlysis of facilitators and barriers. Psychiatr Serv. 2008, 59: 989-995. 10.1176/

    Article  PubMed  Google Scholar 

  27. Curran GM, Thrush CR, Smith JL, Ritchie M, Owen RR, Chadwick D: Implementing research findings into practice using clinical opinion leaders: barriers and lessons learned. Jt Comm J Qual Patient Saf. 2006, 31: 700-707.

    Google Scholar 

  28. Stetler B, Legro MW, Wallace CM, Bowman C, Guihan M, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL: The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006, 2: S1-S8.

    Article  Google Scholar 

  29. Tai B, Straus MM, Lui D, Sparenborg S, Jackson R, McCarty D: The first decade of the National Drug Abuse Treatment Clinical Trials Network: bridging the gap between research and practice to improve drug abuse treatment. J Subst Abuse Treat. 2010, 38 (Suppl 1): S4-S13.

    Article  PubMed  PubMed Central  Google Scholar 

  30. Martino S, Brigham GS, Higgins C, Gallon S, Freese TE, Albright LM, Hulsey E, Krom L, Storti SA, Perl H, Nugent CD, Pintello D, Condon TP: Partnerships and pathways of dissemination: The National Institute on Drug Abuse-Substance Abuse and Mental Health Services Administration: blending Initiative in the Clinical Trials Network. J Subst Abuse Treat. 2010, 38 (suppl 1): S31-S43.

    Article  PubMed  PubMed Central  Google Scholar 

  31. Weber RP: Basic Content Analysis. 1990, Beverly Hills, CA: Sage

    Book  Google Scholar 

  32. Hsieh HF, Shannon SE: Three approaches to qualitative content analysis. Qual Health Res. 2005, 15: 1277-1288. 10.1177/1049732305276687.

    Article  PubMed  Google Scholar 

  33. Bernard HR: Research methods in anthropology: qualitative and quantitative methods (3 rd ed.). 2002, Walnut Creek, CA: AltaMira Press

    Google Scholar 

  34. Mendel P, Meredith LS, Schoenbaum M, Sherbourne CD, Wells KB: Intervention in organizational and community context: a framework for building evidence on dissemination and implementation in health services research. Adm Policy Ment Health. 2008, 35: 21-37. 10.1007/s10488-007-0144-9.

    Article  PubMed  Google Scholar 

  35. Parker LE, Kirchner JE, Bonner LM, Fickel JJ, Ritchie MJ, Simons CE, Yano EM: Creating a quality-improvement dialogue: utilizing knowledge from frontline staff, managers, and experts to foster health care quality improvement. Qual Health Res. 2009, 19: 229-242.

    Article  PubMed  Google Scholar 

  36. Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC: Fostering implementation of health services research finding into practice: a consolidated framework for advancing implementation science. Implement Sci. 2009, 4: 50-10.1186/1748-5908-4-50.

    Article  PubMed  PubMed Central  Google Scholar 

  37. Rogers E: Diffusion of Innovations. 2003, New York, NY: Free Press, 5

    Google Scholar 

  38. Aarons GA, Sawitzky AC: Organizational culture and climate and mental health provider attitudes towards evidence-based practice. Psychol Serv. 2006, 3: 61-72.

    Article  PubMed  PubMed Central  Google Scholar 

  39. Kitson A, Harvey G, McCormack B: Enabling the implementation of evidence based practice: A conceptual framework. Qual Health Care. 1998, 7: 149-158. 10.1136/qshc.7.3.149.

    CAS  Article  PubMed  PubMed Central  Google Scholar 

  40. Improving patient care: The implementation of change in clinical practice. Edited by: Grol R, Wensing, M, Eccles M. 2005, Edinburg, UK: Elsevier

    Google Scholar 

  41. Bray P, Cummings DM, Wolf M, Massing MW, Reaves J: After the collaborative is over: What sustains quality improvement initiatives in primary care practices?. Jt Comm J Qual Patient Saf. 2009, 35 (10): 502-508.

    PubMed  Google Scholar 

  42. Post EP, Metzger M, Duma P, Lehmann L: Integrating mental health into primary care within the Veterans Health Administration. Fam Syst Health. 2010, 28 (2): 83-90.

    Article  PubMed  Google Scholar 

  43. Curran GM, Allee ME, Mukherjee S, Owen R: QUERI series: a process for developing implementation interventions. Implement Sci. 2008, 3: 17-10.1186/1748-5908-3-17.

    Article  PubMed  PubMed Central  Google Scholar 

  44. Borbas C, Morris N, McLaughlin B, Asinger R, Gobel F: The role of clinical opinion leaders in guideline implementation and quality improvement. Chest. 2000, 118: 24S-32S. 10.1378/chest.118.2_suppl.24S.

    CAS  Article  PubMed  Google Scholar 

  45. Fortney J, Pyne J, Smith J, Curran G, Otero J, Enderle M, McDougall S: Steps for implementing collaborative care programs for depression. Popul Health Manag. 2009, 12: 69-79. 10.1089/pop.2008.0023.

    Article  PubMed  Google Scholar 

  46. Bachman J, Pincus HA, Houtsinger JK, Unutzer J: Funding mechanisms for depression care management: opportunities and challenges. Gen Hosp Psychiatry. 2006, 28: 278-288. 10.1016/j.genhosppsych.2006.03.006.

    Article  PubMed  Google Scholar 

Download references


This work was supported by grant K01 DA15102 (Curran) from the National Institute on Drug Abuse and by U01 MH070022 (Sullivan), U01 MH058915 (Craske), U01 MH070018 (Sherbourne), U01 MH057858 (Roy-Byrne), and K24 MH64122 (Stein) from the National Institute of Mental Health (NIMH). This project was also supported by the National Center for Research Resources and the National Center for Advancing Translational Sciences, National Institutes of Health, through grant UL1RR029884. Data from this manuscript were presented at the National Conference on Mental Health Services Research sponsored by NIMH, Washington, DC, July 23-24, 2009, and the Anxiety Disorders Association of America Annual Conference, Baltimore, MD, March 4-5, 2010.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Geoffrey M Curran.

Additional information

Competing interests

The authors declare that they have no competing interests.

Authors' contributions

GMC participated in the design of the study, performed data analyses, and took the lead in drafting the manuscript. GS conceived of the study with other co-authors, participated in its design and coordination, and helped to draft the manuscript. PM conceived of the study with other co-authors, participated in its design and coordination, and helped to draft the manuscript. MGC conceived of the study with other co-authors and participated in its design and coordination. MBS conceived of the study with other co-authors and participated in its design and coordination. CDS conceived of the study with other co-authors and participated in its design and coordination. AM participated in the study's design, helped coordinate the study, and performed data analyses. PRB conceived of the study with other co-authors, participated in its design and coordination, and helped to draft the manuscript. All authors read and approved the final manuscript.

Electronic supplementary material

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and Permissions

About this article

Cite this article

Curran, G.M., Sullivan, G., Mendel, P. et al. Implementation of the CALM intervention for anxiety disorders: a qualitative study. Implementation Sci 7, 14 (2012).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI:


  • Anxiety Disorder
  • Cognitive Behavioral Therapy
  • Generalize Anxiety Disorder
  • Social Anxiety Disorder
  • Panic Disorder