The organizational model for innovation implementation depicted in Figure 1 proved useful for identifying and investigating the organizational factors that facilitated and hindered start-up and early implementation of CBPPR in CCOP organizations. Generally speaking, the model’s constructs were salient in the data (Table 3) and the observed relationships among constructs fit the model. Some constructs exhibited less salience than others, as measured by the number of text units coded for that construct, either because interview questions pertinent to that construct were asked only in specific years (e.g., organizational readiness for change) or because interview participants had less to say about the issue (e.g., management support) or because the construct proved difficult to identify in the natural language responses of interview participants (e.g., IVF). Each year, we asked many questions and heard a great deal about the operational aspects of CBPPR implementation. Not surprisingly, IPP was the most salient construct in our data.
Briefly, results can be summarized as follows. The three CCOP organizations varied in the extent to which they achieved consistency in CBPPR over time and across physicians. All three CCOP organizations demonstrated mixed levels of organizational readiness for change, with strong collective confidence to meet accrual goals but low physician and hospital commitment to CBPPR. Hospital management support and resource availability were limited across CCOP organizations early on, although they improved in one CCOP organization. Through IPPs, all three CCOP organizations provided high levels of research support to some CCOP physicians, but not others. However, none created formal expectations or accountability for physicians to help the CCOP organization meet its accrual goals. Further, all three organizations used relatively weak IPPs to recognize and reward physicians for enrolling patients in clinical trials. As a result, all three created a weak implementation climate. Patient accrual became concentrated over time among those groups of physicians for whom CBPPR exhibited a strong IVF. Several external (environmental) factors also influenced innovation use, complicating and enriching our intra-organizational model of innovation implementation.
Below we describe our results in detail, beginning with implementation effectiveness.
Implementation effectiveness
As an organization-level construct, implementation effectiveness refers to the consistency and quality of aggregate innovation use among intended innovation users. We operationally defined innovation use as patient accrual, or the enrollment of new patients in NCI-sponsored clinical trials. Aggregate innovation use is measured in terms of CCOP-level patient accrual.
Figure 3 shows that CCOP A exhibited the greatest consistency over time in total patient accrual to NCI-sponsored clinical trials (i.e., treatment and CP/C accrual combined). CCOP C exhibited a substantial jump in total patient accrual in the first three years of CCOP funding, followed by an even larger decline. CCOP B exhibited a similar, but less pronounced pattern. All three CCOP organizations struggled with patient accrual in 2010 and 2011 due, in part, to external factors (described below). Viewed somewhat differently, CCOP A met the NCI’s accrual goals in eight of the nine observed years for CP/C trials and six of the nine observed years for treatment trials. CCOP C met the NCI’s accrual goals in eight of the nine observed years for CP/C trials, but only two of the nine years for treatment trials. CCOP B met the NCI’s accrual goals in two of the six observed years for CP/C trials and none of the six years for treatment trials. In sum, the consistency of aggregate innovation use was highest in CCOP A, then CCOP C, then CCOP B.
The quality of aggregate innovation use did not vary meaningfully across the three cases. There were minor audit issues reported around eligibility criteria and protocol selection in all three cases, but the data quality and research procedures were generally robust.
The three CCOP organizations also varied in the consistency with which physicians engaged in CBPPR. To examine how and why this is so, we turn to our organizational model of innovation implementation, beginning with an analysis of organizational readiness for change.
Organizational readiness for change
Success in the start-up phase of implementation depends in large part on ORC. As noted earlier, ORC concerns organizational members’ collective confidence to implement change and collective commitment to pursue courses of action that will lead to successful change. In all three CCOP organizations, collective confidence was high. Prior to becoming a CCOP, all three CCOP organizations had participated in NCI-sponsored clinical trials as research affiliates of cooperative groups. As research affiliates, they gained experience enrolling and managing patients on clinical trials and used the per-case reimbursement that they received for enrolling patients to hire research staff to support participating physicians.
Interview participants said that, when they applied for CCOP funding, they knew they would have to significantly ramp up their clinical trials research operations to meet the NCI’s accrual goals. Moreover, they acknowledged some trepidation about having to enroll patients in CP/C trials, a type of clinical research with which they had less experience. Nonetheless, they felt confident that they could quickly increase the scope and scale of their research operations:
‘We were pretty confident we’d be able to meet the treatment trials [accrual goal], if the right trials were open…. The prevention [trials accrual goal] has always been kind of an iffy in the very beginning, just because there weren’t [large prevention] trials available…. There wasn’t a SELECT [the Selenium and Vitamin E Cancer Prevention Trial], there wasn’t a STAR [Study of Tamoxifen and Raloxifene]. So we thought we’d be able to meet the treatment. We never knew about the prevention.’—CCOP B, CRA group interview.
Although collective confidence was high, collective commitment was more variable in all three CCOP organizations. Some physicians were committed and enthusiastic about becoming a CCOP organization; other physicians were ‘supportive’ or merely ‘interested.’ In retrospect, it is not clear whether physicians understood that, as a CCOP, they collectively would be ‘on the hook’ for a much higher level of accrual than they were used to generating as a research affiliate. To meet the NCI’s expectations, physicians would have to step up their personal participation in the clinical research, enrolling patients routinely rather than occasionally in clinical trials. The CCOP PIs of all three CCOP organizations succeeded in building physician support to become a CCOP organization, but the broad support they garnered did not translate into broad (or deep) commitment to enrolling more patients in clinical trials after the CCOP grant was awarded:
‘I think that they [physicians] were tolerant of it but I don’t know that they would have independently thought of it and I don’t know that they perceived that to become a CCOP was going to move us to a different level or establish a different regard for our practice in the community. But, you know, I do think that they heard me out and I think they were supportive of it when they heard the reasons I felt it was a good idea.’—CCOP B, PI
In addition, it is not clear that the CCOP PIs made the strategic or competitive case to hospital managers for becoming a CCOP organization (or, if they did make the case, that hospital managers understood it). In none of the cases did the decision to apply for CCOP funding come from hospital managers. Instead, the PIs came up with the idea to apply and wrote the CCOP proposal. In one case, the PI waited until a ‘friendly’ but interim hospital manager was in place before asking the hospital to sign-off on the CCOP proposal. While the PI received the sign-off, the stage was set for low levels of management support once the program was funded.
In all three cases, the minimal involvement of hospital management in the decision to apply for CCOP funding set the stage for minimal management support once the grant was awarded. In all three cases, managers viewed the CCOP as a ‘doctor’s program,’ with potential benefits or advantages for the hospital either misunderstood or unrealized:
‘For many years here, they weren’t supportive. Then they became supportive when they thought that it was [good] for the hospital’s existence. So, hospital administrators will not support programs that are for research, and maybe that was my wrong approach because I told them this was a research project. In retrospect, what I should have done is said this is not a research program, this is a quality of care program.’—CCOP C, PI
In sum, the three CCOP PIs overestimated collective confidence and underestimated collective commitment. Although all three CCOP organizations met NCI’s accrual goals right away, the mixed organizational readiness for change exhibited in the pre-implementation phase foreshadowed the problems that all three cases encountered later in implementation.
Management support and resource availability
Management support matters because implementation is resource intensive. Managers can support CBPPR by allocating needed financial, human, and material resources. In addition, they can shape the implementation context by communicating that CBPPR is an organizational priority, encouraging organizational members to implement policies and practices to support CBPPR. Finally, managers can support implementation by overcoming resistance to change and resolving disputes over resource allocations, organizational routines, and lines of authority.
Given the minimal involvement of hospital managers in the decision to become a CCOP organization, management support during implementation was largely symbolic. The managers we interviewed expressed verbal support for the CCOP organization, but did not see CBPPR as important in advancing the hospital’s strategic goals or improving its competitive position:
‘When I attract physicians to the organization, there’s eight or nine things that come to the top and this [the CCOP] would probably be one of I’d say five or six, as opposed to the first five. So, I would tell you it’s not in my top three. I would say that when I’m talking about attracting doctors to the organization, I’m generally talking more at the global level.’—CCOP B, hospital manager.
In two cases, hospital managers made no financial commitment to CBPPR over the course of our study, despite the fact that NCI funds CCOP organizations with expectations that institutional cost-sharing will occur. Instead, managers expected the CCOP organization to ‘live on its CCOP grant’ and any supplemental funding that physicians generated from participating in pharmaceutical industry trials:
‘Basically, it amounts to the CCOP grant, we get a little bit of money from the cooperative groups and we get the money from industry trials and the hospital helps in the sense that they own the foundation that gives us some philanthropic support that helps us fill the space between what we can generate and what our true costs are…they’re [the hospital] kicking in zero dollars for every dollar that we generate.’—CCOP A, PI
In the third case, hospital managers realized that CBPPR had strategic value in light of the hospital’s plans to become a major teaching affiliate of a newly opened medical school. Three years into implementation, hospital managers began making substantial financial commitments to the CCOP organization, starting at one million dollars and, later, as the Great Recession continued, reducing the amount to $800,000:
‘I think it’s evolved through the years, but I think research has taken a greater importance…as that has happened people have seen, well, you have the CCOP, it’s been the gem, it’s been sitting there all these years and it’s just kind of like rediscovered every year. But now, like I said, with the medical school, the importance of research is clearer.’—CCOP C, physician
Although management support increased over time in one case, the modest management support that existed in all three CCOP organizations during start-up and early implementation was, in interview participants’ minds, both a blessing and a curse. On the one hand, the CCOP organization could ‘fly under the radar’ with little management interference. Moreover, it was largely shielded from the staffing reductions and budget cuts that the hospital imposed elsewhere as the ‘Great Recession’ persisted. On the other hand, the CCOP organization could not expand its operations beyond what the CCOP grant could sustain. Further, the CCOP organization could not use hospital managers’ authority and resources to implement policies and practices that would make physicians accountable for helping the CCOP organization meet its accrual goals.
Implementation policies and practices
Organizations can make use of a wide variety of IPPs to enhance organizational members’ means, motives, and opportunities for innovation use. IPPs are cumulative, compensatory, and equifinal[15, 17]. That is to say, all other things being equal, the more IPPs an organization employs to support innovation use the better. However, high-quality IPPs can sometimes compensate for missing or poor-quality IPPSs. For example, excellent in-person training could substitute for mediocre program manuals. Finally, different combinations of IPPs can produce the same results.
Our analysis identified three IPP themes. First, all three CCOP organizations provided a high level of staffing and operational support for some, but not all, of the physicians who wanted to enroll patients in NCI-sponsored clinical trials. Although the specific IPPs used by the three CCOP organizations differed somewhat (e.g., some used a mix of registered nurses and clinical research associates while others used only registered nurses to staff the clinical trials program), all three provided support by screening charts to identify potentially eligible patients, flagging charts to prompt physicians to introduce trials to eligible patients, educating and consenting interested patients, following patients once enrolled in a study, and managing IRB and other regulatory requirements. However, they could only provide such high levels of support to those physician practices located ‘on-site’ or in close proximity to the hospitals where research staff worked. Support for ‘off-site’ physician practices was harder to provide or sustain due to travel time and logistical difficulties. The limited support CCOP organizations could offer these physicians contributed to uneven distribution of patient accrual.
Second, all three CCOP organizations avoided using IPPs to create formal expectations or accountability for physicians to help the CCOP organization meet its accrual goals. The CCOP PIs and CCOP administrators routinely cajoled, sometimes exhorted, and occasionally begged physicians to enroll more patients in clinical trials. They regularly provided physicians with feedback on the CCOP organization’s accrual performance relative to NCI’s accrual goals and usually provided physician-level patient accrual figures. Yet, none were willing to put into place a policy specifying the minimum number of patients per year that a physician had to enroll in order to maintain membership in the CCOP organization and, thereby, enjoy the reputational benefit of having an ‘NCI affiliation’:
‘This institution will not tell the referring doctors that they can’t be members of the CCOP for some criteria. You’ve got to put, let’s say, ten patients on a year. In fact, the institution is going to the opposite direction.’—CCOP C, PI
Moreover, none were willing to confront physicians who maintained their membership in the CCOP organization, yet consistently contributed no patient accrual. Interview participants mentioned several reasons why they were unwilling or unable to implement such policies and practices, including not wanting to shut anyone out who might at some point contribute accrual; not wanting to exclude off-site physicians who received little or no support from the CCOP organization; and not wanting to confront colleagues who, in some cases, were business partners:
‘No, the group has to provide support for the clinical research program but there’s no individual expectation that every physician will accrue ten patients per year to clinical trials. We keep trying to bang that drum and we provide feedback to our doctors on a regular basis about how they are doing, but I can’t tell that they have the requisite shame that I would hope that they would have with some of the numbers.’—CCOP B, PI
CCOP organizations sought to compensate for these missing policies and practices by providing more research support, but with little success.
Finally, all three CCOP organizations used relatively weak IPPs to recognize and reward physicians for enrolling patients in clinical trials. Financial incentives or rewards were out of the question due to ethical, legal, and regulatory concerns. So, CCOP organizations fell back on social recognition and non-monetary rewards, neither of which remained effective over time. Social recognition included verbal praise in meetings, certificates of appreciation, complimentary write-ups in newsletters, and token awards given to the highest-accruing physician. Non-monetary rewards included paid travel to conferences, batches of cookies, and other small gifts. The motivating effects of these rewards diminished over time:
‘I think that we’ve tried all the rewards that probably many other research organizations have tried. You know, we’ve given, boxes of candy and we’ve given gift certificates and cookies…we found our doctors are not going to enter patients on clinical trials for the benefit of winning the chocolate chip cookies or winning a gift certificate.’—CCOP B, PI
In sum, all CCOP organizations employed IPPs unevenly. Although some IPPs can substitute for other IPPs, the challenges that the CCOP organizations experienced providing support to off-site physicians and the minimal or low-power IPPs that they used create expectations, accountability, recognition, and rewards produced a weak implementation climate.
Implementation climate
IPPs influence innovation use through implementation climate. Organizational members ascribe meaning to the IPPs that they experience directly or vicariously. Through experience, observation, and discussion, they develop a shared sense of whether innovation use is expected, supported, and rewarded. This shared sense is important when the organizational benefits of innovation use depend on consistent, high-quality use by many organizational members. Such is the case with CBPPR in CCOP organizations. The NCI evaluates and funds CCOP organizations based on aggregate (i.e., CCOP-level) accrual.
Given the uneven IPPs that CCOP organizations put into place, implementation climate was weak in all three cases. None of the interview participants reported that they were expected either by the CCOP organization or by hospital management to enroll any particular number of patients in clinical trials each year to maintain membership in the CCOP organization. Likewise, none reported that they were recognized or rewarded in a meaningful way for helping the CCOP organization meet its accrual goals. Physicians practicing in or near the hospital felt supported by the CCOP to enroll patients in clinical trials; off-site physicians did not.
Although implementation climate was weak in all three CCOP organizations, all three nonetheless met or exceeded NCI’s accrual goals. To understand how the CCOP organizations achieved this feat, we must examine the pattern of IVF among physician groups.
Innovation-values fit
As noted above, IVF refers to the extent to which organizational members perceive that innovation use will foster the fulfillment of their values. Values are concepts or beliefs that pertain to desirable end states or behaviors. Although individuals can hold different values, so too can groups. Different physician groups, for example, may hold different beliefs about clinical trials participation. Individuals and groups can also differ in the amount of feeling they attach to a concept or a belief. Clinical trials participation might be valued strongly and considered a priority, or it might be valued weakly and considered desirable, but not necessary.
When implementation climate is weak, innovation use depends primarily on IVF. Individuals and groups that strongly value CBPPR will enroll patients in clinical trials even if they perceive little expectation or extrinsic reward for doing so, provided adequate support exists. If inadequate support exists, ‘true believers’ will find it difficult to enroll patients on a consistent basis, leading perhaps to frustration and disappointment. By contrast, individuals and groups that do not strongly value CBPPR will enroll few patients, if any, because they see little reason to do so when enrolling patients is neither expected, nor supported, nor rewarded.
We observed individual differences in IVF in all three cases. Not surprisingly, the CCOP PIs highly valued CBPPR and enrolled many patients. In one CCOP organization, a physician enrolled many patients in pharmaceutical industry trials but relatively few in NCI-sponsored clinical trials. Industry trials exhibited a greater fit with his high-intensity values because, in his view, they gave him access to newer, more exciting drugs. In another CCOP organization, a physician stated that he did not like the NCI’s treatment trials because they were not, in his view, always addressing important scientific questions; however, he was willing to help the CCOP organization achieve its accrual goals by enrolling patients in cancer control trials.
We also observed group-level differences in IVF in all three cases. Medical oncologists as a group more strongly valued CBPPR, and therefore enrolled more patients in NCI-sponsored clinical trials, than did radiation oncologists or other physician groups. Interview participants in two CCOP organizations noted that radiation oncologists as a group were ‘supportive,’ but not ‘committed’ to enrolling patients in NCI-sponsored clinical trials. The radiation oncologists would follow the clinical trial protocol for those patients that the medical oncologists enrolled, but they would not enroll patients in radiation-therapy clinical trials in sufficient numbers to help the CCOP organization meet its accrual goals:
‘The total again is 8% of the patients. So that means they put eight, nine, ten patients on this whole year…I’ve been here for 20 years and this is pretty typical for the way the radiation therapists have performed over the last 20 years. This is nothing different. They just don’t do a lot of randomized, you know, Phase III trials, which is what we do as a CCOP. They do a lot of Phase II trials.’—CCOP C, PI
Even when provided with additional research staff support, radiation oncologists as a group would not contribute much on a consistent basis. CBPPR was simply not a high-intensity value for them, interview participants said.
In a third case, radiation oncologists as a group strongly valued CBPPR, as evidenced by their active in-house research program. However, they preferred to enroll patients in in-house studies rather than NCI-sponsored clinical trials because such studies were more instrumental in fulfilling their values (i.e., they felt such studies were more interesting and easier to do).
Given weak implementation climate and variable IVF, all three CCOP organizations exhibited inconsistent, uneven distribution of patient accrual among individuals and groups of physicians. Of the three cases, CCOP A exhibited the strongest IVF among medical oncologists as a group. This CCOP organization recruited and hired physicians who valued CBPPR, and it reinforced CBPPR as a group value in a variety of formal and informal ways. Reflecting a more distributed (or even) pattern of patient accrual, the PI of CCOP A enrolled on average only 28% of the patients per year that CCOP A accrued to NCI-sponsored clinical trials from 2002 to 2010. By comparison, accrual was more concentrated at CCOP C and CCOP B. In the former case, the average annual accrual contribution of the PI was 34%, while in the latter case it was 44%. At start-up, CCOP C exhibited more evenly distributed accrual among physicians, with the CCOP PI accounting for only 18% of patient accrual in the first year. With off-site physicians receiving little support, and radiation oncologists focused on in-house studies, initial enthusiasm faded and accrual become increasingly concentrated among the few ‘true believers.’ By 2008, six years after start-up, the CCOP PI’s contribution accounted for over 42% of total patient accrual. In CCOP B, the concentration of patient accrual was high at start-up and increased steadily over time. In the first year, the PI’s contribution accounted for 39% of total patient accrual. By 2009, it amounted to 59%, despite personal appeals from the highly regarded PI and increased research support for low- or non-accruing physicians. An interview participant described the situation:
‘And our accruals here are actually up but once again, we’re really concerned that [the CCOP PI] has [put] 40 some odd patients on treatment trials. Everyone else in the CCOP is at single digits. And we don’t know, we’re getting ready to have a discussion with the other physicians and [the PI] and I are mulling things over on how to present this but this can’t go on. It just can’t go on. If you want us to maintain ourselves as a viable CCOP, everybody’s got to pull their weight.’ –CCOP B, CCOP Administrator
External factors
Although the organizational model of innovation implementation used in this study focused our attention on intra-organizational facilitators and barriers, we observed that CCOP organizations are heavily dependent on two external resources needed for CBPPR—trials and patients. In varying degrees, all three CCOP organizations experienced difficulties obtaining these resources, which, in turn, exacerbated the challenges posed by a weak implementation climate and variable IVF.
Interview participants noted that the menu of NCI-sponsored clinical trials available to CCOP organizations has shrunk in the past two years and contains significant gaps in common disease areas, such as colon cancer. Interview participants attributed the trial availability problem to the consolidation of the cooperative group system that is currently taking place in response to an influential, yet critical report by the Institute of Medicine[25], a not-for-profit, non-governmental organization that advises the federal government on issues related to biomedical science, medicine, and health. With fewer treatment trials available, CCOP organizations are struggling to meet their treatment trial accrual goals. In addition, with no large cancer prevention trials available, CCOP organizations are trying to meet their CP/C accrual goals through symptom management trials. These smaller trials often close quickly to new patient accrual, disadvantaging those CCOP organizations (like CCOP C) that work with slow IRBs, which must approve studies locally before CCOP organizations can enroll patients in them.
Interview participants further noted that NCI-sponsored clinical trials are increasingly testing targeted therapies applicable only to small subgroups of cancer patients. Not long ago, breast cancer treatment trials would open to almost any patient with, say, advanced disease and no co-morbidities. Today, trials are testing different therapies for node positive and negative breast cancer, which can be further divided into pre- and post- menopausal categories, which can be further sub-divided into other categories, creating approximately 16 different groups of breast cancer patients. Similar trends are occurring in treatment trials for other cancers. Interview participants referred to this as a ‘needle in a haystack’ problem:
‘What’s happening is [that] what used to be gigantic baskets of patients are [now] being divided up by biology. It’s better medicine to divide them…. Each one of these breast cancers is a bit different, [and] each one of them has a different [treatment] protocol. So now…I just cut breast cancer into 16 different groups…. And I think that accounts for a large degree why we don’t really have any rainmaker studies.’—CCOP A, PI
To meet their accrual goals, some interview participants observed, they cannot afford to miss any opportunity to enroll an eligible patient. Hence, they feel frustrated when low- or non-accruing physicians let an eligible patient go un-approached.
Finally, interview participants commented that meeting the NCI’s expectations for patient accrual became increasingly difficult in the face of persistent economic recession and increased market competition. Midwestern states, where the three COCP organizations operate, have been hit especially hard. Patient volume has declined across the board as patients put off medical care to manage other living expenses. Interview participants noted that patients have voiced more concern about enrolling in a clinical trial because they are worried about their health insurance coverage and out-of-pocket costs. Given job insecurity, patients are more reluctant to take time off work to receive multiple rounds of cancer treatment and return for follow-up clinical visits. At the same time, local market competition has grown. The PI of CCOP A noted, for example, that he and his four partners were the only medical oncologists in town in the first year of the CCOP grant. Now, he has eight partners and there are two competing oncology practices not affiliated with the CCOP organization that have seven more medical oncologists. The number of cancer patients has not increased threefold over time.