Skip to main content

Table 1 Implementation and Improvement Science Proposal Evaluation Criteria

From: Standardizing an approach to the evaluation of implementation science proposals

Criteria

Score

0

1

2

3

The care or quality gap

• No care/quality gap is defined; an issue may be presented, but it is not described as a gap in quality or care

• No information or lack of clarity in the information cited about the potential for improvement or the impact of the proposed implementation and/or improvement science study

• Proposed implementation and/or improvement science study is not linked to a safety net setting

• Unclearly defined care/quality gap is poorly supported with inappropriate/inadequate/irrelevant local setting data (i.e., evidence of chart review or other preliminary data) or citations from the literature

• Insufficient information about the potential for improvement or the impact of the proposed implementation and/or improvement science study

• Proposed implementation and/or improvement science study does not explicitly link to a safety net setting

• Defined care/quality gap is supported by either local setting data (i.e., evidence of chart review or other preliminary data) or citations from the literature

• Adequate information about the potential for improvement, but would benefit from further specification

• Proposed implementation and/or improvement science study links to a safety net setting but may need further clarification

• Clearly defined quality gap is supported by local setting data (i.e., evidence of chart review or other preliminary data) and appropriate citations from the literature

• Explicit, well thought out description of the potential for improvement

• Proposed implementation and/or improvement study is clearly linked to a safety net setting

The evidence-based treatment to be implemented

• No evidence-based or evidence-informed intervention is identified or the intervention justification/background is based on zero/inappropriate/inadequate citations from literature

• Lack of clarity about why the intervention was chosen for the study setting

• Unclear what effect the intervention will have on the selected safety net setting (or study is not based in safety net setting)

• Some literature is cited to provide evidence of limited prior efficacy studies concerning the planned intervention, meeting “evidence-informed” rather than “evidence-based” criteria

• Limited justification about why the intervention was chosen for the study setting and/or justification is based on desire to document efficacy of the proposed evidence-informed practice

• Insufficient information describing what effect the intervention will have on the selected safety net setting

• Sufficient literature is cited demonstrating evidence of prior efficacy studies using the intervention to meet either “evidence-informed” or “evidence-based” criteria

• If the intervention is “evidence-informed,” the innovative use of said intervention in the study setting is compelling enough to consider, or there is appropriate justification about why the evidence-based intervention was chosen for the study setting, and the goal is not based on developing efficacy of the said evidence-informed practice

• Adequate information describing what effect the intervention will have on the selected safety net setting, but may need further clarification

• Clearly discusses evidence of prior efficacy studies concerning the planned intervention, meeting “evidence-based” rather than “evidence-informed” criteria

• Explicit, well thought-out rationale for implementing the intervention in the selected safety net setting including the potential effect it will have on that setting

Conceptual model and theoretical justification

• No conceptual model, framework, or other theoretical grounding is discussed

• Some conceptual model is cited but its basis and constructs are irrelevant to study objectives and/or the study setting

• A conceptual model, framework, or other theoretical grounding is mentioned, but not linked to the study objectives, hypotheses, and measures

• The chosen conceptual model, framework, or other theoretical grounding may be appropriate for the intervention, but the rationale is not clearly supported with citations from the literature

• A conceptual model, framework, or other theoretical grounding is linked in some capacity to the study objectives, hypotheses, and measures, but may need additional clarification

• The chosen conceptual model, framework, or other theoretical grounding is appropriate for the intervention /implementation strategies as evidenced by a well-defined rationale with adequate citationsfrom the literature, but would still benefit from further specificity

• An implementation and/or improvement science-specific conceptual model or framework is clearly described, with theoretical constructions explicitly described within the proposed setting, population, and intervention contexts

• The implementation and/or improvement science-specific conceptual model or framework is used to frame the proposed study in all aspects including the study questions, aims/objectives, hypotheses, process, and outcome measures

• Some discussion may refer and describe how study findings would build upon or otherwise contribute to theory or the larger implementation and/or improvement science fields

Stakeholder priorities, engagement in change

• Zero or extremely limited description of who the stakeholders are or what their preferences and priorities are around the proposed intervention

• No evidence of stakeholder analysis planning, or basic information gathering is discussed in relation to how the applicant developed the implementation strategies

• Limited description of who the stakeholders are, with some key players missing from consideration

• Limited understanding of stakeholder priorities and concerns related to the intervention is demonstrated by easily identified potential issues that are not discussed in the application, or no evidence of stakeholder analysis planning is discussed

• Zero or very limited mention of involving stakeholders in the conceptual design of the intervention, and/or consideration of the implementation strategies, process, or outcomes

• No clear agreement or collaboration between the stakeholders and the applicant is explained

• Sufficient description of who all of the identifiable stakeholders are

• Clear understanding of stakeholder concerns related to the intervention as evidenced by a stakeholder analysis plan that describes how the applicant will collect at least some information on stakeholders interests, interrelations, influences, preferences, or/and priorities

• Somewhat unclear description of how stakeholders were involved in the conceptual design of the intervention, and/or consideration of the implementation strategies, process, or outcomes

• Some type of agreement or collaboration between the stakeholders and the applicant is explained but supporting evidence is limited

• Comprehensive description of who all of the identifiable stakeholders are

• Clear understanding of stakeholder concerns related to the intervention as evidenced by a stakeholder analysis plan that describes how the applicant will collect comprehensive information on stakeholders interests, interrelations, influences, preferences, and priorities

• Detailed description of how stakeholders were involved in the conceptual design of the intervention and in considering the implementation strategies, process, and outcomes

• An explicit agreement (such as a memorandum of understanding) or evidence of collaboration between the stakeholders and the applicant that is explained with relevance to the proposed study process and how findings will be communicated

Settings readiness to adopt new services/treatment/programs

• Zero or very limited rationale/interest for implementing the proposed intervention is discussed

• No information on the study setting’s capacity or readiness for implementation

• No information on how those in the study setting who are opposed to change will be involved with or have their concerns addressed by study processes or components

• Some description of the setting’s interest in the proposed intervention

• Incomplete or unclear description of how the setting will be assessed for capacity and/or readiness for implementation including which methods and tools will be used, or there is a limited description of organizational/political culture and potential contextual barriers or facilitators

• May include a brief discussion on how those opposed to change in the study setting will be involved with or have their concerns addressed by study processes or components

• May not include evidence of support (e.g., letters) from the study setting that address how the proposed study aligns with the organization’s priorities/policies

• Clearly describes the setting’s interest and rationale for the proposed intervention

• Clearly describes how the setting will be assessed for capacity and readiness for implementation including which methods, scales, or other tools will be used

• Thoroughly describes the potential influence of organizational/political culture, and potential contextual barriers or facilitators

• May include strategies for how those opposed to change in the study setting will be involved with or have their concerns addressed by study processes or components

• May not include evidence of support (e.g., letters) from the study setting that address how the proposed study aligns with the organization’s priorities/policies

• Explicitly describes preliminary data on the assessed organizational and political capacity and readiness for implementation (assessment completed prior to application/pilot)

• Preliminary capacity and readiness assessments were completed using a scale with established validity and reliability, or a scale that has undergone some validity and reliability testing

• May include strategies for how those opposed to change in the study setting will be involved with or have their concerns addressed by study processes or components

• Evidence of support (e.g., letters) from the study setting that address how the proposed study aligns with the organization’s priorities/policies

Implementation strategy/process

• No implementation strategies are identified

• Intervention may be incorrectly described as an implementation strategy

• Implementation strategies are not clearly distinguished from the intervention

• Unclear implementation strategies are not theoretically justified and/or do not match with the stated aims/setting/outcome measures of the proposed study

• Limited description linking the implementation strategies to the stated aims/setting/outcome measures of the proposed study with no plan for how strategies will be observed or tested

• Implementation strategies may be unrealistic given the pilot timeline and/or budget constraints

• Implementation strategies are clearly distinguished from the intervention

• Some theoretical justification of the implementation strategies

• Clearly describes how implementation strategies link to the stated aims/setting/outcome measures of the proposed study

• More description is needed to clearly understand how implementation strategies will be observed or empirically tested

• Implementation strategies are mostly feasible given the pilot study timeline and budget constraints

• Explicitly describes and theoretically justifies the implementation strategies

• Explicitly describes how implementation strategies link to the stated aims/setting/outcome measures of the proposed study

• Explicitly describes how implementation strategies will be observed or empirically tested

• Implementation strategies are feasible given the pilot study timeline and budget constraints

Team experience with setting, treatment, and implementation process

• Only the principal investigator’s skills are described

• No additional information, biographical sketches, resumes/CVs are provided beyond the principal investigator

• It is unclear how the team experience relates to the study setting, treatment, and/or processes

• Staffing plan may not facilitate successful study completion without significant support from CIIS

• Team experience is uniform and does not offer multidisciplinary skills or perspective to the proposed study

• Team description, biographical sketches, resumes/CVs depict a multidisciplinary skillset relevant to the proposed study setting, treatment, processes, and other needs

• Staffing plan facilitates successful study completion, with some support from CIIS likely necessary

• No description of the research environment strengths including resources and/or infrastructure

• If principal investigator is considered junior or early career or novice to implementation science, it is unclear what senior leadership outside of CIIS will be available for mentoring and/or consultation

• Clearly describes how team experience relates to the study setting, treatment, and processes

• Team description, biographical sketches, resumes/CVs depict a multidisciplinary skillset relevant to the proposed study setting, treatment, processes, and other needs

• Staffing plan facilitates successful study completion without necessitating CIIS support

• Clearly describes strengths of the research environment including resources and infrastructure

• If principal investigator is considered junior or early career or novice to implementation science, senior leadership outside of CIIS has been identified to support study completion with mentoring and/or consultation

Feasibility of proposed research design and methods

• The proposed study includes methods, interventions, and other components that are beyond the scope of a pilot study and/or inappropriate for a pilot study

• A budget and/or timeline are not included or are unrealistic

• Potential barriers to implementation are not described or are insurmountable

• The proposed study includes methods, interventions, and other components that may be challenging to accomplish

• The budget and/or timeline are not included or unrealistic

• Potential barriers to implementation are not clearly described or are insurmountable

• The proposed study includes appropriate methods, interventions, and other components that are likely achievable as a pilot study

• The budget and/or timeline may need some revising

• Potential barriers to implementation are clearly described but may lack clear description of how those barriers will be overcome

• The proposed study includes appropriate methods, interventions, and other components that are achievable as a pilot study and are justified against potential alternatives

• The budget and timeline are appropriate

• Potential barriers to implementation are clearly identified with potential plans to overcome those barriers

Measurement and analysis section

• Outcomes described are not implementation or improvement science-related

• Outcomes are not linked to the proposed study aims

• The unit of analysis is inappropriate for the proposed study

• No measurement and/or data analysis plan are included to describe how variables and outcomes will be measured

• Outcomes described are implementation and/or improvement science-related

• Outcomes are unclearly linked to the proposed study aims

• The unit of analysis is appropriate for the proposed study

• Measurement and/or data analysis plans do not clearly describe how all variables and outcomes will be measured, or plans are inappropriate for the proposed study

• Outcomes described are implementation and/or improvement science-related

• Outcomes are clearly linked to the proposed study aims

• The unit of analysis is appropriate for the proposed study

• Measurement and/or data analytic plans describe how all variables and outcomes will be measured and is appropriate for the proposed study, but linkage to the theoretical model is unclear

• Outcomes described are implementation and/or improvement science-related

• Outcomes are clearly linked to the proposed study aims

• The unit of analysis is appropriate for the proposed study

• Measurement and data analytic plans robustly describe how all variables and outcomes will be measured and are appropriate for the proposed study through a clear theoretical justification

Policy/funding environment; leverage of support for sustaining change

• No acknowledgement of the internal/external policy trends and/or funding environment for the propose study is included

• Zero or limited discussion of the potential impact of the intervention is included

• Zero or limited discussion of disseminating study findings is included

• The internal/external policy trends and/or funding environment are discussed but additional clarification is needed

• The potential impact of the intervention is not linked to the policy and/or funding context and may not be relevant to a safety net setting

• The dissemination plan for study findings does not clearly indicate a contribution will be made to the broader policy level and safety net setting

• The internal/external policy trends and/or funding environment are clearly described

• The potential impact of the intervention is linked to relevant policies and funding issues associated with a safety net setting but may need further explanation

• The dissemination plan for study findings indicates a contribution will be made to the broader policy level and safety net setting, but what contribution and how it will be achieved is unclear

• The internal/external policy trends and/or funding environment are clearly described

• Potential impact of the intervention is explicitly linked to relevant policies and funding issues associated with a safety net setting

• The dissemination plan for study findings indicates what and how a contribution will be made to the broader policy level and safety net setting