Table 2 Evaluation Matrix
Evaluation questions | Indicators | Methods | Sources |
|---|---|---|---|
1. What role is there for CIHR in enabling/promoting synthesis, iKT, end-of-grant KT, and KT science? | |||
● Theory and empirical evidence related to the role of a funding organization in the KT process | ■ International environmental scan | ■ 33 funding agencies from Tetroe et al. 2008 study | |
● Is the CIHR role consistent with the health needs of Canadians, the improvement of health products and services, and the strengthening of the Canadian healthcare system? | ● Theory and empirical evidence related to the advantages and limitations of iKT research, end-of-grant KT, and KT science | ||
● Degree of alignment of CIHR KT funding program suite with theory and empirical evidence of KT success strategies | |||
● Organizational scan of comparable organizations nationally and internationally | |||
● Expert opinion on the role of a funding organization in the KT process | ■ External expert review | ■ International KT expert panel | |
● Expert opinion on the CIHR funding program mix | |||
● Expert opinion on CIHR strengths, limitations, and strategic vision for KT funding programs | |||
● Indications of incentive induced behaviour of researchers and knowledge users | ■ Case studies | ■ Exceptional funded projects | |
● Indications of unique or innovative KT strategies employed | |||
● Application pressure (total applications per funding program) | ■ Document and EIS data review | ■ EIS application records | |
● Ratio of researchers funded versus applied | ■ CIHR guiding documents | ||
● Ratio of researchers funded versus fundable but not funded | ■ Government of | ||
● Degree of alignment with CIHR mandate and strategic vision | ■ Canada documentation | ||
● Degree of alignment with the government of Canada’s plans and priorities? (i.e. SandT Strategy) | |||
2.To what extent are KT funding programs achieving their expected outcomes? | ● Indications of immediate, intermediate, and long-term outcomes | ■ Surveys | ■ Funded researchers/knowledge users |
● To what extent are immediate outcomes being achieved? | |||
● To what extent are intermediate outcomes being achieved? | |||
■ Key informant interviews | ■ Funded researchers/knowledge users | ||
■ Case studies | ■ Exceptional funded projects | ||
● The number of grants awarded by each program | ■ Document and EIS data review | ■ EIS application records | |
● # of partnerships created (iKT) | ■ End of grant reports | ||
● Comparison of application pressure across funding programs | |||
● Indications of intermediate and long term outcomes | |||
● Degree of alignment of KT funding progam suite with theory and empirical evidence of KT success strategies | ■ International environmental scan | ■ 33 funding agencies from Tetroe et al. 2008 study | |
■ External expert review | ■ International KT expert panel | ||
3. What factors facilitate or inhibit the achievement of outcomes? | ● Indication of influence on program theory from:: internal program processes; external environmental factors; strategic level factors; program delivery level factors | ■ Surveys | ■ Funded researchers/knowledge users |
■ Key informant interviews | ■ Funded researchers/knowledge users | ||
■ Case studies | ■ Exceptional funded KT projects | ||
■ Document and EIS data review | ■ EIS application records | ||
■ Final reports | |||
● Program delivery level factors | |||
4. How effective is the mix of funding programs in achieving CIHR’s expected outcomes? (iKT, End of grant-KT, KT Science, Synthesis) | ● Perceptions of suitability of program mix for promoting/enabling effective KT | ■ Key informant interviews | ■ Funded researchers/knowledgeusers |
■ Surveys | ■ Funded researchers/knowledge users | ||
● Profiles of pathways to program outcomes | ■ Case studies | ■ Exceptional funded KT projects | |
● Degree of alignment of CIHR KT funding program suite with theory and empirical evidence of KT success strategies | ■ External expert review | ■ International KT expert Panel | |
■ International environmental scan | ■ 33 funding agencies from Tetroe et al. 2008 study | ||
5. To what extent have KT funding programs reached a broad and diverse range of knowledge users? | ● Number and type of knowledge users included per iKT grant | ■ Document and EIS data review | ■ EIS application records |
■ Final reports | |||
● Perceptions of meaningful partnerships having been established | ■ Surveys | ■ Funded researchers/knowledge users | |
■ Key informant interviews | ■ Funded researchers/knowledge users | ||
■ Case studies | ■ Exceptional funded KT projects | ||
6. To what extent are KT funding programs being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness? | ● Indications of efficiency and effectiveness in the conversion of program activities into program outputs | ■ Document and EIS data review | ■ EIS application records |
■ Surveys | ■ Funded researchers/knowledge | ||
● Identified success and challenges of the merit review process | ■ users | ||
■ Key informant interviews | ■ Funded researchers/knowledge users | ||
■ Case studies | ■ Exceptional funded KT projects | ||
7. What would be the effect on CIHR-funded researchers and knowledge users if the KT funding program suite no longer existed? What would be the effect on the improvement of health, more effective health services and products, and the strengthening of the healthcare system? | ● Perceived impact of absence of future KT funding on funded researchers, knowledge users, and KT outcomes | ■ Surveys | ■ Funded researchers/knowledge users |
● Perceived future directions for funded researchers, knowledge users, and KT outcomes in the absence of KT funding | ■ Key informant interviews | ■ Funded researchers/knowledge users | |
● Use of alternative funding sources by KT funded teams (leveraging) | ■ Case studies | ■ Exceptional funded KT projects | |
● Use of alternative funding sources by KT researchers and knowledge users not funded by CIHR (Knowledge User partners) | ■ EIS | ■ EIS application records | |
● Organizational scan of similar organizations nationally and internationally | ■ Final reports | ||
■ International environmental scan | ■ 33 funding agencies from Tetroe et al. 2008 study | ||
8. What are the unanticipated outcomes, positive or negative, resulting from the KT funding programs? | ● Identified unintended outcomes of KT funding programs | ■ Document and EIS data review | ■ EIS application records |
■ Final reports | |||
■ Surveys | ■ Funded researchers/knowledge users | ||
■ Key informant interviews | ■ Funded researchers/knowledge users | ||
■ Case studies | ■ Exceptional KT funded projects | ||
■ International | ■ 33 funding agencies | ||
■ environmental scan | ■ from Tetroe et al. 2008 study | ||
■ External expert review | ■ International KT expert panel |