Skip to main content

Table 2 Evaluation Matrix

From: Understanding the performance and impact of public knowledge translation funding interventions: Protocol for an evaluation of Canadian Institutes of Health Research knowledge translation funding programs

Evaluation questions Indicators Methods Sources
1. What role is there for CIHR in enabling/promoting synthesis, iKT, end-of-grant KT, and KT science?    
  Theory and empirical evidence related to the role of a funding organization in the KT process ■ International environmental scan ■ 33 funding agencies from Tetroe et al. 2008 study
Is the CIHR role consistent with the health needs of Canadians, the improvement of health products and services, and the strengthening of the Canadian healthcare system? Theory and empirical evidence related to the advantages and limitations of iKT research, end-of-grant KT, and KT science   
  Degree of alignment of CIHR KT funding program suite with theory and empirical evidence of KT success strategies   
  Organizational scan of comparable organizations nationally and internationally   
  Expert opinion on the role of a funding organization in the KT process ■ External expert review ■ International KT expert panel
  Expert opinion on the CIHR funding program mix   
  Expert opinion on CIHR strengths, limitations, and strategic vision for KT funding programs   
  Indications of incentive induced behaviour of researchers and knowledge users ■ Case studies ■ Exceptional funded projects
  Indications of unique or innovative KT strategies employed   
  Application pressure (total applications per funding program) ■ Document and EIS data review ■ EIS application records
  Ratio of researchers funded versus applied   ■ CIHR guiding documents
  Ratio of researchers funded versus fundable but not funded   ■ Government of
  Degree of alignment with CIHR mandate and strategic vision   ■ Canada documentation
  Degree of alignment with the government of Canada’s plans and priorities? (i.e. SandT Strategy)   
2.To what extent are KT funding programs achieving their expected outcomes? Indications of immediate, intermediate, and long-term outcomes ■ Surveys ■ Funded researchers/knowledge users
To what extent are immediate outcomes being achieved?    
To what extent are intermediate outcomes being achieved?    
   ■ Key informant interviews ■ Funded researchers/knowledge users
   ■ Case studies ■ Exceptional funded projects
  The number of grants awarded by each program ■ Document and EIS data review ■ EIS application records
  # of partnerships created (iKT)   ■ End of grant reports
  Comparison of application pressure across funding programs   
  Indications of intermediate and long term outcomes   
  Degree of alignment of KT funding progam suite with theory and empirical evidence of KT success strategies ■ International environmental scan ■ 33 funding agencies from Tetroe et al. 2008 study
   ■ External expert review ■ International KT expert panel
3. What factors facilitate or inhibit the achievement of outcomes? Indication of influence on program theory from:: internal program processes; external environmental factors; strategic level factors; program delivery level factors ■ Surveys ■ Funded researchers/knowledge users
   ■ Key informant interviews ■ Funded researchers/knowledge users
   ■ Case studies ■ Exceptional funded KT projects
   ■ Document and EIS data review ■ EIS application records
    ■ Final reports
  Program delivery level factors   
4. How effective is the mix of funding programs in achieving CIHR’s expected outcomes? (iKT, End of grant-KT, KT Science, Synthesis) Perceptions of suitability of program mix for promoting/enabling effective KT ■ Key informant interviews ■ Funded researchers/knowledgeusers
   ■ Surveys ■ Funded researchers/knowledge users
  Profiles of pathways to program outcomes ■ Case studies ■ Exceptional funded KT projects
  Degree of alignment of CIHR KT funding program suite with theory and empirical evidence of KT success strategies ■ External expert review ■ International KT expert Panel
   ■ International environmental scan ■ 33 funding agencies from Tetroe et al. 2008 study
5. To what extent have KT funding programs reached a broad and diverse range of knowledge users? Number and type of knowledge users included per iKT grant ■ Document and EIS data review ■ EIS application records
    ■ Final reports
  Perceptions of meaningful partnerships having been established ■ Surveys ■ Funded researchers/knowledge users
   ■ Key informant interviews ■ Funded researchers/knowledge users
   ■ Case studies ■ Exceptional funded KT projects
6. To what extent are KT funding programs being delivered as expected? Can any changes be made to program delivery in order to improve efficiency and effectiveness? Indications of efficiency and effectiveness in the conversion of program activities into program outputs ■ Document and EIS data review ■ EIS application records
   ■ Surveys ■ Funded researchers/knowledge
  Identified success and challenges of the merit review process   ■ users
   ■ Key informant interviews ■ Funded researchers/knowledge users
   ■ Case studies ■ Exceptional funded KT projects
7. What would be the effect on CIHR-funded researchers and knowledge users if the KT funding program suite no longer existed? What would be the effect on the improvement of health, more effective health services and products, and the strengthening of the healthcare system? Perceived impact of absence of future KT funding on funded researchers, knowledge users, and KT outcomes ■ Surveys ■ Funded researchers/knowledge users
  Perceived future directions for funded researchers, knowledge users, and KT outcomes in the absence of KT funding ■ Key informant interviews ■ Funded researchers/knowledge users
  Use of alternative funding sources by KT funded teams (leveraging) ■ Case studies ■ Exceptional funded KT projects
  Use of alternative funding sources by KT researchers and knowledge users not funded by CIHR (Knowledge User partners) ■ EIS ■ EIS application records
  Organizational scan of similar organizations nationally and internationally   ■ Final reports
   ■ International environmental scan ■ 33 funding agencies from Tetroe et al. 2008 study
8. What are the unanticipated outcomes, positive or negative, resulting from the KT funding programs? Identified unintended outcomes of KT funding programs ■ Document and EIS data review ■ EIS application records
    ■ Final reports
   ■ Surveys ■ Funded researchers/knowledge users
   ■ Key informant interviews ■ Funded researchers/knowledge users
   ■ Case studies ■ Exceptional KT funded projects
   ■ International ■ 33 funding agencies
   ■ environmental scan ■ from Tetroe et al. 2008 study
   ■ External expert review ■ International KT expert panel
  1. Note: Indicators and sources presented in this matrix are not static. As the research process progresses, the Evaluation Working Group will be attuned to new information that may create the need for review