Modeling technology innovation: How science, engineering, and industry methods can combine to generate beneficial socioeconomic impacts
© Stone and Lane; licensee BioMed Central Ltd. 2012
Received: 1 August 2011
Accepted: 4 May 2012
Published: 16 May 2012
Government-sponsored science, technology, and innovation (STI) programs support the socioeconomic aspects of public policies, in addition to expanding the knowledge base. For example, beneficial healthcare services and devices are expected to result from investments in research and development (R&D) programs, which assume a causal link to commercial innovation. Such programs are increasingly held accountable for evidence of impact—that is, innovative goods and services resulting from R&D activity. However, the absence of comprehensive models and metrics skews evidence gathering toward bibliometrics about research outputs (published discoveries), with less focus on transfer metrics about development outputs (patented prototypes) and almost none on econometrics related to production outputs (commercial innovations). This disparity is particularly problematic for the expressed intent of such programs, as most measurable socioeconomic benefits result from the last category of outputs.
This paper proposes a conceptual framework integrating all three knowledge-generating methods into a logic model, useful for planning, obtaining, and measuring the intended beneficial impacts through the implementation of knowledge in practice. Additionally, the integration of the Context-Input-Process-Product (CIPP) model of evaluation proactively builds relevance into STI policies and programs while sustaining rigor.
The resulting logic model framework explicitly traces the progress of knowledge from inputs, following it through the three knowledge-generating processes and their respective knowledge outputs (discovery, invention, innovation), as it generates the intended socio-beneficial impacts. It is a hybrid model for generating technology-based innovations, where best practices in new product development merge with a widely accepted knowledge-translation approach. Given the emphasis on evidence-based practice in the medical and health fields and “bench to bedside” expectations for knowledge transfer, sponsors and grantees alike should find the model useful for planning, implementing, and evaluating innovation processes.
High-cost/high-risk industries like healthcare require the market deployment of technology-based innovations to improve domestic society in a global economy. An appropriate balance of relevance and rigor in research, development, and production is crucial to optimize the return on public investment in such programs. The technology-innovation process needs a comprehensive operational model to effectively allocate public funds and thereby deliberately and systematically accomplish socioeconomic benefits.
Achieving societal and economic benefit through evidence-based practice and policy making is an issue that has received increasing attention by social scientists over the last century. The opportunity to influence decisions regarding knowledge implementation with evidence from research, initially pointed out and discussed by Weiss as research utilization, has since been discussed under varying terminology, including knowledge utilization knowledge transfer knowledge dissemination, and knowledge exchange[1, 2]. This opportunity is now often called knowledge translation (KT) by healthcare fields concerned with implementing research findings to generate positive impacts for patients and professionals (i.e., quality-of-life and efficacious treatment protocols). The challenges to implementing research findings are thought to encompass three aspects: effectively communicating the new knowledge to target audiences in form and content, facilitating its implementation within the user’s context, and tracking and measuring the resulting impacts.
Stakeholders involved in the supply side of scientific research and engineering development outputs (sponsors and grantees) have responded to the challenge of modeling and measuring in different ways. Attempts by the US government to improve the results achieved by government-sponsored research and development (R&D) programs have resulted in various legislative measures, such as the Government Performance and Results Act (GPRA) in 1993, and measurement tools, such as the Program Assessment Rating Tools (PART) by the Office of Management and Budget (OMB) . Note that the revisions in the Government Research and Performance Modernization Act of 2010 establish even more specific performance parameters by more specifically defining a governance structure and by more clearly connecting the underlying plans, programs, and performance information to be reported. Agencies sponsoring R&D activity—and therefore responsible for collecting and reporting evidence of results—have responded by adopting planning and management tools such as the logic model[4–7]. In addition, the techniques of research syntheses, meta-analyses, and systematic reviews are assessing the quality of work (performed in terms of rigor and relevance) . These ex-post facto techniques measure effects from past studies or assess the validity of their findings to determine their worthiness for future application. By definition, a supply push orientation requires a retrospective orientation because analysis follows completion of the activity. The National Institutes of Health (NIH) catchphrase “bench to bedside” is an explicit statement of such a supply push orientation, where scientific activity is a given, so a focus on their application is a fait accompli. The science, technology, and innovation (STI) policies and programs are beginning to consider achieving impact by proactively applying knowledge translation through a demand pull orientation, where the analysis of the problem and the solution precedes the activity—“from bedside to bench and back” [9, 10]. Balancing the issues of rigor and relevance becomes even more complex when the government programs that sponsor extramural programs support engineering development activity as well as scientific research activity. Engineering development activity represents a different yet equally rigorous methodology. The difference is critically important for highly regulated and monitored fields such as medical and health technologies . Scientific research methods are designed to generate data analyzed as findings (conceptual discoveries) that can be attributed to the relationship between variables under study and not due to chance. In contrast, engineering development methods are designed to demonstrate that a conceptual relationship between variables can be embodied in a tangible form (prototype invention). The prototype is proof of the principles represented by the conceptual discovery. As the first demonstration of both novelty and feasibility, this prototype can be considered an invention, with the underlying intellectual property entitled to protection under patent laws. However, in order for these outputs from research and development to contribute to socioeconomic benefits, they must be embodied in goods or services exchanged in the commercial marketplace. That is, the invention outputs from engineering development—grounded in the scientific research knowledge base—become inputs to the commercial production process. It is the goods and services produced by industry and deployed in the commercial marketplace that have the capacity to generate social and health benefits to targeted groups, and corresponding economic benefits to a nation . In healthcare, these market innovations are as diverse as medical or assistive devices, pharmaceuticals, and treatment protocols .
That said, let’s denote the boundaries of the ensuing analysis. Not all R&D is conducted through grants, nor is all R&D sponsored by governments. However, the substantial investment of public funding with the expressed intent to benefit society warrants a focus on government-operated R&D programs and the innovation-oriented projects that they sponsor.
This paper does not address, and therefore should not be perceived as a criticism of, government-sponsored basic research variously known as fundamental, curiosity-driven, undirected, investigator-inspired, or Mode 1 research. Basic research in the physical, biological, and social sciences is appropriate and contributes directly to the knowledge base in the short term and indirectly to society in unanticipated and often serendipitous ways in the long term. Since serendipity is neither deliberate nor systematic, it is not amenable to advanced planning and therefore outside the boundaries of this paper.
This paper is specifically concerned with those government programs established for the explicit purpose of achieving beneficial socioeconomic impacts through the deliberate and systematic creation and diffusion of technology-based innovations. The allocation of public funding to R&D activities in university, government, and or corporate laboratories is justified by stating that the expected impacts meet national needs that are not being addressed through standard market forces. Many nations operate such technology-based innovation programs. The United States has the National Science Foundation’s Engineering Research Centers (ERCs) , Industry/University Cooperative Research Centers (I/UCRC) , and Innovation Corps (I-Corps) ; National Institutes of Health’s Program on Public Private Partnerships ; National Institutes of Standards and Technology’s Technology Innovation Program (TIP) ; along with two government agency-wide programs sponsored by the Small Business Administration: Small Business Innovation Research (SBIR); Small Business Technology Transfer (STTR) . Canada funds technology innovation through R&D programs within the Natural Science and Engineering Research Council (NSERC) of Canada, such as the Business-led Network of Centers of Excellence , while the European Union jointly funds and coordinates the Research Framework Programme  (currently in its seventh 5-year cycle) and the Competitiveness and Innovation Framework Programme. All of these programs support global competitiveness through directed and applied R&D for technology-based innovation. These and others around the globe—including China’s 2050 market-oriented innovation policies —constitute a nontrivial level of public funding invested to deliberately and systematically advance technology-based innovation. All should have process models and performance measures.
In light of the foregoing, this paper presents a conceptual framework for applying KT to the planning, implementation, and evaluation of both research activities and development activities to generate the outputs necessary as inputs to production activities, so industry can generate the socioeconomic impacts desired by society. The focus here is the specific case of translating technological outputs (i.e., conceptual discoveries and tangible inventions that are eventually transformed into innovative devices or services in the commercial marketplace). Therefore, the models must accommodate stakeholders beyond the academic community.
A technology-oriented framework defends the argument that, in deriving societal impact from R&D outputs, the issue is not so much how to track and measure the impact as how to plan for and obtain it. Rigor, in both research and development methodologies, is essential to ensuring the credibility of discoveries or verifying the attributes of the prototypes they generate, particularly concerning the efficacy of medical devices or pharmaceuticals. However, relevance is equally indispensable to both methodologies, in terms of relevance to industry as well as to the intended beneficiaries of the innovations. For R&D programs that intend to generate beneficial impacts, the authors contend that relevance is an essential precursor to any project, regardless of whether the project methodology is for scientific research, engineering development, or industrial production.
Evaluation, as a process that investigates the merit and worth of whatever it addresses, plays a crucial but often overlooked role in ensuring both rigor and relevance . The typical R&D project does not engage evaluation systematically, with appropriate emphases on formative and summative forms. This paper argues that the evaluation process can be readily integrated with logic modeling through the Context-Input-Process-Product (CIPP) model developed by Stufflebeam [24–28]. The integration seeks to provide a more comprehensive conceptual framework for R&D project planning, where evaluation explicitly supports both the design of project activities that are relevant as well as a follow-up of their results to impacts. Further, in developing such logic models to represent program theories, a case is made for emphasizing context evaluations that link programs with their funded projects. Context evaluations ensure relevance through specific needs assessments at the project level and through broader situation analyses at the program level. To be recognized as a field, implementation science requires such comprehensive frameworks to establish the merit and worth of government-sponsored programs intending to generate socioeconomic impacts through technology-based innovations. This will demonstrate their value to society.
Knowledge translation concepts
A more refined model is needed to address the current under-utilization of outputs from sponsored research and/or development projects. The concept of KT as a solution involves a strategic communication of the knowledge outputs to those interested in using them. A key question is whether knowledge use can be caused by the knowledge producer or if use is determined by the knowledge recipient.
The most commonly used definition of KT, according to the literature, is the one by the Canadian Institutes of Health Research (CIHR). It states the following:
Knowledge translation is a dynamic and iterative process that includes synthesis, dissemination, exchange and ethically sound application of knowledge to improve the health of [citizens], provide more effective health services and products and strengthen the healthcare system . The concept of knowledge translation arose in the context of healthcare, so this definition reflects that context. However, the concept is readily applied to any other field of knowledge creation and application.
Among the many efforts to develop models of KT, the Knowledge-to-Action (KTA) model by Graham and colleagues is notable for its comprehensive inclusion of the aspects involved in communicating knowledge to precipitate action. It incorporates both a Knowledge Creation component and a corresponding Action Cycle component to identify applications and communicate the research-based knowledge to stakeholders [30, 31]. The former takes knowledge from the inquiry stage to the tools stage. The latter proposes to identify and address problems relevant to the application of these tools, including the importance of adapting the knowledge to the user’s context.
Knowledge translation implementation
The KTA model contains two variations: end-of-grant KT and integrated KT. End-of-grant KT focuses on translating outputs from completed research projects, while integrated KT involves external stakeholders throughout the research project process, from design through to application. Both variations recognize the need to translate the outputs from research projects to eventually demonstrate evidence of output use by stakeholders. In doing so, both restrict the source of knowledge to scientific research methods, the primary actor as a scholarly researcher, and the supporting resources as issuing from a sponsored grant. That is, the KTA model is centered on the professional world of the university scholar. This is appropriate for the majority of research activity, where both sponsors and scholars now seek to extend knowledge use beyond traditional borders.
The KTA model authors recognized the need for yet a third variation that would extend KT beyond scientific research, to encompass the development and production activities required for technology-based innovations. Such an expanded model was needed to address instrumental knowledge use in the creation of devices, pharmaceuticals, and services (personal communications with Dr. Ian Graham). At that time, these downstream knowledge applications were generically represented in the CIHR model of KT as ovals labeled “Contextualization of Knowledge” and “Application of Knowledge” . In this paper, the contents of these generic ovals are described as engineering development and industrial production, respectively.
Appreciating the full potential role for KT in the technology-based innovation process requires one to stand away from the articulated role for academia, in order to view the broader society in which innovation necessarily occurs. Academia relies on a closely aligned network of actors to judge the merit and worth of fellow scholars—hence the term peer review. By definition, peers know the traditions, prior literature, and current trajectories for any given topic, and they value knowledge in the form of conceptual discoveries. There is a level of reciprocity involved in mutual progress within a field of study, which includes an expectation that contributions will be cited—that is, the knowledge will be used—by colleagues.
But what happens when those conceptual discoveries from scientific research are offered for uptake and use by stakeholders outside this peer network? Nonacademic stakeholders carry their own value systems on which to judge merit and worth. That is the scenario facing technology-oriented scholars whose own R&D activities are sponsored by government programs that expect to see evidence of downstream application of the scholarly outputs, in the form of prototype inventions arising from engineering development activity, and subsequent commercial goods and services generated through industrial production activity.
Scientific knowledge in the form of conceptual discoveries resides in scholarly literature or in prior applications of that knowledge in practical forms (e.g., existing base of prototype inventions or commercial innovations). When the decision to seek and apply scientific knowledge rests with the target audience, and the decision requires that audience to invest their own time and resources to refine that knowledge for a specific application, then scholars intending to achieve impact must shoulder an additional burden to prove that the knowledge represents value for the decision makers in those target audiences. That burden exceeds the proof required under the original two KTA model variants.
When considering the probability of achieving deliberate and systematic technology-based innovations, the end-of-grant KTA variation is clearly the most risky—a supply push orientation with no assurance that the resulting conceptual discovery is needed, wanted, or even relevant. Even the integrated KTA variation is grounded in the assumption that conducting a research project is a given, and stakeholder engagement will help define the utility and eventual applications. Both KTA variations assume that the knowledge creation begins with some sponsored research activity and ends with the application of the knowledge created.
In contrast, for technology-based innovations, scientific research is only one of three methods involved; research outputs represent knowledge generated in only one of three states, and the question to conduct or not to conduct research is a legitimate one raised by decision makers representing a range of nonacademic stakeholders.
The KTA variation described below as “prior to grant” makes no such assumptions regarding a given and initiating role for scientific research. The opportunity to achieve a technology-based innovation may not require any new conceptual discovery—and therefore no justification to sponsor and conduct scientific research. The state of conceptual science residing in the literature may provide all that is needed, and all may be publicly accessible in the scholarly literature. Similarly, the state of practical engineering may provide all that is needed regarding prototype development, and this too may be publicly accessible in the patent database and industrial literature. If so, there may be little or no need to invest time and money in new prototype inventions.
The authors contend that government programs intending to generate beneficial impacts through technology-based innovations should have the option to sponsor projects that design and implement innovation in response to a validated need, and only then consider what combination of delivery mechanisms and activities to undertake. To this end, Lane & Flagg proposed the third KTA variation called prior to grant.
Beginning with the end in mind figuratively substitutes a rifle for a shotgun. Articulating and validating a need and then identifying potential solutions permits a program to consider the extent to which the states of science and/or engineering have already generated the knowledge needed to generate the envisioned innovation. That in turn identifies any gaps in the needed knowledge so the sponsor can call for focused scientific research or narrowly defined engineering development activity. The results should maximize the return on investment and facilitate documenting and demonstrating evidence of effectiveness.
Relevance versus rigor in knowledge creation and adoption
Motivating users to apply technology-oriented knowledge might be a heavier burden on the KT process than is typically assumed, particularly when considering the broader perspective of knowledge use outside of academia and by multiple stakeholder groups. Resistance to change in general or to adopting novel discoveries in particular are concerns that have already been raised in diffusion literature . Beyond that, time and resources available to people or organizations are limited and precious commodities, so any allocation of either is an opportunity cost precluding their allocation to something else. This is the reality of the constant trade-offs required in a fast-paced world besieged by multiple interests competing for support.
A person or organization assessing new knowledge for potential implementation perceives value from its relevance as much as from its rigor. While merit and worth, as proposed by the Joint Committee, are important aspects of knowledge in assessment, new knowledge with high rigor but low relevance is less likely to motivate the effort necessary for adoption . Conversely, new knowledge with high relevance but low rigor may be readily adopted but lack sufficient quality for sustained use in practice.
As noted, these issues are compounded for the use of new technology-oriented knowledge in the context of commercial innovations because the users risk wealth and health, as well as time and opportunity cost. Any KT effort involving the adoption and use of outputs from technology-oriented research and/or development activities might be best served by considering both the merit and the worth at the point of project conceptualization, that is, by identifying the most appropriate methods (rigor) for the activities required, while considering the context and values (relevance) of the target adopters . Fortunately, the earlier-mentioned CIPP Model is up to this task because it emphasizes the target adopter’s context as a critical factor, thus supporting utilization-focused research and/or development as central to new knowledge creation.
From a public policy perspective, a societal context justifies government funding of programs intended to benefit that societal context. The context may be a gap in service provision, a geographic disparity in support, or demographic inequities in access to resources. Government programs address societal issues when the standard market mechanisms lack the necessary economic incentives, but that does not diminish a government’s need to consider those market mechanisms as essential to delivering and sustaining a technology-based innovation underwritten by public funding. Once the issue is framed in terms of societal problems and technology-based solutions, the parity between rigor and relevance becomes self-fulfilling. This reasoning fits the prior-to-grant KT perspective.
How do technology-oriented programs generate socioeconomic impacts?
The path of knowledge translation
Research and development
Short-term/mid-term changes [in user context]
Long-term benefits [to users]
The unstated implication for technology transfer is that R&D activities culminate in technology-based innovations in the marketplace. That “black box” of innovation can be opened and described in much more detail. In general, the downstream sequence of innovation is the translation of research (R) activity outputs (conceptual discovery) to inputs for development activities, followed by the transfer of development (D) activity outputs (prototype inventions) to inputs for production activities, followed by the deployment of production (P) outputs (finished goods and services) in the market.
Extending research and development outputs to production impacts
The explicit integration of the three methods of research, development, and production (R-D-P) is necessary for sponsors and managers alike to trace the path of technology-based innovations from concept to impact. Achieving technology-based innovations requires some combination of R-D-P activity, conducted by the appropriate actors, in a reasonably systematic and deliberate fashion. As noted, some of the required R&D activity may have occurred in the past or in a different field of application, in which case the existing outputs need to be identified and transferred into this new application. Some of the relevant prior work may be a serendipitous example of a solution meeting a need, which is a welcome opportunity. However, serendipity is not a basis for policies surrounding systematic and deliberate innovation.
An R-D-P orientation accommodates the dynamic interplay of multiple stakeholders, methods, and outputs/inputs missing from the linear model of innovation, which have also been missing from government STI policies since the 1940s. It was during that time that Dr. Vannevar Bush first suggested that national R&D programs could address important societal needs by connecting basic research to development and application . Despite his intention to coordinate and integrate the sectors and their activity, what survived was an emphasis on basic research, whereby scholars would create a repository of findings from which applied scientists and technology developers could independently draw solutions to problems .
Under the linear model of innovation, public agencies fund science-based research, which is assumed to possess sufficient value to spark private investment in the downstream activities necessary to generate socioeconomic benefits. Its dominance over national policies is grounded in an assumed cause-and-effect relationship between the front-end investment in science and the back-end generation of market innovations. The linear model of innovation is often supported or refuted anecdotally, but the critical cause-and-effect link has not been empirically demonstrated as being either reliable or systematic [36, 37]. The absence of a proven causal link becomes especially problematic when the sponsored activities do not match the expectations of public policy. For example, government increases the investment in science to improve domestic quality of life or to compete in the global marketplace. Yet, there is a lack of evidence that the investment causes the intended socioeconomic benefits. In the short term, innovation policies expecting outputs from development projects (prototype inventions) or from production projects (commercial goods or service) have difficulty linking these expectations with the prior outputs from research projects (conceptual discoveries). In the long term, the absence of models to track the progress of knowledge through the innovation process works against generating any evidence that might exist.
For policies and projects that intend to precipitate technology-based innovations with socioeconomic benefits, their context includes the full range of R-D-P activities and beyond to their outcomes through stakeholders and, further, to their eventual impacts. This greatly increases the complexity and difficulty of tracking progress from the initial government investments (inputs), through the series of R-D-P activities (inputs/outputs), then on to mid-term outcomes realized by stakeholder applications, and on to the long-term beneficial socioeconomic impacts.
At a minimum, both sponsors and grantees for such innovation programs should align their expectations with society. Doing so would require them to treat the full R-D-P process underlying the introduction of market innovations explicitly in all phases of government-sponsored programs, including (1) requests for proposals, (2) preparation of proposal submissions, (3) proposal reviews, and (4) project management and monitoring. This would be a significant change in the culture and systems through which the public monies are allocated and dispersed, but the gains in documentation and demonstration would be worthwhile. The system would then at least have delivered the envisioned innovations to the stakeholders positioned to deploy and apply them, where use by target audiences would determine the actual impacts and the level of benefit derived.
The CIPP model to achieve merit and worth in outputs
The fundamental issue underlying accountability in government-sponsored R&D is less about documenting impacts and more about planning for and obtaining such impacts. Effectively planning and implementing research (or development) projects facilitates future tracking and documentation of any contributions made to downstream industrial production and commercialization outcomes, even after the knowledge changes states and when the credit to upstream investigators and sponsors is lost. We reiterate that rigor (merit) and relevance (worth) of knowledge are equally important to any KT effort, particularly those concerning technology-based innovations. The role of evaluation is underappreciated but timely given current demands for demonstrated results.
The CIPP evaluation model proposed by Daniel Stufflebeam [25, 27, 28, 38] is relevant to this discussion because it links evaluation with decision making within systems—such as the technology-based innovation system—and it bears relevance to all of the elements within the knowledge-creation process. Stufflebeam first introduced it in 1966 to guide mandated evaluations of US federally funded projects . The CIPP model shows the enlightening role of evaluation for guiding decisions made in the context of planning for change . The model has since been expanded into a “comprehensive, realistic, practical, and philosophically grounded approach to conducting transparent, defensible, and effective evaluations” . The CIPP model is management-oriented in approach and committed to program improvement and accountability .
The CIPP model takes a systemic approach by referring to project goals, inputs, processes and outputs. Needs analysis, which is central to context evaluation, lends direction to the project, taking it closer to the target audience needs, while bringing relevance (worth) to the planned output. Input evaluation ensures that the project is feasible. Process evaluation for research, development, or production methods promotes efficiency and effectiveness. Output evaluation ensures and assesses the quality (merit) of the output and continues to follow up. Thus, evaluation builds relevance from the beginning. It also builds quality by repeated assessment during formative evaluation. Systematic evaluation done according to the CIPP model brings both merit and relevance to the project output.
The need to knowledge model
The R, D, and P processes are performed in stages 1 to 9, where stages 1, 2, and 3 correspond to research activity; stages 4, 5, and 6 to development activity; and stages 7, 8, and 9 to production activity, respectively.
Problem validation and project goal (solution) definition initiates the R stage in stages 1 and 2. We note that the goal corresponds to the final P output in stage 8, although the P stage continues into stage 9 with assessment and revision.
Note that evaluation supplies the beneficiary’s (end user) needs well before stage 3. Stages 1 and 2 imply a context evaluation conducted at the level of specificity of a needs analysis for the project. So, the KT process starts before conceptualizing any formal R process—a prior-to-grant perspective. Thus, relevance to intended outcomes and impacts is built into the planning of R and/or D, to the extent the project’s plan determines the need to generate new conceptual knowledge beyond what already exists in the published literature.
Stages 1 and 2 perform the structured data collection and synthesis in preparation for R, while stage 3 performs the empirical research and yields any required conceptual discovery outputs. Stages 4, 5, and 6 prepare for and conduct any needed D leading to tangible prototypes. Stages 7, 8, and 9 apply P methods to transform the prototype into a device or service for the marketplace, commence the product’s launch, and follow its ensuing market performance.
The flow of knowledge through the R-D-P process is continuous, but the R-D-P activities can also be completed as separate projects. Projects can start at D if the necessary conceptual knowledge from R activity is already available, obviating the need to conduct new research. Or a project can commence at the P phase, if the prior conceptual discoveries are made and the knowledge has been reduced to practice as a functional prototype (i.e., R&D have already been completed). Or, each R, D, or P project can stop at its respective outputs. This is where one often finds a disconnect between the stated intentions of a federally funded program and the project performance of a grantee. A federal program may be funded with the intent to generate innovative devices or services in the marketplace. But grantees may only plan to seek the outcomes for which they are rewarded. For example, scholars may stop at the publication of R outputs to satisfy requirements for tenure and promotion, or inventors may stop at the proof-of-concept prototype necessary to secure patent protection. However, the CIPP model implies that when a plan is comprehensive, the relevance of the final output—the device or service innovation—is integrated from the beginning, during the context evaluation stage, no matter which activity is being conducted in a particular funded project.
Manufacturers are a key stakeholder group in transforming R outputs into D and P outputs. The government and academic sectors do not typically engage in product design, production, distribution, and support. Outputs of projects that intend to generate socioeconomic impacts on a national scale typically pass through the industrial sector. These may be public or private corporations that may operate as for-profit or non-profit entities. The production of goods and services for the marketplace is the domain of industry. It is through industry that technology-based innovations achieve the intended socioeconomic impacts. In this context, society is best served when the government and academia sectors orient their STI programs toward industry as their primary customer.
While a myriad of other groups have a stake in the innovation process, their input is marginal relative to the decisions and commitments of industry. Despite its critical role in the innovation process, industry is not widely embraced by the federally sponsored R&D community, precisely because its critical role is not well represented in the program planning and evaluation models of the government and academic sectors.
Getting outcomes and impacts from R-D-P project outputs
Evaluation proactively builds relevance into the final output of an R-D-P process by providing evaluative information on beneficiary needs for incorporation into planning. It also guides the intrinsic quality (merit) of output throughout the process, generating formative and summative data useful for program improvement and accountability. Thus, evaluation is an enlightening companion to program planners and managers who set out to obtain R outputs that meet quality standards that are potentially relevant to the ensuing methods of D and P. The ultimate test of relevance of the R-D-P outputs, however, lies in their outcomes (changes achieved through the stakeholders) and their impacts on beneficiaries. The next logical step is to consider how evaluation helps to track evidence of progress beyond the outputs of R or D projects, to assess their actual outcomes through P projects and their eventual socioeconomic impacts.
Linking the CIPP model to the logic model
Stufflebeam discusses in detail the CIPP model’s explicit guidance in shaping the desired output, which can be applied to R, D, or Production methods. On the other hand, the CIPP model’s guidance in tracking the path of changes beyond the output (i.e., outcomes and impacts) is less explicit—although they are implicit in the perspective of planning for change. Sponsors and investigators engaged in R-D-P projects must follow some structured guidance to assess the extent to which they eventually achieve their intended impacts. The widely accepted process at program level called “logic modeling” serves this purpose because it articulates the structure for linking inputs to the expected outputs, outcomes, and impacts in an a priori way. Frechtling refers to the logic model as a tool that describes the theory of change underlying an intervention, product, or policy . An intervention, for example, can be designed to change behavior and thereby solve an identified societal problem. As Rogers points out, logic models can be constructed prospectively for new programs or retrospectively for existing programs . In practice, they have ranged from simple linear versions to nonlinear forms that reflect the complexity of the programs that they represent. They have been used in various settings and by a wide range of public and not-for-profit agencies [46–49].
Evaluators have proposed constructing logic models as a way to describe a program’s theory, or as the causal model that links the intervention with particular results . Logic models are increasingly recognized for their value to both planning and evaluation [51–56]. Some disagree. Stufflebeam and Shinkfield see the value of using a pre-existing theory that is appropriate, but they caution against spending undue time on constructing a model and against considering it as a fixed or validated theory . Scriven and Coryn suggest that program theory construction is neither necessary nor sufficient for conducting sound program evaluation . They point out that it is both expensive and diversionary. However, it is generally recognized that a program theory can explain how the program brings about (or fails to bring about) results, especially if the theory represents an intervention with the cause-effect relations duly identified and described. Among the several uses of a logic model or program theory, Rogers points out that it can provide program staff and other stakeholders a common, motivating vision. It can also help them report a performance story to funders and senior decision makers .
In Figure 3, the activities (Box B) of the funded project generate the project’s planned output or its immediate results (Box C) using the funding input (Box A). It then disseminates these results to targeted audiences . For technological outputs, these audiences include many categories of stakeholders, including other researchers, manufacturers, practitioners, policy makers/implementers, and technology/information brokers, as well as the end consumers as the beneficiaries from the market innovation . These stakeholders are all potential agents of change in the program’s context, where the changes themselves represent the short-term and mid-term outcomes (Boxes D and E). In fact, any of these stakeholder groups may be sufficient to block progress toward the intended outcome, if their particular interests are not accounted for and served to the extent necessary to gain their support. Ultimately, these outcomes should lead to the program’s broader goal or the desired impact (Box F)—sometimes called the long-term outcome.
As an example of this logical sequence, consider an application by the National Institute for Disability and Rehabilitation Research in its long-range plan for 2005–2009 . Eliminating disparities between people with disabilities and the general population is a long-term area identified in the model. In operational terms, in the context of technology-based R&D, this outcome calls for increased functional independence of consumers with disabilities through the use of assistive technology devices or services. Therefore, actions by manufacturers to introduce new or improved devices or services are viewed as short-term and mid-term outcomes. In turn, the funded R&D projects in the field of rehabilitation engineering are expected to generate the conceptual discoveries and prototype inventions that industry needs to improve upon the existing state of the practice. Boxes A, B, and C describe the project (or program) itself and how it operates, while Boxes D, E, and F describe the effects expected to be caused by them. This causal link is the basis for documenting and demonstrating results.
Related to this specific example, the authors conducted longitudinal case studies on 11 national R&D centers funded in five-year cycles, with the expectation that they would generate technology-based innovations beneficial to persons with disabilities. The case studies revealed that these R&D centers—all affiliated with major US research universities—were less successful at generating technology-oriented outputs, and in achieving external adoption and use by stakeholders, than they had initially estimated in their funded grant proposals. A review of the original proposal narratives showed a marked absence of planning and budgeting for the downstream relationships and activities of development and production. Apparently both the grantees and the reviewers were naïve about the requirements to advance beyond research outputs, and the ensuing grant period was wholly consumed by the research activities in which the grantees were highly trained. A comparison case involving an R&D center experienced in the downstream activities demonstrated a much higher success rate for transfer and commercialization outcomes .
These success rates and barriers are similar to those found in analyses of commercialization success by universities in other fields of application [59–61]. To explicitly claim that academic professionals and institutions have the expertise, infrastructure, and incentive systems to independently deliver technology-based innovations benefitting society would sound naïve. Yet that is the implicit claim underlying STI polices. However, this perspective may be shifting to consider how university R&D must be coordinated with downstream stakeholders and activities. According to an analysis by Seigel et al., “A key conclusion is that universities and regions must formulate and implement coherent and feasible technology transfer/commercialization strategies.”
Knowledge translation and logic models
KT represents a more active and tailored approach to communicating new knowledge in any state than the relatively passive approaches of gradual diffusion or scholarly dissemination. KT is also more expansive because the target audiences include multiple stakeholder groups rather than peers within a field of study or practice. The KT arrow connecting the input box to the activities box is a reminder that a KT effort should start even before the conceptualization of the R&D program to ensure optimal relevance to the problem at hand—and thereby to the intended knowledge users. This prior-to-grant perspective is built into the CIPP model through context evaluation, which addresses both program priorities (broader context) and user needs for the project (specific context), and in the logic model’s focus on impact. Combining the two models relates the intended project impacts to the sponsor’s broader program goals, which the authors contend should be the hallmark of any publicly sponsored program intending to benefit society.
Integrating CIPP with the logic model to represent technology-based R&D programs
The CIPP evaluation activities are juxtaposed around the project activities box, connected respectively to its objectives, structure, and process.
The project activities box is expanded to show R-D-P projects and their relations. Note the KT bridges within the project process and how they link methodological outputs vertically and horizontally. Note also the initial gate before the R phase, where we can avoid the time and expense involved in sponsoring new research if the necessary knowledge already exists in the literature base inside or outside the field of application.
The KT bridges go outward from outputs to outcomes; note the forked KT symbols going to outcomes. Here, KT happens in two ways: a general KT and a more focused KT. The first case involves delivering outputs to all stakeholders with potential interest. In the second case, KT may be limited to a specific group, organization, or individual, such as a manufacturer, positioned to treat the knowledge as input to the next method.
Figure 6 then extends the logic model over time. It shows the part of the model where the outputs cause effects in the form of actions by stakeholders (labeled as outcomes). Figure 6 presents the sequence of outcomes resulting from outputs of the R, D, and P activities separately. It also shows two outcome chains in each KT case—general KT versus focused KT. The programs/projects are expected to sequentially obtain outcomes, such as changing the state of stakeholder knowledge from unaware to aware, then on to interest in the knowledge, and eventually to implementation. Implementation should result in changes to practice (e.g., evidence-based applications, prototype construction and testing, commercial device and service manufacturing) or to policy changes (e.g., regulation and reimbursement of devices and services).
The time and effort required to progress through this sequence is partly dependent on the path taken. In Figure 6, one can trace paths of differing length from the output of any method (R, D, or P) to the outcome and impact. The timeframe for research outputs to achieve impacts—particularly for technology-based projects—is extended due to the need to pass through the two downstream methods of development and production. For research projects, achieving impact is likely beyond the scope of funding and beyond the project’s timeframe. This is an important point for project and program accountability. Because accountability requirements only extend to the termination of the funding timeframe, it is not feasible for research projects to demonstrate impact during the award. At best, they can demonstrate the downstream plan through which they or other stakeholders will complete the development and/or production activities and thereby transform outputs into impacts.
Figure 7 combines all of the prior components into a comprehensive diagram to show how the role of evaluation spans the entire innovation process and how KT serves to bridge the components. It shows the links between R, D, and P methods and how they combine to create and deliver a technology-based innovation to the marketplace. It shows the mechanisms involved in generating the socioeconomic benefits expressed in public policies and supported through government programs. Figure 7’s integrated logic model emphasizes the importance of performing context evaluation prior to initiating any efforts intended to generate technology-based innovations. This is an opportune point to apply two forms of context evaluation: (1) project context—the needs and opportunities analysis specific to the project’s immediate context, which provides information necessary for defining project objectives; and (2) program context—the analysis of the broader situational context around the project’s identified problem, which informs funding priorities and request for proposals. Through this approach, the project objectives and the program priorities all become evidence-based.
At the program level, evidence-based information about socioeconomic needs amenable to technology-based innovations helps funding agencies assess grant proposals for relevance, define indicators of impact, and determine how to monitor and evaluate funded projects. The context evaluation ensures that needs remain central to funding priorities and project deliverables.
At the project level, the evidence-based needs analysis aligns the project deliverables with the sponsor’s mission, while ensuring the relevance of project outputs to the intended knowledge users, prior to initiating activities. Figure 7 shows why that approach is preferable to the end-of-grant or integrated approaches to KT for those programs and projects intending to result in beneficial socioeconomic impacts from technology-based innovations . Given the option to do so, why would programs pursue any other path?
Figure 7 integrates key concepts and creates connections to guide construction of integrated logic models for utilization-focused R&D. As a static graphic, it is of necessity simple and linear in form. Yet it can serve as a basis for constructing nonlinear and complex models, as needed, to incorporate and explicate elements that have a bearing on the causal sequence represented by this simple model. It can be readily expanded by individual programs and projects to reflect their unique program characteristics and contexts.
The main point of this integration is to champion the cause of relevance, alongside rigor, through a continuous KT effort that starts at the very beginning of an R&D effort. Future efforts will address how to better integrate government and academic R&D programs supported with public revenues, with privately funded industry efforts to implement the outputs from R&D in technology-based innovations. The outcomes in the commercial marketplace are necessary as incentives for companies to generate the desired socioeconomic benefits. These incentives include revenues from sales paid to corporations to cover their costs (e.g., salaries, materials, and facilities) and profits to owners and shareholders. Of course, a portion of these revenues are paid to the government as taxes (profits to companies, income to employees, taxes on sales), which cycle back through the public coffers to be allocated as public funds used to sponsor R&D. Profitable companies benefit their home nations, so the balance of trade translates directly into national R&D capacity.
This paper addresses the low level of outcomes and impacts from funded R&D projects, supported by programs expressly intending to generate technology-based innovations with beneficial impacts. This issue is at the heart of accountability for evidence of outcomes and impacts from publicly funded R&D programs. Pointing out that relevance is as important for knowledge utilization as methodological rigor, the paper advocates for starting KT activities at project conceptualization stage when decisions about research and development are open to question, rather than assuming that research (or development) is a required element of any innovation effort. In effect, we are calling for designing utilization-focused R&D, rather than research-driven KT.
The paper argues that government-sponsored technology-oriented R&D projects, intending to generate innovations with socioeconomic benefits, should orient their efforts and direct their outputs toward the commercial marketplace. This venue is where a return on the public investment can be realized through three outcomes: broad and economical diffusion of the innovation to target audiences, revenue from sales captured as profit returned to the innovation producer, and generation of new tax revenue back to the government. The narrative explains that such projects necessarily involve three types of activities (i.e., research, development, and production), each with their own methods and knowledge states. These projects are utilization focused in principle, so the models had to capture key concepts that drive the logic of programs and projects that involve such a comprehensive range of sectors, methods, and activities. The narrative culminated in a conceptual framework that can guide logic-modeling processes for such innovation programs and projects.
The proposed framework integrates elements from the CIPP model of evaluation into the basic, linear logic model format that currently guides program planning and evaluation practice. What links the two models in this framework is the provision of a context evaluation activity prior to initiating any activity. This prior-to-grant perspective elevates the quality of relevance to parity with the quality of rigor—an orientation encompassing the stakeholders who determine success or failure of the entire effort. As a result, funding agencies can focus program goals to ground project objectives in the context of validated needs. The framework also clarifies the roles of process and product evaluations that strengthen the merit and worth of project outputs. The role of outcome evaluations beyond the traditional measures of outputs (i.e., publications, patents) is to assess the actual socioeconomic impacts and deliver those evidence-based results to funders and stakeholders alike.
The next step for expanding the proposed framework in the direction of a well-constructed program theory is an examination of its current simple and linear version. Clearly, the contexts of initiatives for generating technology-based innovations are far from simple and linear. They involve networks of stakeholder interactions that move outputs to outcomes to cause the intended impacts. Often, the interactions are iterative and involve stakeholders in multiple roles (e.g., implementation, managerial, regulatory, evaluative, advisory), so they are not strictly linear. These nonlinear relations should be reflected in an expanded logic model, once the basic concepts and the underlying paradigm are understood and embraced.
Given the possible nonlinear nature of relationships between variables in a technology-based innovation program, it is also useful to think of a systems approach to the creation of logic models. In fact, Rogers and Williams point out the “need to explore systems-based alternatives to the traditional linear logic models used to demonstrate the program theory” . According to them, one way to help people reflect constructively and deeply on the assumptions that underpin the theory as well as the program under study is to incorporate aspects of systems, group dynamics, and learning and cultural theories.
On another level, systems-based thinking may also allow policy makers to appropriately position the logic model within a broader context of the decision-making space in which the funding agency is embedded. For example, it might permit exploration of the model’s link to indicators in the broader system by situating it within the Innovation Systems Framework (ISF) proposed by Jordan, Hage, and Mote . In the ISF framework, the indicators relate to the micro-level, where funds allocation by arena and profile takes place, while analysis of performance by sectors and arena takes place at the meso level of the overall system. Such a systemic approach to the logic model might also explicitly represent and clarify the current distribution of responsibilities regarding accountability within organizations—such as data collection, performance monitoring, and agency oversight—whereby it will address and align the organizational practice with the needs for generating impact-oriented research.
It is clear from the foregoing that building a useful integrated model calls for a joint effort of the evaluator with all the relevant stakeholders involved in the planning of the program. The task calls for dual expertise and clearly is not an either/or proposition. In this sense, the framework proposed in this paper establishes an initial foothold on which a team of planning and evaluation experts could build program theories in their specific contexts of interest.
Finally, efforts to improve society while competing economically necessarily include programs that support technology-based innovations. Quality-of-life issues are paramount in the fields of medicine and healthcare. The process through which scientific knowledge is translated, and technological knowledge is transferred, should be accurately modeled for planning, implementation, and evaluation purposes. Describing the mechanisms underlying technology-based innovations and tracking the indicators of progress are necessary for establishing coherent milestones and accomplishing systematic results. If successful, sponsoring organizations will shift their perspective from the solution-driven “bench to bedside” to the need-driven “bedside to bench and back.”
The appropriate balance of relevance and rigor across all three methods of research, development, and production will help optimize the return on public investment in STI programs. The processes of KT and technology transfer are key to ensuring the progression of knowledge through the various states of conceptual discovery, tangible prototype, and commercial device. A coherent and comprehensive model of the technology-based innovation process is essential for the effective expenditure of public resources through government agencies for the expressed purpose of generating new knowledge with socioeconomic impacts beneficial to society.
Ensuring awareness and support for—as well as effective implementation of—technology-based outputs from research and/or from development by external stakeholders calls for a targeted and active intervention using KT, rather than a more general and passive dissemination/diffusion effort.
The issue of relevance is a critical factor in the decision by stakeholders to adopt and apply knowledge from external sources that involves their commitment to apply internal resources, so it is as important as the issue of rigor during knowledge creation through research and/or development.
Prior-to-grant KT with its demand-pull orientation represents a better opportunity to build relevance into innovation-focused R&D programs than do either end-of-grant or integrated models of KT. As a strategy for achieving intended impacts, it is effective for achieving optimal downstream knowledge implementation and efficient for focusing resources and activities on the specified need and the values of the downstream stakeholders responsible for transforming outputs into outcomes and impacts.
Both the CIPP model and the logic model can individually serve as planning tools for achieving intended change, but integrating them can provide a more complete framework, particularly for designing and tracking innovation-focused R&D involving a prior-to-grant KT perspective.
The integrated framework presented in this paper considers the implementation focus implied in innovation policies and for those R&D programs oriented toward technology-based innovations intended to generate beneficial socioeconomic impacts. It links the underlying key concepts from scientific research, engineering development, and industrial production. The authors intend this generic model format to be widely applicable for the construction of logic models underlying the unique attributes of technology-based innovation programs and their supporting policies.
This is a work in progress at the KT4TT Center, which is funded by the National Institute on Disability and Rehabilitation Research of the US Department of Education, under grant number H133A080050. The opinions contained in this paper are those of the grantee and do not necessarily reflect those of the US Department of Education. The authors acknowledge valuable input from our colleagues Jennifer Flagg and Michelle Lockett and to Dr. Margaret Campbell at NIDRR for their insightful comments on the integrated logic models.
- Weiss CH: The Many Meanings of Research Utilization. Public Administration Review. 1979, 39: 426-431. 10.2307/3109916.View ArticleGoogle Scholar
- International Development Research Centre: Knowledge Translation: Basic Theories, Approaches and Applications. 2007, [http://www.idrc.ca/en/ev-125826-201-1-DO_TOPIC.html]Google Scholar
- Averch HA: Using Expert Judgment. In Handbook of Practical Program Evaluation. Edited by: Wholey JS, Hatry HP, Newcomer KE. 2004, Jossey-Bass, San Francisco, 292-309.Google Scholar
- Wholey JS: Evaluability Assessment: Developing Program Theory. In New Directions for Program Evaluation. 1987, Jossey-Bass, San Francisco, 77-92.Google Scholar
- Wholey JS: Evaluability Assessment. JS Wholey, HP Hatry, KE Newcomer. 2004, Jossey-Bass, San Francisco, 33-62.Google Scholar
- McLaughlin JA, Jordan GB: Logic Models: a tool for telling your performance story. Evaluation and Program Planning. 1991, 22: 65-72.View ArticleGoogle Scholar
- McLaughlin JA, Jordan GB: Using logic models. In Handbook of Practical Program Evaluation. Edited by: Wholey JS, Hatry HP, Newcomer KE. 2004, Jossey-Bass, San Francisco, 7-32.Google Scholar
- Boruch RF, Petrosino A: Meta-analysis, Systematic Reviews, and Research Syntheses. In Handbook of Practical Program Evaluation. Edited by: Wholey JS, Hatry HP, Newcomer KE. 2004, Jossey-Bass, San Francisco, 176-203.Google Scholar
- Special Issue. Recognizing the supply of and demand for research in the science of science and innovation policy. Edited by: Pleke R. 2011, Policy SciencesGoogle Scholar
- Group of 8 Australia: Go8 Backgrounder 23 – Measuring the impact of research – the context for metric development. 2011, [http://www.go8.edu.au/university-staff/go8-policy-_and_-analysis/2011/go8-backgrounder-23-measuring-the-impact-of-research-the-context-for-metric-development]
- Herzlinger RE: Why Innovation in Health Care is so Hard. 2011, Harvard Business Publishing, BostonGoogle Scholar
- Lane JP, Flagg JL: Translating three states of knowledge - discovery, invention, and innovation. Implement Sci. 2010, 5: 9-10.1186/1748-5908-5-9. [http://www.implementationscience.com/content/5/1/9]View ArticlePubMedPubMed CentralGoogle Scholar
- Berwick DM: Disseminating innovations in healthcare. JAMA. 2003, 289: 15-View ArticleGoogle Scholar
- National Science Foundation’s Engineering Research Centers. [http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=13526]
- National Science Foundation’s Industry/University Cooperative Research Centers: [http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=5501]
- National Science Foundation’s Innovation Corps. [http://www.nsf.gov/funding/pgm_summ.jsp?pims_id=504672]
- National Institutes of Health’s Program on Public and Private Partnerships. [http://ppp.od.nih.gov/]
- National Institutes of Standards and Technology’s Technology Innovation Program. [http://www.nist.gov/tip/]
- US Small Business Administration, Small Business Technology Transfer.http://archive.sba.gov/aboutsba/sbaprograms/sbir/index.html,
- Natural Science and Engineering Research Council of Canada. [http://www.nserc-crsng.gc.ca/NSERCCRSNG/Index_eng.asp]
- Research Framework Programme. [http://ec.europa.eu/cip/]
- Yonxiang L: Science & Technology in China: A Roadmap to 2050: In Strategic General Report of the Chinese Academy of Sciences. 2011, Science Press Beijing, SpringerGoogle Scholar
- Joint Committee on Standards for Educational Evaluation: The Program Evaluation Standards. 1994, Sage Publications, Thousand Oaks, CA, 2Google Scholar
- Stufflebeam DL, Foley WJ, Gephart WJ, Guba EG, Hammond RL, Merriman HO, Provus MM: Educational Evaluation and Decision Making. 1971, Peacock, Itasca, ILGoogle Scholar
- Stufflebeam DL: Evaluation models. In New Directions for Evaluation. 2001, Jossey-Bass, San Francisco, 7-98.Google Scholar
- Stufflebeam DL: The 21stCentury CIPP Model: Origins, Development and Use. In Evaluation Roots. Edited by: Alkin MC. 2004, Sage, Thousand Oaks, CA, 245-266.View ArticleGoogle Scholar
- Stufflebeam DL: In International Handbook of Educational Evaluation. CIPP model (context, input, process, product). Edited by: Mathison S. 2005, Sage, Thousand Oaks, CA, 60-65.Google Scholar
- Stufflebeam DL, Shinkfield AJ: Evaluation Theory, Models, and Applications. 2007, Jossey-Bass, San Francisco, CAGoogle Scholar
- Canadian Institute of Health Research. 2009, About Knowledge Translation [http://www.cihr-irsc.gc.ca/e/29418.html]
- Sudsawad P: Knowledge Translation: Introduction to Models, Strategies, and Measures. 2007, Southwest Educational Development Laboratory, National Center for the Dissemination of Disability Research, Austin, TXGoogle Scholar
- Graham ID, Logan J, Harrison MB, Straus SE, Tetroe J, Caswell W, Robinson N: Lost in Translation: Time for a Map?. J Contin Educ Heal Prof. 2006, 26: 13-24. 10.1002/chp.47.View ArticleGoogle Scholar
- Rogers EM: Diffusion of Innovations. (5th Ed.). 2003, Free Press, Simon and Schuster Inc, NYGoogle Scholar
- Lane JP, Rogers JD: Engaging national organizations for knowledge translation: comparative case studies in knowledge value mapping. Implement Sci. 2011, 6: 1-10.1186/1748-5908-6-1. [http://www.implementationscience.com/content/6/1]View ArticleGoogle Scholar
- Bush V: Science: The Endless Frontier. 1960, National Science Foundation, Reprinted by Washington, D.CGoogle Scholar
- Logar N: Scholarly science policy models and real policy, RSD for SciSIP in US Mission Agencies, Policy Sciences. 2011, [http://sciencepolicy.colorado.edu/publications/special/sip_logar.pdf]Google Scholar
- Stokes DE: Pasteur’s quadrant: Basic science and technological innovation. 1997, Brookings Institution Press, Washington, DCGoogle Scholar
- Sarewitz D, Pielke RA: The neglected heart of science policy: Reconciling supply of and demand for science. Environmental Science & Policy. 2007, 10: 5-16. 10.1016/j.envsci.2006.10.001.View ArticleGoogle Scholar
- Stufflebeam DL: Stufflebeam’s improvement-oriented evaluation. In Systematic Evaluation. Edited by: Stufflebeam DL, Shinkfield AJ. 1985, Kluwere-Nijhoff, Boston, 151-207.View ArticleGoogle Scholar
- Worthen BR, Sanders JR, Fitzpatrick JL: Program Evaluation: Alternative Approaches and Practical Guidelines. 1997, Longman, New York, 2Google Scholar
- Scriven M: The methodology of evaluation. In Educational Evaluation: Theory and Practice. Edited by: Worthen BR, Sanders JR. 1973, Wadsworth, Belmont, CA, 60-106.Google Scholar
- Scriven M: Evaluation Thesaurus. 1991, Sage, Newbury Park, CA, 4Google Scholar
- Center on Knowledge Translation for Technology Transfer: The Need to Knowledge Model for Commercial Devices and Services. 2009, Buffalo, NY, [http://kt4tt.buffalo.edu/knowledgebase/model.php]
- The PDMA Handbook of New Product Development. Edited by: Kahn KB, Castellion G, Griffin A. 2005, John Wiley & Sons, Inc, Hoboken, NJGoogle Scholar
- Frechtling JA: Logic Modeling Methods in Program Evaluation. 2007, Jossey-Bass, San FranciscoGoogle Scholar
- Rogers PJ: Logic model. In Encyclopedia of Evaluation. Edited by: Mathison S. 2004, Sage, Newbury Park, CA, 232-4.Google Scholar
- United Way of America: Measuring Program Outcomes: A Practical Approach. 1996, United Way of America, Arlington, VAGoogle Scholar
- W K Kellogg Foundation: Logic Model Development Guide. 2001, Battle Creek, MI, [http://www.wkkf.org/knowledge-center/resources/2006/02/WK-Kellogg-Foundation-Logic-Model-Development-Guide.aspx]Google Scholar
- Williams VL, Elseman E, Landree E, Adamson DM: Demonstrating and Communicating Research Impact. 2009, Rand Corporation, Santa Monica, CAGoogle Scholar
- USDE/NIDRR: United States Department of Education/National Institute on Disability Rehabilitation Research: Notice of Final Long-Range Plan for Fiscal years 2005–2009. Federal Register. 2006, 71:31-8166–8200. NoticesGoogle Scholar
- Rogers PJ: Purposeful program teory. In workshop material distributed at the Annual Meeting of the American Evaluation Association Conference. 2009, Orlando, FLGoogle Scholar
- Chen HT: Theory-Driven Evaluation. 1990, Sage, Thousand Oaks, CAGoogle Scholar
- Chen HT: Theory-driven evaluation. In Encyclopedia of Evaluation. Edited by: Mathison S. 2004, Sage, Newbury Park, CA, 415-419.Google Scholar
- Donaldson SI, Lipsey MW: Roles for theory in contemporary evaluation practice: developing practical knowledge. 2006, Sage Publications, In The Sage handbook of evaluation, Edited by IF Shaw, JC Greene, MM Mark, Thousand Oaks, CA, 56-75.Google Scholar
- Rogers PJ: Using programme theory to evaluate complicated and complex aspects of interventions. Evaluation. 2008, 14: 29-48. 10.1177/1356389007084674.View ArticleGoogle Scholar
- Rogers PJ, Hasci TA, Petrosino A, Huebner TA: Program theory in evaluation: challenges and opportunities. In New Directions for Evaluation 87. 2000, Jossey-Bass, San FranciscoGoogle Scholar
- Weiss CH: Evaluation: Methods for Studying Programs and Policies. 1998, Prentice Hall, Englewood Cliffs, NJGoogle Scholar
- Scriven M, Coryn CLS: The logic of research evaluation. In New Directions for Evaluation. 2008, Jossey-Bass, San Francisco, 89-105.Google Scholar
- Lane J: At the confluence of academic research and business development-merging technology transfer with knowledge translation to deliver value. Focused Issue: State of the Science for Technology Transfer, Assistive Technology Outcomes and Benefits. 2010, 6 (1): 1-38. [http://www.atia.org/files/public/ATOBV6N1ArticleTwo.pdf]Google Scholar
- Litan RE, Mitchell L, Reedy EJ: Commercializing University Innovations: Alternative Approaches. 2007, Social Science Research Network, [http://papers.ssrn.com/sol3/papers.cfm?abstract_id=976005]Google Scholar
- Siegel DS, Waldman D, Link A: Assessing the impact of organizational practices on the relative productivity of university technology transfer offices: an exploratory study. Res Policy. 2003, 32: 27-48. 10.1016/S0048-7333(01)00196-2.View ArticleGoogle Scholar
- McDaniel SA: Where Science, Technology and Innovation Indicators hit the Road and Roadblocks. 2006, OECD Blue Sky II, Forum, Ottawa, Canada, [http://www.oecd.org/dataoecd/23/47/37436324.pdf]Google Scholar
- Siegel DS, Veugelers R, Wright M: Technology transfer offices and commercialization of university intellectual property: performance and policy implications. Oxford Rev Econ Policy. 2007, 23 (4): 640-660. 10.1093/oxrep/grm036. [http://oxrep.oxfordjournals.org/content/23/4/640.abstract]View ArticleGoogle Scholar
- Rogers PJ, Williams B: Evaluation for practice improvement and organizational learning. In The Sage handbook of evaluation. Edited by: Shaw IF, Greene JC, Mark MM. 2006, Sage Publications, Thousand Oaks, 76-97.Google Scholar
- Jordan GB, Hage J, Mote J: A theories-based systemic framework for evaluating diverse portfolios of scientific work, Part I: micro and meso indicators. In New Directions for Evaluation. 2008, Jossey-Bass, San Francisco, 7-24.Google Scholar
This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/2.0), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.