Skip to main content

Table 4 Challenges to the validity of essential elements for the SPIRIT process evaluation and suggested responses

From: Figuring out fidelity: a worked example of the methods used to identify, critique and revise the essential elements of a contextualised intervention in health policy agencies

Challenges: the putative essential element was… Definition Essential element example Comments Suggested response
Redundant The strategy described by the element was not essential The provider encouraged participants to ask questions This was unnecessary in discussion-based sessions where participants interacted as co-contributors Remove this element
Poorly articulated The element description was unclear, too specific or not specific enough The session was introduced by a leader (senior person in the agency e.g. CEO, member of executive) This failed to capture the many times that less senior staff introduced sessions that were attended by leaders. This essential element was a proxy for visible endorsement/support (modelling) by organisational leaders which we concluded was also achieved when they attended and contributed enthusiastically to the session in other ways Hone the description so that it accurately captures the essential element
Infeasible The essential element described a strategy that was not possible to implement as intended Participants were facilitated to identify one or more change goals We found this was achievable only in agencies that had developed a research utilisation reform agenda prior to SPIRIT and felt able to use intervention sessions to discuss their goals openly. Other agencies needed more time and different processes to identify goals Modify or develop alternative strategies. In some cases, the outcomes themselves may need be modified
Ineffective In practice, the strategy described by the essential element did not effectively deliver the change principles The provider had experience presenting to policy/program developers This seemed intuitively reasonable as one of several criteria for securing providers with the expertise and credibility stipulated by our change principles, yet there was no correlation between this criterion and our evaluation of session quality or general participant satisfaction feedback Consider whether this element can simply be removed or if the change principles require further operationalisation to capture an essential aspect of the intervention
Paradoxical When implemented, the strategy described by the essential element counteracted the session goals or the change principles No examples of this were identified Interventions can have counterintuitive impacts. While the process evaluation identified examples of this in other aspects of the trial, none related specifically to the essential elements Remove this element and consider possible implications for other parts of the intervention
Absent or suboptimal Additional or more effective ways of operationalising the change principles were identified The provider persuasively articulated his/her commitment to using research. Despite being briefed to do so, many providers did not articulate their commitment to using research. However, some used case examples that powerfully illustrated the value of research, and facilitated discussion that enabled participants to express it themselves. This strategy was more sophisticated and a better fit with the adult learning orientated change principles that emphasise interactivity, shared reflection and harnessing participant expertise Introduce absent elements and modify sub-optimally operationalised elements that the essential aspects of the intervention are captured