Table 4 Challenges to the validity of essential elements for the SPIRIT process evaluation and suggested responses

Redundant

The strategy described by the element was not essential

‘The provider encouraged participants to ask questions’

This was unnecessary in discussion-based sessions where participants interacted as co-contributors

Remove this element

Poorly articulated

The element description was unclear, too specific or not specific enough

‘The session was introduced by a leader (senior person in the agency e.g. CEO, member of executive)’

This failed to capture the many times that less senior staff introduced sessions that were attended by leaders. This essential element was a proxy for visible endorsement/support (modelling) by organisational leaders which we concluded was also achieved when they attended and contributed enthusiastically to the session in other ways

Hone the description so that it accurately captures the essential element

Infeasible

The essential element described a strategy that was not possible to implement as intended

‘Participants were facilitated to identify one or more change goals’

We found this was achievable only in agencies that had developed a research utilisation reform agenda prior to SPIRIT and felt able to use intervention sessions to discuss their goals openly. Other agencies needed more time and different processes to identify goals

Modify or develop alternative strategies. In some cases, the outcomes themselves may need be modified

Ineffective

In practice, the strategy described by the essential element did not effectively deliver the change principles

‘The provider had experience presenting to policy/program developers’

This seemed intuitively reasonable as one of several criteria for securing providers with the expertise and credibility stipulated by our change principles, yet there was no correlation between this criterion and our evaluation of session quality or general participant satisfaction feedback

Consider whether this element can simply be removed or if the change principles require further operationalisation to capture an essential aspect of the intervention

Paradoxical

When implemented, the strategy described by the essential element counteracted the session goals or the change principles

No examples of this were identified

Interventions can have counterintuitive impacts. While the process evaluation identified examples of this in other aspects of the trial, none related specifically to the essential elements

Remove this element and consider possible implications for other parts of the intervention

Absent or suboptimal

Additional or more effective ways of operationalising the change principles were identified

‘The provider persuasively articulated his/her commitment to using research’.

Despite being briefed to do so, many providers did not articulate their commitment to using research. However, some used case examples that powerfully illustrated the value of research, and facilitated discussion that enabled participants to express it themselves. This strategy was more sophisticated and a better fit with the adult learning orientated change principles that emphasise interactivity, shared reflection and harnessing participant expertise

Introduce absent elements and modify sub-optimally operationalised elements that the essential aspects of the intervention are captured