Study design and procedure
This research is informed by two previous projects undertaken by this research team, and parallels data collection time frames to enable comparisons of implementation following the addition of the new intervention component. This project is supported by an Australian Research Council grant (LP09900708), with financial and in-kind contributions by the industry partners. Partner organisations are five well-established community-managed organisations involved in the direct provision of services to individuals living with severe and recurrent mental illnesses in the community. The partner organisations will nominate a number of suitable worksites drawn from across their service base to participate in the intervention and research components. We anticipate access to teams across a number of Australian states and government areas, allowing us to compare and control for the effects of geographical variables. In total, approximately 200 mental health workers from across 22 sites will be randomised and referred for intervention.
Participants
Participants will be randomised by work-site to either the experimental (values) or control (implementation). Cluster-randomisation will be adopted to increase the feasibility of roll out in the organisational setting (e.g., consistency across what ‘change’ for individuals within a single worksite will look like) and minimise contamination across conditions (i.e., individual randomisation would likely lead to decreased fidelity to condition due to inevitable interactions between individual staff within teams). A computerised random integer program will be used to refer worksites to condition. Once randomised, staff from within sites will be referred for training and invited to consent to participate in the research process. Blinding will not be used at the participant or worksite level for pragmatic reasons (i.e., ongoing coaching will require condition-specific protocol) and to maximise fidelity (i.e., workers will consult with colleagues from worksites in the alternate condition during the course of duties and need to understand the importance of staying within assigned protocol). Information about previous exposure to the training (i.e., staff who have participated in some CRM training before time one) will be sought to allow screening prior to inclusion in the final data set. Staff who do not consent to participation in research but are within randomised sites will still participate in all intervention components to maximise the benefit of this project for the partners, to promote intervention fidelity within the workplace, and to allow all mental health workers access to contemporary evidence-based practices and techniques.
The intervention will be facilitated by our team and comprises both training (three days) and a 12-month coaching intervention. All teams will receive the same two days of training in the CRM [10]. On the third day of training, they will receive different training activities, according to the condition (values or implementation). In-service coaches (to be trained by our team) will conduct coaching. The in-service coaches will be supported with monthly, group-based coaching-support sessions facilitated by an appropriately skilled project member. Features of each intervention condition are as follows.
Values condition
The values intervention is delivered as two components: Activities to support values clarification and commitment delivered on day three of training; and values-based coaching using CRM tools for 12 months within the workplace.
The aims of the values condition include:
-
1.
Increase awareness of the values in which CRM is embedded.
-
2.
Increase the extent to which the personal and professional values of the staff are explicit and expressed in the workplace.
-
3.
Create opportunities for individual staff to identify the overlap or concordance of the CRM with their own values.
-
4.
Regular and sustained (12 one-hour sessions each month) investment in the professional and personal development of staff via clarification and commitment to values-based goal-setting using CRM tools as the framework.
Protocol for day three values intervention
The one-day values intervention is experiential in nature, using a structured values-clarification exercise with demonstrated utility in a range of clinical and non-clinical settings [36]. The purpose of this task is to help participants identify what values are most important to them, and to increase their awareness of the potential to actively pursue valued directions in both personal and professional domains of their life (i.e., increase the extent to which values are consciously used as a driver of purposeful behaviour). Staff members will be exposed to the concept of values in the standard CRM training (days one and two), and will have a basic theoretical and operational understanding of both the merit and applicability of working with values generally. Additionally, a focus on related concepts of willingness and commitment [37] that have been emphasised as important to values work will follow in day three.
Participants are given a set of 60 cards, each featuring a specific principle of living that is associated with a universal value-domain as outlined in Schwartz’ model [38, 39]. They are then facilitated through a structured sorting task that titrates their focus down to the 15 valued directions each individual identifies as ‘most important to them.’
Following the card-sorting task, an additional intervention component is employed to foster intent and commitment to take purposeful steps toward valued areas of living. In this stage, the focus is on ‘life in general,’ and participants are asked to rate the extent to which they have purposefully been trying to enact a variety of values in the past 12 weeks. They rate their subjective success at moving toward each of the 15 specific valued directions identified as ‘most important.’ This process is structured around a worksheet based on the Personal Strivings methodology developed by Sheldon et al. that has been used extensively within the goal setting research [35, 40].
Participants are facilitated through this process a second time, adopting a workplace focus. Participants are given the instruction; ‘conduct the card-sorting task again, this time focusing on what is most important to you in your current job.’ Following this, participants complete a second worksheet to rate the intent and self-reported success of recent striving toward the work values they have identified as most important at work.
The components thereafter focus on increasing awareness of the potential concordance between personal and professional values through a facilitated discussion session. Participants are invited to discuss commonalities between their ‘life in general’ list and their ‘workplace’ list. They are then asked to identify ways they can bring their ‘life in general’ principles into their workplace before being facilitated through the completion of the specific CRM values tool (known as the ‘camera’ [41]) as an initial commitment to this process.
Protocol for values coaching
Individual coaching sessions adopt a structure known as the GROW model structure in both conditions. GROW was made popular by Graham Alexandar and John Whitmore and it is widely used in organisational and coaching contexts as a method of setting a basic frame to a coaching session [42]. It is particularly attractive in this case due its accessibility to those with little or no prior coaching experience. GROW is an acronym for the basic components of a coaching session—namely, goal, reality, options, wrap up/where to.
Individuals identified as suitable coaches within each partner organisations will be referred to a further half-day of coaching training conducted by the research team. Potential coaches are identified by managers within each of the organisations and also through a call for expressions of interest. Trained coaches are assigned to mental health workers within the same experimental condition, but outside line management to increase role clarity. Coaching consists of 12 hour-long sessions scheduled once per month and conducted in the course of paid working hours for both participants. The CRM tools are used as the framework for recording and structuring the recipients’ development across the coaching period, such that the participating staff members are using the tools that are part of the organisational change initiative themselves (i.e., in relation to their own values-based goals). The particular focus within the values condition is on the establishment of work-related goals that fit with the values stated by the recipient in initial training, and clarified as the coaching process continues.
Implementation condition
The delivery of the implementation condition intervention components follow the same format as in the values condition but differ in focus and content. The day three of training in the implementation group is experiential in nature, but focuses on addressing opportunities for and barriers to the implementation of the CRM into practice within the workplace. The methodology used to structure and support the implementation intervention is the ‘Strengths, Weaknesses, Opportunities, and Threats (SWOT) Analysis’ developed by Albert Humphrey and used extensively in organisational contexts [43, 44].
Coaching in the control condition is consistent in format and overall structure to the values condition. Implementation coaching adopts an alternate focus on identification and resolution of issues related to implementation of CRM in the workplace as identified by participants. For example, pragmatic issues (e.g., addressing technical issues associated with new practices) or attitudinal issues (e.g., working through resistance to change from clients or colleagues).
The aims of the implementation condition are as follows:
-
1.
Ongoing exposure to and skills-based practice with the CRM.
-
2.
Opportunity to identify and develop strategies to address factors that are impeding implementation (e.g., resistance from clients or co-workers).
-
3.
Regular and sustained investment (12 x 1 month coaching sessions) in the professional and personal development of staff using CRM tools as the framework.
In both conditions, participants are using the same protocol and model of practice they are to use with their clients upon implementation. Coaching therefore promotes experiential learning [45] in both conditions. We expect this will result in increased transfer of CRM into practice compared to our previous research. In the values condition there is an additional parallel process such that mental health staff will be actively encouraged to work with the CRM practices in relation to their own lives and values, just as their clients would. That is, they are applying both the practices and the underlying processes ‘for real’ (values), rather than just practising in the use of the CRM (implementation). The value of parallel relationships in transferring knowledge from one dyad (e.g., supervisor and clinician) to another (e.g., clinician and client) has been elaborated within counselling literature [46]. We hypothesise this additional parallel relationship will increase implementation as a result of enhanced sense of meaning and connection with the CRM such that it becomes more internalised [47].
Data collection and handling
The study will last 18 months with data collection at multiple times points. Primary data comprises a questionnaire battery completed by participants at five time points throughout the intervention (specific measures are outlined below). Data is also collected from coaches and recipients’ at each monthly coaching session to assess adherence to the GROW framework, integrity to experimental condition, and elements of the coaching alliance. In addition to self-reported measures of implementation, the study uses objective measures of transfer (also outlined below).
Data collection handling is in accordance with the procedures specified in the ethics approval obtained from the Human Research Ethics Committee at the University of Wollongong (HE09/221). A prime focus is on maintaining confidentiality of participants, which is promoted by the use of a unique self-generated identifier established at the first data collection point and re-used at subsequent collections. Additionally, because the research focuses on work-related variables and is being conducted in a work setting, individual data is forwarded to the research team directly (i.e., handed personally in sealed envelopes when on-site). This is to reduce possible biased responding and staff concerns that their individual information could be seen by superiors or other personnel within their organisation.
Measures and data
The battery of measures will also include measures of intention to leave [48] job satisfaction [49], and burnout [50]. This additional data will be utilised by the research team to explore another set of hypotheses distinct from those being investigated here. Questionnaires pertaining to the outlined hypotheses are as follows.
Staff knowledge and attitudes toward recovery
A range of questionnaires will be used to determine the knowledge, attitudes, and beliefs of participants related to the concept of recovery. These measures will be compared with previous research conducted by this team [51].
The Recovery Knowledge Inventory (RKI) [51] is a 20-item instrument that assesses mental health staff knowledge and attitudes about recovery using a five-point Likert scale. It has been used in previous research to assess pre-post change following intervention and has satisfactory psychometric properties [51].
The Staff Attitudes Towards Recovery Scale (STARS) is a 19-item measure developed and evaluated as part of the Crowe et al. study [52] and assesses attitudes and hopefulness related to the goal striving and recovery possibilities for clients. It has a five-point Likert response scale (strongly disagree to strongly agree) and higher scores reflect more hopeful attitudes. The STARS has satisfactory psychometric properties (α =.81) as established in previous work [52].
Autonomy and values concordance
The Collaborative Recovery Model Values Questionnaire (CRM-VQ) is a modified version of the Personal Strivings questionnaire developed by Sheldon et al. [40]. The modification involves the use of perceived locus of causation from Sheldon’s strivings questionnaire, applied to the six components of CRM. This measure will examine the degree to which values embedded within the CRM are concordant with the personal values of participants. It will also assess the extent to which strivings toward the CRM practices are done for autonomous versus controlled (externally regulated) reasons.
Self-reported transfer
CRM-VQ
This measure also includes items that assess the implementation intentions and self-reported success of acting purposefully toward the valued directions encompassed within the CRM. Items include ‘to what extent have you made specific plans about when, where, and how to put this value into play?’ and ‘In the last 12 weeks, I have been this successful in living this value,’ to which responses will be elicited on a five-point Likert scale. An additional item relating to anticipation of implementation barriers (‘to what extent have you anticipated possible distractions and obstacles to putting this value into play?’) has been included and will be rated in the same manner. This item will allow us to investigate the impact of deliberately addressing implementation barriers (control condition).
Evidence of transfer within the workplace
Transfer indicators include time to implementation and maintenance of change. Transfer indicators will replicate those used previously by the research team [51]. We will seek direct evidence of implementation of any of the specific tools with the CRM (known as LifeJET) that are used to structure and record the recovery-focused practice of mental health workers. That is, examples of completed LifeJET documents within participant files. Time to implementation is calculated by determining the number of days lapsed between the date of training and the first date an example of completed LifeJET protocols was identified for participants. Maintenance of change is calculated by determining the proportion of staff work samples that are evidencing transfer (i.e., completed examples of CRM) after 12 months in comparison to time periods shortly after training (e.g., one month, six months).
Objective audit of participant work samples for quality and overall transfer
A further objective audit is undertaken using an enhanced version of the Goal Instrument for Quality to determine whether principles of effective goal setting that underpin the CRM have transferred into practice following intervention. That is, the former transfer indicators ask ‘is the new practice being done, and to what extent?’ while this measure will enable exploration of the question ‘is there improvement in the quality of what is being done?’ This audit is conducted on-site by trained in-service assessors, and will allow investigation of the overall work practices of staff for evidence of changed practice. Client files will be randomly selected from three time-periods (i.e., zero to six months pre-intervention, zero to six months post-intervention, and six to twelve months post-intervention) from participants in each organisation. The care plans within files will be assessed on 17 elements of effective recovery-based goal setting by two in-service auditors. A copy of the de-identified file material is forwarded to the research team, who will also conduct the 17-point assessment of the care plan, enabling inter-rater reliability testing.
Qualitative assessment of concordance between work samples and CRM values
A novel aspect of this research is the utilisation of a process-oriented protocol to assess worker adherence to the principles embedded within CRM. For example, the principle of personalisation (i.e., evidence of unique and person-centred approach to service delivery) is acknowledged as important in recovery [3]. Personalisation will be assessed across two dimensions (the content in the sample reflects unique expression of ideas; the language/presentation of content reflects individuality) using a three-point scale (0 = no, 1 = partially, 2 = yes). De-identified research copies of work samples are to be forwarded by participating mental health workers on a monthly basis by an on-site coordinator. Following power calculations, an online number generator will be used to facilitate random identification of an appropriate number of staff whose work-samples are to be reviewed. The samples of randomly identified staff will be reviewed using the six-point rating system at three time points (one month post-training, six months post-training, and 12 months post-training). Two trained assessors from within the research team will independently review the work samples for worker process and satisfactory inter-rater reliability will be established. We will compare observed adherence to CRM principles with the self-reported transfer and measures of autonomy obtained via the CRM-VQ (described above) at matching time periods (i.e., six and 12 months) during booster sessions to explore these differing elements of implementation.
Data analysis
The intervention analyses will focus on two major questions: What aspects of Transfer does the values interventions positively influence? By what processes does the intervention work? We are primarily interested in investigating effects at the cluster-level, though will explore overall effects of the addition coaching-component on the entire sample and participant-level data in the qualitative processes described. The figure below presents a model of the analyses. Model A represents the total effect of values intervention versus control condition (implementation) (X) on Transfer (Y). Model B represents the direct effect of X on Y, and the indirect effect through the mediator (M), our psychological process variable (autonomy).
Model C is a multiple mediation model, and will allow us to test the extent that our intervention targets multiple process variables (e.g., value importance, value success, value commitment). Contemporary research in the area of autonomous motivation has begun to challenge the utility of aggregated measures as outlined above [e.g., 25]. Rather than being mutually exclusive or ‘either or’ constructs, it may be more relevant to investigate the particular elements of motivation as described on the continuum outlined above (Figure 2). This seems especially relevant in this case given the specific goals individuals are striving toward are not self-generated and occur in the workplace where ‘enjoyment’ is not necessarily a motivational force that is amenable or desirable as a target for change. So, we will also be able to test effects of various regulators (i.e., guilt, enjoyment, importance, fun) on the outcome variable. We will use the bootstrapping method described by Preacher and Hayes [53] to test the meditational models.
We will deal with missing data using full-information-maximum-likelihood estimation (FIML). Traditional approaches to missing data (e.g., list-wise or pair-wise deletion) can lead to considerable bias in parameter estimates. In contrast FIML provides a superior approach to dealing with missing data that uses all the available information for parameter estimation [54].
Standard multiple regression analysis will test for predictors of transfer. In terms of knowledge transfer related to the new workplace practices, we will compare current findings with the pre-post effect sizes extracted from the previous work conducted by the research team [51]. In that study, effect sizes were moderate based on Cohen’s criteria (STARS = 0.25, RKI = 0.52) with a sample size of 75. Given the additional components of this intervention we are expecting slightly higher effect sizes. Data collected in the CRM-VQ will enable comparison of these predictors with participant self-reported implementation of the new work practices.
Work-sampling audits will also be used as an objective measure of transfer. We are anticipating a sample size of 200 at time point one, but have allowed for a more modest sample size of 100 in calculating power analyses. In our previous study, we found 37% of people transferred training. With this information we are able to estimate the percentage of people who need to demonstrate transferred practices in work samples in order to show a significant increase in the proportions between the former and present studies. Using Z test for proportions we have calculated that we need at least 48 out of the 100 participants to obtain a significant difference at p<0.01 to detect a 10% increase in transfer. We believe this difference is very achievable with the addition of the values-focused interventions.