Pediatric eMental healthcare technologies: a systematic review of implementation foci in research studies, and government and organizational documents
© The Author(s). 2017
Received: 11 October 2016
Accepted: 12 June 2017
Published: 21 June 2017
Researchers, healthcare planners, and policymakers convey a sense of urgency in using eMental healthcare technologies to improve pediatric mental healthcare availability and access. Yet, different stakeholders may focus on different aspects of implementation. We conducted a systematic review to identify implementation foci in research studies and government/organizational documents for eMental healthcare technologies for pediatric mental healthcare.
A search of eleven electronic databases and grey literature was conducted. We included research studies and documents from organization and government websites if the focus included eMental healthcare technology for children/adolescents (0–18 years), and implementation was studied and reported (research studies) or goals/recommendations regarding implementation were made (documents). We assessed study quality using the Mixed Methods Appraisal Tool and document quality using the Appraisal of Guidelines for Research & Evaluation II. Implementation information was grouped according to Proctor and colleagues’ implementation outcomes—acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability—and grouped separately for studies and documents.
Twenty research studies and nine government/organizational documents met eligibility criteria. These articles represented implementation of eMental healthcare technologies in the USA (14 studies), United Kingdom (2 documents, 3 studies), Canada (2 documents, 1 study), Australia (4 documents, 1 study), New Zealand (1 study), and the Netherlands (1 document). The quality of research studies was excellent (n = 11), good (n = 6), and poor (n = 1). These eMental health studies focused on the acceptability (70%, n = 14) and appropriateness (50%, n = 10) of eMental healthcare technologies to users and mental healthcare professionals. The quality of government and organizational documents was high (n = 2), medium (n = 6), and low (n = 1). These documents focused on cost (100%, n = 9), penetration (89%, n = 8), feasibility (78%, n = 7), and sustainability (67%, n = 6) of implementing eMental healthcare technology.
To date, research studies have largely focused on acceptability and appropriateness, while government/organizational documents state goals and recommendations regarding costs, feasibility, and sustainability of eMental healthcare technologies. These differences suggest that the research evidence available for pediatric eMental healthcare technologies does not reflect the focus of governments and organizations. Partnerships between researchers, healthcare planners, and policymakers may help to align implementation research with policy development, decision-making, and funding foci.
KeywordseHealth Mental health Implementation science Healthcare planning Organizational innovation Decision-making Healthcare organizations
The global prevalence of mental disorders in children and adolescents is reported to be as high as 30% [1–4]. Under-diagnosis and under-treatment of childhood mental disorders are well-documented concerns [2, 5–7]. The current distribution, demand, structure, and costs that underpin pediatric mental healthcare services make them relatively unavailable to many of those who need them . Electronic mental healthcare (eMental healthcare) technologies, which broadly include Internet-, mobile-, and, computer-based programs and resources as well as mobile phone applications, are considered promising approaches to enable more efficient use of mental healthcare resources, lower access barriers to traditional face-to-face mental healthcare, and provide flexibility in terms of standardization and personalization, interactivity, and consumer engagement [9–14].
Researchers (e.g., those developing and/or evaluating eMental healthcare technologies), healthcare planners (e.g., administrators in agencies either using or desiring to use eMental healthcare technologies), and policymakers (e.g., individuals with authority to set eMental healthcare policy for a healthcare organization or system) all convey interest and a sense of urgency in using eMental healthcare technologies to improve pediatric mental healthcare availability and access. To date, these three stakeholder groups have focused on different aspects of implementation. Researchers have studied user satisfaction [13, 15, 16] to determine that eMental healthcare technologies are acceptable to children and adolescents, and their parents and mental healthcare professionals. Healthcare planners and policymakers have discussed issues such as the cost and feasibility of eMental healthcare technologies to deliver pediatric mental healthcare . Current priorities that are relevant to all three stakeholders groups are generating evidence to demonstrate how eMental healthcare technologies can be optimally incorporated within an existing healthcare system and how technology implementation can be supported within an organization or system (e.g., governance, policy, funding) [18–20]. These priorities may be optimally achieved through collaborations between researchers, healthcare planners, and policymakers that aim to generate evidence for integrating and supporting eMental healthcare technologies in healthcare systems. To provide recommendations for such collaborations, we conducted a systematic review to identify what aspects of implementation have been studied and reported on for pediatric eMental healthcare technologies and what implementation goals/recommendations are present in government and organizational documents relating to eMental healthcare technologies for pediatric mental healthcare. Our aim was to identify and compare current areas of focus among research studies and government/organizational documents and to use these areas of focus to propose recommendations for implementation research, policies, and funding.
We systematically reviewed the literature to identify research studies and government and organizational documents with information regarding the implementation of eMental healthcare technologies for children and adolescents. We used a protocol that was developed a priori to define the objective, outline the search strategy, establish selection (inclusion/exclusion) criteria, determine implementation findings, guide the data collection process, and define the analysis. We followed the PRISMA statement checklist for reporting .
A research librarian developed and implemented the systematic search strategies using language (English) restrictions. The search was conducted in 11 electronic bibliographic databases: Medline, CINAHL, Embase, EBM Reviews, ProQuest Theses and Dissertations, Ovid MEDLINE In-Process & Other Non-Indexed Citations, OVID HealthStar, Cochrane Database of Systematic Reviews, Health Technology Assessment Database, ACP Journal Club, and SocIndex. The final Medline strategy is provided (see Additional file 1). Search terms focused on population and technology parameters used to screen for study eligibility. We also included terms related to “attitudes,” “preferences,” and “diffusion of innovation.” Thus, the search strategy was broad in order to identify potentially eligible studies that may not have been indexed using specific implementation science terms. The search was executed in each database from inception to September 30, 2015. Although the search was executed from inception to September 30, 2015, we restricted inclusion to studies and documents published after 2005.
To identify unpublished research and research-in-progress, we searched Google, the U.S. National Institutes of Health Clinical Trials database, the Australian New Zealand Clinical Trials Registry, the International Clinical Trials Registry Platform, and the UK Clinical Trials Gateway. To identify government and organizational documents, we conducted a two-pronged search: (1) targeted Google searches for relevant government, health, and technology organizations having clearly stated goals and/or funding relating to eHealth and behavioural technologies and (2) recommendations from members of our team. Overall, we created a list of 38 government and organizational websites, which was reviewed by the research team (see Additional file 2 for the full list). We also sought documents from key contacts responsible for leadership, policy, research, and information technology considered to be influential in the use of eHealth technologies. These contacts represented Canada’s Mental Health Commission e-mental health steering committee and the Ontario Centre of Excellence for Child and Youth Mental Health, the Australian Government’s Mental Health Commission and Department of Health and Ageing e-mental health expert advisory committee, the Netherland’s Dutch Association of Mental Health and Addiction Care, New Zealand’s National Health IT Board, and United Kingdom’s National Collaborating Centre for Mental Health and Mental Health Network. Reference lists of included studies and documents were also searched.
Criteria for including studies and documents for this review
We restricted the study and government/organizational document inclusion to countries from the largest English speaking eHealth markets . These countries were Australia, Canada, the Netherlands, New Zealand, the United Kingdom, and the USA. Studies of any design were eligible for inclusion.
Population of interest
Studies and government/organizational documents that included children and/or adolescents (0–18 years) as participants (studies only) or a population of interest (documents) were considered for inclusion. Government and organizational documents that focused on eMental healthcare technologies for all ages (including children and/or adolescents) were also eligible for inclusion.
eMental healthcare technology
Studies and government/organizational documents were eligible for inclusion if they evaluated/focused on eMental healthcare technology that met our definition: Internet-, computer-, or mobile-based programs and applications (‘apps’). Studies and documents of eMental healthcare technologies focused exclusively on phone calls or teleconferencing were excluded from the review as these technologies did not meet our definition of eMental healthcare. Studies and documents that focused on eMental healthcare technology use with parents of children with a mental health need or pediatric healthcare professionals were eligible for inclusion.
Implementation findings, goals, and/or recommendations
All outcomes, goals, and recommendations relating to implementation were considered. We used Proctor and colleagues’ eight outcome categories for implementation research  as a framework to identify outcomes, objectives, goals, and recommendations of interest in the studies and documents. To be included, a study needed to evaluate and report on at least one of the eight categories, and a document needed to contain at least one goal and/or recommendation that related to a category. The eight categories are as follows: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability . These outcomes have shaped traditional mental health services integration in routine care [24–28] and provide a common taxonomy for examining eMental healthcare technology implementation. For the purpose of this review, the categories were defined as follows: acceptability, a measure of satisfaction with the technology (including attitudes, functionality, preferences, and user experience); adoption, the intention, initial decision, or action to try or employ technology (e.g., uptake and utilization); appropriateness, the perceived fit, relevance, usefulness/helpfulness, or compatibility of the technology for a given practice setting, professional, or user and/or perceived fit of a technology to address a particular issue or problem; cost, the financial impact of an implementation effort (including measures of cost-effectiveness or cost-benefit); feasibility, the extent to which a technology could be successfully used or carried out within a setting (including utility, compatibility, and barriers); fidelity, the degree to which a technology was implemented as it was intended such as adherence; penetration, the integration of a practice within a service setting and its subsystems (e.g., “spread” or “reach”); and sustainability, the extent to which a newly implemented treatment is maintained or integrated within a service setting’s ongoing, stable operations .
Screening for eligibility
Studies and documents were organized and screened using EndNote X7.2.1. Studies and documents were first screened at the title and abstract level (stage 1 screening) to determine whether they met the inclusion criteria. At stage 1, two reviewers (NDG, AS) independently screened the title and abstract for the first 100 studies/documents in the library and subsequently calculated inter-rater agreement with the kappa statistic . The agreement was not sufficiently high (Cohen’s kappa, κ = 0.70), and we determined that “implementation” was an unclear term to aid in determining whether studies and documents provided implementation information on implementation outcomes, objectives, goals, and/or recommendations. We introduced Procter’s implementation outcomes framework  to the review protocol at this time, and two reviewers independently screened another 100 studies/documents in the library. This screening resulted in “almost perfect agreement” (Cohen’s kappa, κ = 0.84) , which indicated consensus on the definition of implementation. The remaining studies/documents in the library were then divided in two with each reviewer taking a respective half to screen using the title and abstract. Any studies/documents where it could not be determined whether they met inclusion criteria using the article’s title and abstract progressed to a review of the full-text (stage 2 screening) to determine eligibility. Any discrepancies were discussed between the reviewers and taken to a third party (ASN) if no agreement could be reached.
Data were extracted into a standardized form (Microsoft Excel; Microsoft, Redmond, Washington, USA). Extracted data were: (1) key article characteristics (e.g., author, date of publication, country); (2) study or document objectives; (3) technology type(s) (Internet-, mobile-, or computer-based) and services/treatments delivered; (4) details about research study design; (5) target population/study participants; and (6) implementation outcomes and findings from studies and implementation goals and/or recommendations from government/organizational documents according to Proctor et al. (acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability) . Included studies and documents were divided between two reviewers (NDG, AS) who extracted the data from their respective half and then checked the other’s extraction for accuracy and completeness. Discrepancies were resolved by discussion and/or by contacting corresponding authors of included studies/documents for clarification.
The quality of the research studies was assessed using the Mixed Methods Appraisal Tool (MMAT) . The MMAT is applicable to quantitative, qualitative, and mixed methods studies. The scoring scale ranges from 0 (low quality) to 100 (high quality) and has been pilot tested for reliability in systematic reviews . Ratings are specific to particular methodologies and are based on control of confounding factors, completeness of outcome data, minimization of selection bias, representativeness of sample, appropriateness of measures, response and withdrawal rates, appropriateness of study design to answer the research questions, and consideration of limitations. Two reviewers (NDG, AS) independently completed the MMAT for each included study and inter-rater agreement was considered “almost perfect” (Cohen’s kappa, κ = 0.81) . Discrepancies were resolved by a third party (ASN).
The quality of government and organizational documents was assessed using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) . AGREE II assesses six domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. While originally intended for clinical practice guidelines, most domains are applicable to government and organizational documents. Two domains, “clarity of presentation” and “applicability,” have criteria specific to guideline recommendations; however, because we were interested in document recommendations, these domains were applicable to our use. The AGREE II provided an overall quality score from 0% (low quality) to 100% (high quality), and a recommendation as to whether the document was recommended: (1) for use, (2) for use with modifications, or (3) not for use. Two reviewers (NDG, AS) independently completed the AGREE II for each document. Domain and overall quality scores were calculated by summing up the scores given by two reviewers . Recommendations for document use were decided by consensus between the two reviewers. Discrepancies in recommendations for document use were resolved by a third party (ASN).
Data analysis followed two approaches. First, two reviewers (NDG, AS) conducted a narrative (descriptive) synthesis  to produce a summary of the research studies and government/organizational documents included in the review. This summary encompassed the five domains for which we extracted data. Second, implementation data from each research study and government/organizational document were grouped according to the eight implementation outcomes framework . A cell remained empty if there was no relevant data pertaining to an implementation outcome. Data categorization was reviewed and discussed by NDG, ASN, and LW until consensus was achieved, and all data were coded into appropriate outcomes.
As a final step, a template approach to text analysis  was undertaken by two reviewers (NDG, AS). This approach involved bringing together the narrative synthesis and grouped implementation data under the implementation outcomes framework. This analytic step allowed the research team to identify the implementation foci of research studies as compared to government/organizational documents so that recommendations for implementation research, policies, and funding could be formulated.
Literature search and selection
Research study and government/organizational document characteristics
Government and organizational reports on eMental healthcare implementation
Report (year, country)
Children and adolescents
A Way Forward: Equipping Australia’s Mental Health System for the Next Generation (2015, AUS) 
1. To examine the current and future states of mental health and mental health service provision in Australia, specifically in terms of cost-effectiveness
- Mental Health Commission
E-Mental Health Services in Australia 2014: Current and Future (2014, AUS) 
1. To propose achievable and scalable solutions to unlock the even greater potential of eMental health for the community
- Mental Health Commission
Strategies for Adopting and Strengthening E-mental Health (2014, AUS) 
1. To identify the benefits, issues and obstacles for the provision of online health services in the mental healthcare system
2. To determine what investment and development, including infrastructure and processes, is required for improved integration of evidence-based online mental health services
- Mental Health Commission
E-therapies Systematic Review for Children and Young People with Mental Health Problems
(2014, UK) 
1. To determine the availability of computer-based applications on the Internet for children and young people with mental health problems
2. To determine the acceptability of programs and to investigate aspects of concern and value to young people
The Future’s Digital: Mental Health and Technology
(2014, UK) 
1. To examine the case for digital transformation
2. To determine how digital technology is currently being used in the design and delivery of mental health services
3. To determine what actions national bodies and individual professionals, as well as commissioners, take to ensure digital technology is leveraged to its full potential
- Healthcare professionals
E-mental Health in Canada: Transforming the Mental Health System using Technology (2014, CAN) 
1. To examine the spectrum of current eMental health technology/tools
2. To determine the key considerations examining the transformational potential of and barriers to the use of eMental health
3. To examine national and international eMental health approaches, including promising-practices
4. To provide recommendations for the future of eMental health
- Healthcare professionals
- Funders (i.e., insurers)
- eMental healthcare technology developers
E-mental Health in the Netherlands (2013, NED) 
1. To showcase how eMental Health utilizes technological developments to respond to today’s challenges, while at the same time increasing the number of people in reach of mental healthcare, thus decreasing the treatment gap
- Individuals at all levels in the Netherlands
Using Technology to Deliver Mental Health Services to Children and Youth in Ontario (2013, CAN) 
1. To engage with policy- and decision-makers to identify their perspectives on eMental Health
2. To conduct a review of the literature on eMental Health
3. To provide recommendations
- Healthcare professionals
The eHealth Readiness of Australia’s Medical Specialists (2011, AUS) 
1. To explore whether Australian medical specialists are ready to adopt and use eHealth technologies and solutions today and in a way consistent with policy direction in the future
2. To identify the barriers impacting eHealth readiness and adoption and explore how to minimize them
3. To identify eHealth enablers and explore how to apply them to drive adoption and effective usage
- Healthcare professionals
Research studies that have examined the implementation of eMental healthcare technologies, listed in order of publication date
Author (year, country)
Implementation of technology
Participants and setting
Individuals studied as part of implementation evaluation
Hetrick et al.
(2015, AUS) 
Online monitoring tool of depressive symptoms, suicidality, and side effects (via iPad).
Adolescents completed the tool once a week for up to 3 months. They could fill in the tool at any location with Internet access at any time with the exception of suicidal ideation items; these items were completed at the beginning of their regular treatment session with their clinician on an iPad.
Clinicians received a chart of scores after 4 weeks so that they could share with their patient and received an email regarding side effects that were endorsed for immediate action.
Adolescents, aged 14–24 years, receiving mental healthcare at the Youth Mood Clinic (YMC)
Mental health clinic
Reuland et al.
(2014, USA) 
CBM-I (Cognitive Bias Modification for Interpretation)
Online intervention where adolescents were instructed to read and imagine themselves in 50 scenarios per session that were ambiguous in meaning until a word fragment near the end of the scenario resolved the ambiguity in a positive way (e.g., in a way inconsistent with socially anxious beliefs)
Socially anxious adolescents, aged 10-15 years, and their mothers
No specific setting
Gonzales et al.
(2014, USA) 
Young people, aged 12–24 years, receiving outpatient or residential substance abuse treatment
Outpatient and residential substance abuse treatment programs
Gladstone et al.
(2014, USA) 
CATCH-IT: Competent Adulthood Transition with Cognitive-behavioural Humanistic and Interpersonal Training
14 online modules of Internet training to teach adolescents how to reduce behaviors that increase depressive disorders. Modules use CBT, behavioral activation, interpersonal psychotherapy, and community resiliency concept model.
Young people, aged 14–21 years, with a general primary care concern
Primary care settings
Eisen et al. (2013, USA) 
Primary care professionals and young people, aged 14–21 years, with a general primary care concern
Primary care settings
Iloabachie et al.
(2011, USA) 
Young people, aged 14–21 years, with a general primary care concern who had positive screens for sub-threshold depressive symptoms, and parents of those who were <18 years
Primary care settings
Fothergill et al.
(2013, USA) 
Online screener (via computer or tablet)
25 questions, that can branch into as many as 57 questions based on responses, regarding somatic and mental health concerns, general health risk, anxiety, and parental depression. The screener calculates scores for the validated scales it contains. A summary screen tallies the positive responses within broad categories and highlights scores for the validated assessments above the standard cut-offs
Primary care professionals and parents presenting for a well child visit
Primary care settings
Branson et al.
(2013, USA) 
Reminders sent the evening before each scheduled therapy session (e.g., “C u Wed @8”)
Adolescents, aged 13–17 years
Hospital-based outpatient mental health clinic
Han et al.
(2013, USA) 
Toolkit on the MDPC Website
Health questionnaire (PHQ-9), education material for patients, guides to diagnostic and treatment approaches, specialty care referral forms, slide presentation, training manuals, publications, cost calculator (investment savings for employers)
Healthcare professionals using the MDPC website
Salloum et al. (2013, USA) 
Camp Cope-A-Lot: cCBT program for childhood anxiety within community mental health centers
Therapist provides monitoring and coaching as the child completes the program. The therapist is present during program completion to answer any questions and build therapeutic alliance. 12 weekly sessions: sessions 1 to 6 focus on skill-building and sessions 6 to 12 are exposure-based sessions where the therapist provides direct coaching
Children aged 7–13 years, with an anxiety disorder, their parents, administrators, study therapists
Community-based mental healthcare settings
Merry et al.
(2012, NZ) 
SPARX (Smart, Positive, Active, Realistic, X-factor thoughts)
Interactive fantasy game designed to deliver cognitive behavioural therapy for the treatment of clinical depression. 7 modules delivered over a period of 4 to 7 weeks. A “guide” puts the game into context, provides education, gauges mood, and sets and monitors real-life challenges
Adolescents, aged 12–19 years, seeking help for mild to moderate depressive symptoms that were assessed by a clinician as being fit for self-help and not being a high risk for suicide or self-harm
Primary care settings
Ahmad et al.
(2012, CAN) 
Computer-assisted interactive health risk assessment tool
A health risk assessment tool that provides feedback to both the patient and the physician on psychosocial health. The tool considers the contextual details of the patient (e.g., violence, drug or alcohol abuse)
Healthcare professionals (nurses, physicians, social workers, etc.)
Primary and acute care settings
Murphy et al.
(2011, USA) 
Electronic Outcomes Rating Form (e-ORF) in conjunction with a web-based patient tracking system
The e-ORF is an electronic form filled out by parents of all intake patients using a digital pen. The form includes assessment tools (BPRS-C, CGAS). The e-ORF automatically prints outcome forms of routine paperwork for intake visit and follow-up forms every 90 days to reduce burden on the administrative and clinical staff. The digital pens have the ability to enter the assessment data directly into the hospital’s database
Children and adolescents ≤18 years undergoing outpatient mental health evaluation
Outpatient child psychiatric clinic
Diamond et al.
(2010, USA) 
BHS (Behavioural Health Screen)
Screening tool assesses risk behaviors and psychiatric symptoms in 13 modules. Patient completes the BHS in a waiting room, the report printed at primary care office, and the summary of assessment given to physician
Adolescents with a general primary care concern
Primary care settings
Fein et al.
(2010, USA) 
BHS-ED (Behavioural Health Screen–Emergency Department)
Psychosocial assessment tool designed for adolescents in non-psychiatric medical settings. Nurses or medical technicians logged the patient onto the website and registered them with a password and medical record number. The BHS-ED began with a slide and audio show that explained the rationale for the screening and the standard limits of confidentiality
Adolescents, aged 14–18 years, without acute or critical injuries or illness, presenting with non-psychiatric symptoms
Emergency department of an urban tertiary care children’s hospital
Stallard et al.
(2010, UK) 
Focus was whether mental health professionals would consider the delivery of CBT via computer technology
Mental health professionals
National conference, British Association of Behavioral and Cognitive Psychotherapy
Pretorious et al.
(2010, UK) 
Web-based CBT for bulimic disorders
8 interactive, multimedia sessions, electronic message board for participants and parents, and email support provided by therapist (flexible weekly support and advice via email)
Young women, aged 16–20 years, with bulimia nervosa or atypical bulimia nervosa
Horwitz et al.
(2008, USA) 
CHADIS (Child Health and Development Interactive System)
CHADIS provides access to 23 different questionnaires and asks parents prioritize their concerns so clinicians can plan agenda for the upcoming appointment
Parents of children <8 years presenting for a well-child visit and pediatricians
Primary care and community-based mental healthcare settings
John et al.
(2007, USA) 
Personal digital assistant (PDA) decision support system (DSS)
Screening questions supporting the PDA application: Short Mood and Feeling Questionnaire (SMFQ) and four additional questions, two related to family history of depression and two related to suicide
Pediatric Advanced Practice Nursing students treating children aged 8 to 18 years
University-based medical centre
Hanley et al.
(2006, UK) 
Online counseling services for youtha
Implementation outcomes investigated by research studies and addressed/recommended in government and organizational documents
Author (year, country)
Implementation outcome investigated
Hetrick et al. (2015, AUS) 
Gonzales et al. (2014, USA) 
Reuland et al. (2014, USA) 
Gladstone et al. (2014, USA) 
Eisen et al. (2013, USA) 
Fothergill et al. (2013, USA) 
Han et al. (2013, USA) 
Salloum et al. (2013, USA) 
Branson et al. (2013, USA) 
Ahmad et al. (2012, CAN) 
Merry et al. (2012, NZ) 
Murphy et al. (2011, USA) 
Iloabachie et al. (2011, USA) 
Diamond et al. (2010, USA) 
Fein et al. (2010, USA) 
Pretorious et al. (2010, UK) 
Stallard et al. (2010, UK) 
Horwitz et al. (2008, USA) 
John et al. (2007, USA) 
Hanley et al. (2006, UK) 
Implementation outcome addressed/recommended
Government and organizational documents
Reach Out (2015, AUS) 
eMHA (2014, AUS) 
NCCMH (2014, UK) 
MHCC (2014, CAN) 
Sax (2014, AUS) 
MHN (2014, UK) 
OCE (2013, CAN) 
GGZ (2013, AUS) 
DHA (2011, AUS) 
Fourteen research studies (70%) examined acceptability (ten quantitatively [46, 49–54, 56, 58, 64], three [48, 59, 62] qualitatively, and one using both qualitative and quantitative methods ). The majority of studies were of good or excellent quality [45, 46, 51–54, 56, 58, 59, 62, 64]; three were of poor quality [48–50]. How acceptability was defined varied considerably across studies: satisfaction with the technology [46, 56, 64], functionality of the technology , attitudes towards the technology , technology preferences [51, 59], user experience using the technology [52, 56, 58, 59], and acceptability of the technology [45, 48, 54]. Studies of acceptability reported that participants responded favorably to eMental healthcare technologies [45, 46, 48–50, 53, 54, 58, 62, 64] and liked that they provided autonomy, convenience, anonymity, and accessibility [51, 52, 56, 59, 62, 64]—particularly when technologies were designed specifically for the youth [54, 56, 64].
Five government/organizational documents (56%) included goals or recommendations relating to the acceptability of eMental healthcare technology. Documents were mainly of medium quality (range 58–83%) [17, 37, 39, 44]; although, one document was of high quality . These documents described the need for better incorporation of consumer preference during design planning to ensure such services are user-centered and individualized [17, 39, 43, 44] and emphasized the need to increase public awareness of the acceptability of eMental healthcare technology [17, 37, 43].
Five studies (25%), of good or excellent quality, measured adoption of eMental health technology for pediatric mental healthcare [46, 47, 53, 55, 57]: pre-post measurements of treatment attendance , screening rates [47, 55], number of intake appointments , and uptake into healthcare practice . Studies showed that children and adolescents receiving the eMental healthcare technology demonstrated significantly higher rates of attendance , and electronic technologies were associated with improved clinician completion rates ; however, studies concerned with healthcare professional adoption showed moderate screening rates and uptake into practice [47, 53, 55]. Government and organizational documents did not address actual eMental healthcare technology adoption; although, documents did discuss the intent to use technology to provide mental healthcare.
Appropriateness was examined in ten studies (50%) [46, 47, 50, 52, 59–64]. One study was of poor quality , and the remaining were of good or excellent quality [46, 47, 52, 59–64]. The definition of appropriateness varied considerably across studies: appropriateness [46, 47, 50, 52, 60, 61, 63], relevance [59, 63], usefulness [59, 61–64], suitability [59, 61, 64], and perceived fit . Of these studies, five measured this construct qualitatively [47, 52, 59, 60, 62], four quantitatively [46, 50, 63, 64], and one using both qualitative and quantitative measures . Most studies examined the appropriateness of eMental healthcare technology for healthcare professionals and their settings [46, 47, 50, 52, 60–63], while some studies examined the appropriateness of the technology for children and adolescents [59, 61, 64]. Computer-based treatments and online management systems were deemed appropriate to mental healthcare practices in that the technologies allowed for less preparation time and provided facilitation and appointment planning [52, 62]. Adolescents found eMental healthcare helpful for improving appointment attendance , and one study demonstrated healthcare professional competency and compatibility at employing the technology within practice . Healthcare professionals regarded a tool’s use and appropriateness as very important for successful implementation  and believed eMental healthcare had the potential to be helpful for children and adolescents, especially for mild to moderate problems . However, some studies showed that healthcare professionals perceived interference with the therapeutic relationship due to eMental healthcare technologies [47, 60]. Similarly to acceptability, the accessibility, flexibility, and anonymity of eMental healthcare were factors that influenced treatment preference for web-based over face-to-face interventions  and eMental healthcare was at least as good a treatment as usual in primary healthcare sites .
Five government and organizational documents (56%) addressed appropriateness. One document was of high quality , while the remaining were of medium quality (range 58–83%) [17, 38, 42, 44]. Documents considered appropriateness from the perspectives that (1) additional research and development should be conducted to assure healthcare quality and safety standards are not compromised [17, 40, 42, 44], (2) new eMental healthcare technologies should be integrated and evaluated within existing health and technology policies (including IT aspects) [17, 40, 42, 44], and (3) technologies need to be accessible to rural, regional, and indigenous communities [38, 40] to ensure appropriateness to these populations.
Of the 20 research studies, one qualitative study (5%) of good quality considered cost issues from system and organization perspectives and described the need to address cost coverage in terms of who covers treatment costs and how to integrate third party payers .
All nine government and organizational documents (100%) cited cost as a major component of future eMental healthcare implementation efforts [17, 37–44]. One of these documents one was of low quality , four were of medium quality (range 58–83%) [37, 38, 42, 44], and one was of excellent quality . Across the government and organizational documents, start-up costs for developing or implementing technology into practice [17, 37, 38, 40], reimbursement to consumers from health insurance companies [17, 39, 44], the cost-effectiveness of blended care models , and billing requirements for healthcare professionals providing eMental healthcare services were discussed [17, 39, 44]. Overall, the government/organizational literature recommended establishing and evaluating a sustainable funding model to address high development and continuing maintenance costs [17, 37, 39, 42–44] and allocating more government funds to further support research and development .
Two studies (10%) quantitatively measured eMental healthcare technology feasibility [48, 63]. These studies were of poor quality  and excellent quality . The definition of feasibility did not vary between the studies. While one of the studies found eMental healthcare technology to be feasible , the other found that healthcare professionals perceived professional development and workload as feasibility challenges (for example, rate changes to workload may be required when adopting an eMental healthcare technology within a professional’s clinical workflow) .
Seven government/organizational documents (78%) of excellent [40, 43] and medium quality (range 58–83%) [17, 37, 38, 42, 44] described the need to address feasibility if implementation of eMental healthcare technologies are to be successful [17, 37, 38, 40, 42–44]. Document recommendations included ensuring training and education programs for healthcare professionals and organizations as a means to ensure technology feasibility. That is, although a technology may be appropriate for a given setting, it may not be feasible to implement it if resources and training are not available [17, 37, 38, 40, 42–44].
Three studies (15%) examined fidelity by quantifying adherence to the technology [48, 57] and reasons for non-completion . Two of these studies were of excellent quality [57, 64] and one was of poor quality . Studies reporting on fidelity found either perfect  or good [57, 64] adherence to the technology protocol, finding reasons for non-completion such as technical glitches and lack of time and/or interest .
Recommendations from seven government and organizational documents (67%) concentrated on the importance of fidelity in implementation evaluations [17, 38–41, 43, 44]. Of these documents one was of poor quality , four were of medium quality (range 58–83%) [17, 38, 39, 44], and two were of excellent quality [40, 43]. Government and organizational documents also described concerns about dropout and lack of follow-up with the use of eMental healthcare technologies [17, 38], and the need for evaluations to determine if such technologies are being delivered as intended and are as innovative as described [17, 38–41, 43, 44].
Penetration and sustainability
Penetration and sustainability of eMental healthcare technologies were not considered in the research studies included in this review, but were addressed in government and organizational documents. Eight documents (89%) described recommendations related to the penetration of eMental healthcare technologies [17, 37–40, 42–44], while six described sustainability recommendations [17, 38, 40, 42–44]. Six of the documents recommending penetration outcomes were of medium quality (range 58–83%) [17, 37–39, 42, 44] and two were of excellent quality [40, 43]. Four documents examining sustainability outcomes were of medium quality (range 58–83%) [17, 38, 42, 44], and two were of excellent quality [40, 43]. Penetration recommendations included linking traditional and eMental healthcare services together at multiple points using a stepped care model to allow for cross-referral [17, 37–40, 42, 43] and conducting further research, development, and evaluation in routine clinical settings [39, 44] with large sample sizes to ensure scalability . Sustainability recommendations included the need for policy reform by way of establishing standards for privacy and security [17, 40, 42, 44], devising a national eMental healthcare strategy/protocol and creating stricter governance to facilitate technology implementation [38, 42, 43], and using technology to foster collaboration .
There is an increasing interest and sense of urgency from the perspectives of researchers, healthcare planners, and policymakers to use eMental healthcare technologies in pediatric mental healthcare to improve healthcare availability and access. To date, however, these stakeholders have tended to focus on different aspects of implementation. If effective eMental healthcare services are to become a core component of routine service delivery, these different areas of focus need to be identified so that alignment of research, policies, and funding can occur. We undertook a systematic review to identify research studies and government and organizational documents that describe eMental healthcare technology implementation in healthcare systems for children and adolescents and explore what areas of implementation researchers, healthcare planners, and policymakers have historically focused on. This approach allowed us to identify areas of focus in the current eMental healthcare landscape in terms of implementation outcomes among different stakeholder groups and to use these areas of focus to propose recommendations for implementation research, policies, and funding. The takeaway points of this review are as follows:
Implementation foci differ between researchers, healthcare planners, and policymakers
We found multiple differences in implementation outcome foci between government and organizational literature and research studies. Consistent with other reviews [18, 65], the research studies in this review predominantly focused on patient and clinician outcomes (e.g., use, satisfaction, acceptability). Areas of focus in government and organizational reports were cost, penetration, and feasibility. Differences in underlying positions between stakeholder groups may lead to opposing criteria for successful eMental healthcare implementation and/or may jeopardize stakeholder engagement . The task of disseminating evidence on the success of these initiatives does not occur in separate asocial and apolitical bubbles . They are often produced by, and in turn feed back into, the political process of deciding priorities and allocating resources to pursue them.
Distinct knowledge holders often have differing mentalities; governing bodies are often highly incentivized by the lowest cost for the most efficient use of resources, while research tends to be incentivized by innovation and/or validating their work [68, 69]. Thus, the differences in underlying priorities may reflect different values and goals of governing bodies and researchers. For example, due to underlying incentives, government and organization priorities may naturally gravitate towards long-term benefits of eMental healthcare (i.e., penetration and sustainability) as this aligns with the mandate of cost versus benefits. Given that eMental healthcare is a relatively new field, research may be appropriately focused on establishing an eMental healthcare innovation as acceptable and appropriate to users before studying other implementation constructs. As the field matures, there is an opportunity to study implementation outcomes, such as cost, penetration, and feasibility, that would be of interest and value to governments and organizations [23, 70]. Moving forward, alignment of funding interests and partnerships between stakeholder groups may allow for the development of funding models to study outcomes that require longer-term investigation and substantial effort such as fidelity, penetration, and sustainability. Future research endeavors may also benefit from using a structured, theory-driven methodology to compile, evaluate, and integrate eMental healthcare information such as the Health Information Technologies—Academic and Commercial Evaluation (HIT-ACE) methodology .
A taxonomy for implementation outcomes is needed for the eMental healthcare technology field
Consistent with the broader literature, this review reflects a nonstandard usage of terminology and lack of consensus on a taxonomy relating to implementation metrics [72–74]. For example, there was overlap and inconsistency in the operationalization and discussion of appropriateness and acceptability; adoption was often used interchangeably with uptake; fidelity was often referred to as adherence; feasibility described as acceptability; and sustainability was reflected in the literature by varying, but congruent definitions, such as maintenance, incorporation, and integration [23, 75–78].
Findings from a recent review to identify instruments to measure implementation in mental healthcare included an uneven distribution of instruments across implementation outcomes . Although constructs such as acceptability and adoption are well recognized among instruments, other constructs are either underdeveloped or do not lend themselves easily to instrumentation, yielding few instruments to measure a range of implementation outcomes. More groundwork is needed to enable consistency in the definition and measurement of implementation constructs rather than considering them at an abstract level. This approach will help to align areas of focus and discussions between researchers, healthcare planners, and policymakers.
This systematic review has several limitations. First, our research study search results may be limited given “behavioral health” or its derivatives were not used in our search string. While not all eMental healthcare studies will be indexed solely with these terms, there may be some that are. These studies would not have been included in our review. Additionally, while our search to identify unpublished research and research-in-progress was extensive, we did not search other databases, such as the NIH Reporter, which may have yielded additional eMental health technology studies. While some of the studies in the NIH Reporter may have been additionally registered in the registries we searched, some may not have been.
Another limitation is our decision to restrict inclusion of government and organizational reports from countries with known expertise and knowledge in the area of eMental healthcare. Thus, this review’s findings may not generalize to low and middle-income countries where mental healthcare systems may be organized differently and technologies used in different ways. That none of our included government/organizational documents originated in the USA, while 15 of our research studies did, could display an unrepresented sample in the review, and may explain the lack of alignment between the groups of literature we studied. Further, not all of the government and organizational websites we reviewed may have made implementation information available on the web. It was not feasible for the research team to contact all organizations by email or phone to request information that may have been in paper-based form only.
That our review was limited to English language documents also limits the literature we included in the review, and thus the generalizability of the results. We also believe that there are eMental healthcare technologies being deployed in healthcare systems that have not been scientifically investigated; thus, important implementation data for these technologies would not be available for our review.
The decision to use Proctor and colleagues’ implementation outcomes as a guiding framework was not applied a priori. Therefore, specific implementation outcome terms were not included in our search string and could have resulted in potentially relevant studies not being identified in the search results if they were indexed solely using implementation outcome terms. However, the search strategy was developed using terms specific to our population and interventions of interest. Studies of eMental healthcare technologies for children and/or adolescents are more likely to be solely indexed according to these terms and may or may not include indexing with implementation terms. Our descriptive approach to synthesizing implementation may also be criticized; however, in doing so, we believe we were able to identify an important perspective of how eMental healthcare is currently being discussed at the governmental/organizational level alongside research developments.
Finally, the inconsistent use of terminology [72–74] across the literature in this review required us to make judgment calls regarding how to categorize implementation outcomes. There also remains conceptual ambiguity and overlap among implementation outcomes (e.g., acceptability and appropriateness), which could have resulted in some factors being arguably different constructs [23, 79]. However, the difficulty in grouping these metrics within the implementation outcomes further confirms the lack of agreement regarding constructs hypothesized to affect implementation success and the identifiable measures of these constructs .
This systematic review identified differing implementation foci between research studies and government/organizational documents, and a lack of consistent implementation taxonomies and metrics. These differences mean that the research evidence available to support eMental healthcare technologies for pediatric mental healthcare is not well aligned with the focus of governments and organizations. Using the results of this review as a guide, partnerships between researchers, healthcare planners, and policymakers may help to align implementation research, policies, and funding foci.
The authors would like to thank the librarian who helped develop the search strategy. The authors would also like to thank the corresponding authors of included studies who responded to inquiries requesting further information about their research.
Funding for this project was provided by the Canadian Institutes of Health Research (CIHR 201404KRS). Drs. Newton and Hartling hold the CIHR New Investigator Awards. Dr. McGrath held a Tier I Canada Research Chair.
Availability of data and materials
The data from included studies and reports analyzed during this review are available from the corresponding author on reasonable request.
All authors contributed extensively to the review and were involved in the design of this work. NDG, AS, and ASN were responsible for the screening, data extraction, and analysis. NDG, ASN, and AS were responsible for the initial drafting of the manuscript and LW, KB, AH, LH, MPD, and PM provided comments on it at all stages. All authors have read and approved the final version of the manuscript.
The authors declare that they have no competing interests.
Consent for publication
Ethics approval and consent for publication
This systematic review did not require ethics approval nor does it contain any individual person’s data in any form.
The authors have no financial relationships relevant to this article to disclose.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
- Belfer ML. Child and adolescent mental disorders: the magnitude of the problem across the globe. J Child Psychol Psychiatry. 2008;49(3):226–36. doi:https://doi.org/10.1111/j.1469-7610.2007.01855.x.PubMedView ArticleGoogle Scholar
- Waddell C, McEwan K, Shepherd CA, Offord DR, Hua JM. A public health strategy to improve the mental health of Canadian children. Can J Psychiatry. 2005;50(4):226–33. doi:https://doi.org/10.1177/070674370505000406.PubMedView ArticleGoogle Scholar
- Steel Z, Marnane C, Iranpour C, et al. The global prevalence of common mental disorders: a systematic review and meta-analysis 1980-2013. Int J Epidemiol. 2014;43(2):476–93. doi:https://doi.org/10.1093/ije/dyu038.PubMedPubMed CentralView ArticleGoogle Scholar
- Polanczyk GV, Salum GA, Sugaya LS, Caye A, Rohde LA. Annual research review: a meta-analysis of the worldwide prevalence of mental disorders in children and adolescents. J Child Psychol Psychiatry. 2015;56(3):345–65. doi:https://doi.org/10.1111/jcpp.12381.PubMedView ArticleGoogle Scholar
- Merikangas KR, He J, Brody D, Fisher PW, Bourdon K, Koretz DS. Prevalence and treatment of mental disorders among US children in the 2001-2004 NHANES. Pediatrics. 2010;125(1):75–81. doi:https://doi.org/10.1542/peds.2008-2598.PubMedView ArticleGoogle Scholar
- World Health Organization. Mental Health Atlas 2014. Geneva, Switzerland: World Health Organization; 2013. http://www.who.int/entity/mental_health/evidence/atlas/executive_summary_en.pdf?ua=1, Accessed 8 Jul 2016.Google Scholar
- Pelletier L, O’Donnell S, Dykxhoorn J, McRae L, Patten S. Under-diagnosis of mood disorders in Canada. Epidemiol Psychiatr Sci. 2016;6:1–10. doi:https://doi.org/10.1017/S2045796016000329.View ArticleGoogle Scholar
- Hickie IB, McGorry PD. Increased access to evidence-based primary mental health care: will the implementation match the rhetoric? Med J Aust. 2007;187(2):100–3.PubMedGoogle Scholar
- Riper H, Andersson G, Christensen H, Cuijpers P, Lange A, Eysenbach G. Theme issue on e-mental health: a growing field in internet research. J Med Internet Res. 2010;12(5):e74. doi:https://doi.org/10.2196/jmir.1713.PubMedPubMed CentralView ArticleGoogle Scholar
- Christensen H, Hickie IB. E-mental health: a new era in delivery of mental health services. Med J Aust. 2010;192(11 Suppl):S2–3.PubMedGoogle Scholar
- Christensen H, Hickie IB. Using e-health applications to deliver new mental health services. Med J Aust. 2010;192(11 Suppl):S53–6.PubMedGoogle Scholar
- Lal S, Adair CE. E-mental health: a rapid review of the literature. Psychiatr Serv. 2014;65(1):24–32. doi:https://doi.org/10.1176/appi.ps.201300009.PubMedView ArticleGoogle Scholar
- Boydell KM, Hodgins M, Pignatiello A, Teshima J, Edwards H, Willis D. Using technology to deliver mental health services to children and youth: a scoping review. J Can Acad Child Adolesc Psychiatry. 2014;23(2):87–99.PubMedPubMed CentralGoogle Scholar
- Hollis C, Morriss R, Martin J, et al. Technological innovations in mental healthcare: harnessing the digital revolution. Br J Psychiatry. 2015;206(4):263–5. doi:https://doi.org/10.1192/bjp.bp.113142612.PubMedView ArticleGoogle Scholar
- Burns JM, Birrell E, Bismark M, et al. The role of technology in Australian youth mental health reform. Aust Health Rev. 2016. doi:https://doi.org/10.1071/AH15115.PubMedGoogle Scholar
- Struthers A, Charette C, Bapuji SB, et al. The acceptability of e-mental health services for children, adolescents, and young adults: a systematic search and review. Can J Commun Ment Health. 2015;34(2):1. doi:https://doi.org/10.7870/cjcmh-2015-006.View ArticleGoogle Scholar
- Mental Health Commission of Canada. e-Mental health in Canada: transforming the mental health system using technology. A briefing document. Mental Health Comission of Canada. 2014. http://www.mentalhealthcommission.ca/sites/default/files/MHCC_E-Mental_Health-Briefing_Document_ENG_0.pdf. Accessed 13 Jan 2016.
- Meurk C, Leung J, Hall W, Head BW, Whiteford H. Establishing and governing e-Mental health care in Australia: a systematic review of challenges and a call for policy-focussed research. J Med Internet Res. 2016;18(1):e10. doi:https://doi.org/10.2196/jmir.4827.PubMedPubMed CentralView ArticleGoogle Scholar
- Whiteford H, Harris M, Diminic S. Mental health service system improvement: translating evidence into policy. Aust N Z J Psychiatry. 2013;47(8):703–6. doi:https://doi.org/10.1177/0004867413494867.PubMedView ArticleGoogle Scholar
- Young S. Evidence-based policy: a practical guide to doing it better. Pol Stud Rev. 2014;12(2):299–300. doi:https://doi.org/10.1111/1478-9302.12053_82.View ArticleGoogle Scholar
- Moher D, Liberati A, Tetzlaff J, Altman DG. Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the PRISMA statement. Int J Surg. 2010;8:336–41. doi:https://doi.org/10.1016/j.ijsu.2010.02.007.PubMedView ArticleGoogle Scholar
- World Health Organization. Atlas of eHealth country profiles: The use of eHealth in support of universal health coverage: based on the findings of the third global survey on eHealth 2015. Geneva, Switerland: World Health Organization 2016. http://apps.who.int/iris/bitstream/10665/204523/1/9789241565219_eng.pdf?ua=1. Accessed 18 Jun 2016.
- Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health. 2011;38(2):65–76. doi:https://doi.org/10.1007/s10488-010-0319-7.PubMedView ArticleGoogle Scholar
- Torrey WC, Bond GR, McHugo GJ, Swain K. Evidence-based practice implementation in community mental health settings: the relative importance of key domains of implementation activity. Adm Policy Ment Health. 2012;39(5):353–64. doi:https://doi.org/10.1007/s10488-011-0357-9.PubMedView ArticleGoogle Scholar
- McMillen JC, Lenze SL, Hawley KM, Osborne VA. Revisiting practice-based research networks as a platform for mental health services research. Adm Policy Ment Health. 2009;36(5):308–21. doi:https://doi.org/10.1007/s10488-009-022-2.PubMed CentralView ArticleGoogle Scholar
- Brooks H, Pilgrim D, Rogers A. Innovation in mental health services: what are the key components of success? Implement Sci. 2011;6(1):1. doi:https://doi.org/10.1186/1748-5908-6-120.View ArticleGoogle Scholar
- Southam-Gerow MA, Rodriguez A, Chorpita BF, Daleiden EL. Dissemination and implementation of evidence based treatments for youth: challenges and recommendations. Prof Psychol Res Pr. 2012;43(5):527–34.View ArticleGoogle Scholar
- Stanhope V, Tuchman E, Sinclair W. The implementation of mental health evidence based practices from the educator, clinician and researcher perspective. Clin Soc Work J. 2011;39(4):369–78. doi:https://doi.org/10.1007/s10615-010-0309-y.View ArticleGoogle Scholar
- Altman D. Practical statistics for medical research. London, UK: Chapman and Hall; 1991.Google Scholar
- Viera AJ, Garrett JM. Understanding interobserver agreement: the kappa statistic. Fam Med. 2005;37(5):360–3.PubMedGoogle Scholar
- Pluye P, Robert E, Cargo M, et al. A mixed methods appraisal tool for systematic mixed studies reviews. Montreal: McGill University; 2011. p. 1–8.Google Scholar
- Pace R, Pluye P, Bartlett G, et al. Testing the reliability and efficiency of the pilot Mixed Methods Appraisal Tool (MMAT) for systematic mixed studies review. Int J Nurs Stud. 2012;49:47–53. doi:https://doi.org/10.1016/j.ijnurstu.2011.07.002.PubMedView ArticleGoogle Scholar
- Brouwers MC, Kho ME, Browman GP, et al. AGREE II: advancing guideline development, reporting and evaluation in healthcare. Can Med Assoc J. 2010;182(18):E839–42. doi:https://doi.org/10.1503/cmaj.090449.View ArticleGoogle Scholar
- The AGREE Next Steps Consortium. Appraisal of Guideelines for Research & Evaluation II: AGREE II Instrument. 2013. http://www.agreetrust.org/wp-content/uploads/2013/10/AGREE-II-Users-Manual-and-23-item-Instrument_2009_UPDATE_2013.pdf. Accessed 5 Aug 2016.
- Popay J, Roberts H, Sowden A, et al. Guidance on the conduct of narrative synthesis in systematic reviews: a product from the ESRC methods programme. 2006.Google Scholar
- Crabtree B, Miller W. A template approach to text analysis: developing and using codebooks. In: Miller WL, editor. Doing qualitative research: research methods for primary care. Newbury Park, CA: Sage Publications, Inc.; 1992. p. 93–109.Google Scholar
- Hosie A, Vogl G, Carden J, Hoddinott J, Lim S. A way forward: equipping Australia’s mental health system for the next generation: Reach Out Australia EY. 2015. http://www.ey.com/Publication/vwLUAssets/Equipping-Australias-mental-health-system/$FILE/EY-Equipping-Australias-mental-health-system.pdf. Accessed 13 Jan 2016.
- The Sax Institute. Strategies for adopting and strengthening e-mental health: Mental Health Commission of New South Wales. 2014. http://nswmentalhealthcommission.com.au/sites/default/files/assets/File/Report%20-%20The%20Sax%20Institute%20E-Mental%20Health%20Evidence%20Review%20cover%20page.pdf. Accessed 13 Jan 2016.
- National Collaborting Centre for Mental Health. E-therapies systematic review for children and young people with mental health problems: National Collaborating Centre for Mental Health; MindEd E-portal Consortium; 2014. https://www.minded.org.uk. Accessed 13 Jan 2016.
- Boydell KM, Hodgins M, Pignatiello A, Edwards H, Teshima J, Willis D. Using technology to deliver mental health services to children and youth in Ontario: Ontario Centre of Excellence for Child and Youth Mental Health. 2013. http://www.excellenceforchildandyouth.ca/sites/default/files/policy_using_technology_0.pdf. Accessed 13 Jan 2016.
- GGZNederland. E-mental health in the Netherlands: Dutch Association of Mental Health and Addiction Care; 2013. http://www.ggznederland.nl/uploads/assets/20130514%20Factsheet%20eHealth.pdf. Accessed 13 Jan 2016.
- Mental Health Network. The future’s digital: mental health and technology: Mental Health Network NHS Confederation; 2014. http://www.nhsconfed.org/~/media/Confederation/Files/Publications/Documents/the-futures-digital.pdf. Accessed 13 Jan 2016.
- Christensen H, Proudfoot J, Woodward A, et al. e-mental health services in Australia 2014: current and future. 2014. https://emhalliance.fedehealth.org.au/wp-content/uploads/sites/42/2014/10/e-Mental-Health-in-Australia-2014.pdf. Accessed 13 Jan 2016.
- Department of Health and Ageing. The eHealth readiness of Australia’s medical specialists. Canberra, Australia: Commonwealth of Australia; 2011. http://www.health.gov.au/internet/publications/publishing.nsf/Content/ehealth-readiness-medical-specialists-toc. Accessed 13 Jan 2016.
- Gonzales R, Douglas Anglin M, Glik DC. Exploring the feasibility of text messaging to support substance abuse recovery among youth in treatment. Health Educ Res. 2014;29(1):13–22. doi:https://doi.org/10.1093/her/cyt094.PubMedView ArticleGoogle Scholar
- Branson CE, Clemmey P, Mukherjee P. Text message reminders to improve outpatient therapy attendance among adolescents: a pilot study. Psychol Serv. 2013;10(3):298–303. doi:https://doi.org/10.1037/a0026693.PubMedView ArticleGoogle Scholar
- John R, Buschman P, Chaszar M, Honig J, Mendonca E, Bakken S. Development and evaluation of a PDA-based decision support system for pediatric depression screening. Stud Health Technol Inform. 2007;129(2):1382–6.PubMedGoogle Scholar
- Reuland MM, Teachman BA. Interpretation bias modification for youth and their parents: a novel treatment for early adolescent social anxiety. J Anxiety Disord. 2014;28(8):851–64. doi:https://doi.org/10.1016/janxdis.2014.09.011.PubMedPubMed CentralView ArticleGoogle Scholar
- Gladstone T, Marko-Holguin M, Henry J, Fogel J, Diehl A, Van Voorhees BW. Understanding adolescent response to a technology-based depression prevention program. J Clin Child Adolesc Psychol. 2014;43(1):102–14. doi:https://doi.org/10.1080/15374416.2013.850697.PubMedView ArticleGoogle Scholar
- Eisen JC, Marko-Holguin M, Fogel J, et al. Pilot study of implementation of an internet-based depression prevention intervention (CATCH-IT) for adolescents in 12 US primary care practices: clinical and management/organizational behavioral perspectives. Prim Care Companion CNS Disord. 2013;15(6). doi:https://doi.org/10.4088/PCC.10m01065
- Iloabachie C, Wells C, Goodwin B, et al. Adolescent and parent experiences with a primary care/Internet-based depression prevention intervention (CATCH-IT). Gen Hosp Psychiatry. 2011;33(6):543–55. doi:https://doi.org/10.1016/j.genhosppsych.2011.08.004.PubMedView ArticleGoogle Scholar
- Fothergill KE, Gadomski A, Solomon BS, Olson AL, Gaffney CA, Wissow LS. Assessing the impact of a web-based comprehensive somatic and mental health screening tool in pediatric primary care. Acad Pediatr. 2013;13(4):340–7. doi:https://doi.org/10.1016/j.acap.2013.04.005.PubMedPubMed CentralView ArticleGoogle Scholar
- Han C, Voils C, Williams J. Uptake of web-based clinical resources from the MacArthur Initiative on depression and primary care. Community Ment Health J. 2013;49(2):166–71. doi:https://doi.org/10.1007/s10597-011-9461-2.PubMedView ArticleGoogle Scholar
- Diamond G, Levy S, Bevans KB, et al. Development, validation, and utility of internet-based, behavioral health screen for adolescents. Pediatrics. 2010;126(1):e163–70. doi:https://doi.org/10.1542/peds.2009-3272.PubMedView ArticleGoogle Scholar
- Fein JA, Pailler ME, Barg FK, et al. Feasibility and effects of a web-based adolescent psychiatric assessment administered by clinical staff in the pediatric emergency department. Arch Pediatr Adolesc Med. 2010;164(12):1112–7. doi:https://doi.org/10.1001/archpediatrics.2010.213.PubMedView ArticleGoogle Scholar
- Salloum A, Crawford EA, Lewin AB, Storch EA. Consumers’ and providers’ perceptions of utilizing a computer-assisted cognitive behavioral therapy for childhood anxiety. Behav Cogn Psychother. 2015;43(01):31–41. doi:https://doi.org/10.1017/S1352465813000647.PubMedView ArticleGoogle Scholar
- Murphy JM, Masek B, Babcock R, et al. Measuring outcomes in outpatient child psychiatry: the contribution of electronic technologies and parent report. Clin Child Psychol Psychiatry. 2011;16(1):146–60. doi:https://doi.org/10.1177/1359104509352895.PubMedView ArticleGoogle Scholar
- Horwitz SM, Hoagwood KE, Garner A, et al. No technological innovation is a panacea: a case series in quality improvement for primary care mental health services. Clin Pediatr (Phila). 2008;47(7):685–92. doi:https://doi.org/10.1177/0009922808315215.View ArticleGoogle Scholar
- Pretorius N, Rowlands L, Ringwood S, Schmidt U. Young people’s perceptions of and reasons for accessing a web-based cognitive behavioural intervention for bulimia nervosa. Eur Eat Disord Rev. 2010;18(3):197–206. doi:https://doi.org/10.1002/erv.985.PubMedView ArticleGoogle Scholar
- Hanley T. Developing youth-friendly online counselling services in the United Kingdom: a small scale investigation into the views of practitioners. J Psycol Psychother Res. 2006;6(3):182–5. doi:https://doi.org/10.1080/14733140600857535.Google Scholar
- Stallard P, Richardson T, Velleman S. Clinicians’ attitudes towards the use of computerized cognitive behaviour therapy (cCBT) with children and adolescents. Behav Cogn Psychother. 2010;38(5):545–60. doi:https://doi.org/10.1017/S1352465810000421.PubMedView ArticleGoogle Scholar
- Hetrick SE, Dellosa MK, Simmons MB, Phillips L. Development and pilot testing of an online monitoring tool of depression symptoms and side effects for young people being treated for depression. Early Interv Psychiatry. 2015;9(1):66–9. doi:https://doi.org/10.1111/eip.12127.PubMedView ArticleGoogle Scholar
- Ahmad F, Norman C, O’Campo P. What is needed to implement a computer-assisted health risk assessment tool? An exploratory concept mapping study. BMC Med Inform Decis Mak. 2012;12:149-. doi:https://doi.org/10.1186/1472-6947-12-149
- Merry SN, Stasiak K, Shepherd M, Frampton C, Fleming T, Lucassen MF. The effectiveness of SPARX, a computerised self help intervention for adolescents seeking help for depression: randomised controlled non-inferiority trial. BMJ. 2012;344:e2598. doi:https://doi.org/10.1136/bmj.e2598.PubMedPubMed CentralView ArticleGoogle Scholar
- Mair FS, May C, O’Donnell C, Finch T, Sullivan F, Murray E. Factors that promote or inhibit the implementation of e-health systems: an explanatory systematic review. Bull World Health Organ. 2012;90(5):357–64. doi:https://doi.org/10.2471/blt.11.099424.PubMedPubMed CentralView ArticleGoogle Scholar
- Owens C, Ley A, Aitken P. Do different stakeholder groups share mental health research priorities? A four‐arm Delphi study. Health Expect. 2008;11(4):418–31. doi:https://doi.org/10.1111/j.1369-7625.2008.00492.x.PubMedPubMed CentralView ArticleGoogle Scholar
- Greenhalgh T, Robert G, Bate P, Macfarlane F, Kyriakidou O. Diffusion of innovations in health service organisations: a systematic literature review. Milbank Q. 2004;82(4):581-629. doi: 10.1111/j.0887-378X.2004.00325.xGoogle Scholar
- Jansen MW, Van Oers HA, Kok G, De Vries NK. Public health: disconnections between policy, practice and research. Health Res Policy Syst. 2010;8(1):37. doi:https://doi.org/10.1186/1478-4505-8-37.PubMedPubMed CentralView ArticleGoogle Scholar
- Choi BC, Pang T, Lin V, et al. Can scientists and policy makers work together? J Epidemiol Community Health. 2005;59(8):632–7. doi:https://doi.org/10.1136/jech.2004.031765.PubMedPubMed CentralView ArticleGoogle Scholar
- Peters DH, Tran NT, Adam T. Implementation research in health: a practical guide. Alliance for Health Policy and Systems Research, World Health Organization; 2013.Google Scholar
- Lyon AR, Lewis CC, Melvin A, Boyd M, Nicodimos S, Liu FF, Jungbluth N. Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology: description and application to clinical feedback systems. Implement Sci. 2016;11(1):128. doi:https://doi.org/10.1186/s13012-016-0495-2.PubMedPubMed CentralView ArticleGoogle Scholar
- Dixon BE, Zafar A, McGowan JJ. Development of a taxonomy for health information technology. Stud Health Technol Inform. 2007;129(2):616–20.PubMedGoogle Scholar
- Pagliari C, Sloan D, Gregor P, et al. What is eHealth (4): a scoping exercise to map the field. J Med Internet Res. 2005;7(1):e9. doi:https://doi.org/10.2196/jmir.7.1.e9.PubMedPubMed CentralView ArticleGoogle Scholar
- Oh H, Rizo C, Enkin M, Jadad A. What is eHealth (3): a systematic review of published definitions. J Med Internet Res. 2005;7(1):e1. doi:https://doi.org/10.2196/jmir.7.1.e1.PubMedPubMed CentralView ArticleGoogle Scholar
- Johnson K, Hays C, Center H, Daley C. Building capacity and sustainable prevention innovations: a sustainability planning model. Eval Program Plann. 2004;27(2):135–49.View ArticleGoogle Scholar
- Turner KM, Sanders MR. Dissemination of evidence-based parenting and family support strategies: learning from the Triple P—Positive Parenting Program system approach. Aggress Violent Behav. 2006;11(2):176–93.View ArticleGoogle Scholar
- Rabin BA, Brownson RC, Haire-Joshu D, Kreuter MW, Weaver NL. A glossary for dissemination and implementation research in health. J Public Health Manag Pract. 2008;14(2):117–23. doi:https://doi.org/10.1097/01.PHH.000031888.06252.bb.PubMedView ArticleGoogle Scholar
- Glasgow RE, Vogt TM, Boles SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. 1999;89(9):1322–7.PubMedPubMed CentralView ArticleGoogle Scholar
- Lewis CC, Fischer S, Weiner BJ, Stanick C, Kim M, Martinez RG. Outcomes for implementation science: an enhanced systematic review of instruments using evidence-based rating criteria. Implement Sci. 2015;10:155. doi:https://doi.org/10.1186/s13012-015-0342-x.PubMedPubMed CentralView ArticleGoogle Scholar
- Chaudoir SR, Dugan AG, Barr CH. Measuring factors affecting implementation of health innovations: a systematic review of structural, organizational, provider, patient, and innovation level measures. Implement Sci. 2013;8(1):22. doi:https://doi.org/10.1186/1748-5908-8-22.PubMedPubMed CentralView ArticleGoogle Scholar