Pediatric eMental healthcare technologies: a systematic review of implementation foci in research studies, and government and organizational documents

Background Researchers, healthcare planners, and policymakers convey a sense of urgency in using eMental healthcare technologies to improve pediatric mental healthcare availability and access. Yet, different stakeholders may focus on different aspects of implementation. We conducted a systematic review to identify implementation foci in research studies and government/organizational documents for eMental healthcare technologies for pediatric mental healthcare. Methods A search of eleven electronic databases and grey literature was conducted. We included research studies and documents from organization and government websites if the focus included eMental healthcare technology for children/adolescents (0–18 years), and implementation was studied and reported (research studies) or goals/recommendations regarding implementation were made (documents). We assessed study quality using the Mixed Methods Appraisal Tool and document quality using the Appraisal of Guidelines for Research & Evaluation II. Implementation information was grouped according to Proctor and colleagues’ implementation outcomes—acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability—and grouped separately for studies and documents. Results Twenty research studies and nine government/organizational documents met eligibility criteria. These articles represented implementation of eMental healthcare technologies in the USA (14 studies), United Kingdom (2 documents, 3 studies), Canada (2 documents, 1 study), Australia (4 documents, 1 study), New Zealand (1 study), and the Netherlands (1 document). The quality of research studies was excellent (n = 11), good (n = 6), and poor (n = 1). These eMental health studies focused on the acceptability (70%, n = 14) and appropriateness (50%, n = 10) of eMental healthcare technologies to users and mental healthcare professionals. The quality of government and organizational documents was high (n = 2), medium (n = 6), and low (n = 1). These documents focused on cost (100%, n = 9), penetration (89%, n = 8), feasibility (78%, n = 7), and sustainability (67%, n = 6) of implementing eMental healthcare technology. Conclusion To date, research studies have largely focused on acceptability and appropriateness, while government/organizational documents state goals and recommendations regarding costs, feasibility, and sustainability of eMental healthcare technologies. These differences suggest that the research evidence available for pediatric eMental healthcare technologies does not reflect the focus of governments and organizations. Partnerships between researchers, healthcare planners, and policymakers may help to align implementation research with policy development, decision-making, and funding foci. Electronic supplementary material The online version of this article (doi:10.1186/s13012-017-0608-6) contains supplementary material, which is available to authorized users.


Introduction
The global prevalence of mental disorders in children and adolescents is reported to be as high as 30% [1][2][3][4]. Under-diagnosis and under-treatment of childhood mental disorders are well-documented concerns [2,[5][6][7]. The current distribution, demand, structure, and costs that underpin pediatric mental healthcare services make them relatively unavailable to many of those who need them [8]. Electronic mental healthcare (eMental healthcare) technologies, which broadly include Internet-, mobile-, and, computer-based programs and resources as well as mobile phone applications, are considered promising approaches to enable more efficient use of mental healthcare resources, lower access barriers to traditional face-to-face mental healthcare, and provide flexibility in terms of standardization and personalization, interactivity, and consumer engagement [9][10][11][12][13][14].
Researchers (e.g., those developing and/or evaluating eMental healthcare technologies), healthcare planners (e.g., administrators in agencies either using or desiring to use eMental healthcare technologies), and policymakers (e.g., individuals with authority to set eMental healthcare policy for a healthcare organization or system) all convey interest and a sense of urgency in using eMental healthcare technologies to improve pediatric mental healthcare availability and access. To date, these three stakeholder groups have focused on different aspects of implementation. Researchers have studied user satisfaction [13,15,16] to determine that eMental healthcare technologies are acceptable to children and adolescents, and their parents and mental healthcare professionals. Healthcare planners and policymakers have discussed issues such as the cost and feasibility of eMental healthcare technologies to deliver pediatric mental healthcare [17]. Current priorities that are relevant to all three stakeholders groups are generating evidence to demonstrate how eMental healthcare technologies can be optimally incorporated within an existing healthcare system and how technology implementation can be supported within an organization or system (e.g., governance, policy, funding) [18][19][20]. These priorities may be optimally achieved through collaborations between researchers, healthcare planners, and policymakers that aim to generate evidence for integrating and supporting eMental healthcare technologies in healthcare systems. To provide recommendations for such collaborations, we conducted a systematic review to identify what aspects of implementation have been studied and reported on for pediatric eMental healthcare technologies and what implementation goals/recommendations are present in government and organizational documents relating to eMental healthcare technologies for pediatric mental healthcare. Our aim was to identify and compare current areas of focus among research studies and government/ organizational documents and to use these areas of focus to propose recommendations for implementation research, policies, and funding.

Design
We systematically reviewed the literature to identify research studies and government and organizational documents with information regarding the implementation of eMental healthcare technologies for children and adolescents. We used a protocol that was developed a priori to define the objective, outline the search strategy, establish selection (inclusion/exclusion) criteria, determine implementation findings, guide the data collection process, and define the analysis. We followed the PRISMA statement checklist for reporting [21].

Search strategy
A research librarian developed and implemented the systematic search strategies using language (English) restrictions. The search was conducted in 11 electronic bibliographic databases: Medline, CINAHL, Embase, EBM Reviews, ProQuest Theses and Dissertations, Ovid MEDLINE In-Process & Other Non-Indexed Citations, OVID HealthStar, Cochrane Database of Systematic Reviews, Health Technology Assessment Database, ACP Journal Club, and SocIndex. The final Medline strategy is provided (see Additional file 1). Search terms focused on population and technology parameters used to screen for study eligibility. We also included terms related to "attitudes," "preferences," and "diffusion of innovation." Thus, the search strategy was broad in order to identify potentially eligible studies that may not have been indexed using specific implementation science terms. The search was executed in each database from inception to September 30, 2015. Although the search was executed from inception to September 30, 2015, we restricted inclusion to studies and documents published after 2005.
To identify unpublished research and research-inprogress, we searched Google, the U.S. National Institutes of Health Clinical Trials database, the Australian New Zealand Clinical Trials Registry, the International Clinical Trials Registry Platform, and the UK Clinical Trials Gateway. To identify government and organizational documents, we conducted a two-pronged search: (1) targeted Google searches for relevant government, health, and technology organizations having clearly stated goals and/or funding relating to eHealth and behavioural technologies and (2) recommendations from members of our team. Overall, we created a list of 38 government and organizational websites, which was reviewed by the research team (see Additional file 2 for the full list). We also sought documents from key contacts responsible for leadership, policy, research, and information technology considered to be influential in the use of eHealth technologies. These contacts represented Canada's Mental Health Commission e-mental health steering committee and the Ontario Centre of Excellence for Child and Youth Mental Health, the Australian Government's Mental Health Commission and Department of Health and Ageing e-mental health expert advisory committee, the Netherland's Dutch Association of Mental Health and Addiction Care, New Zealand's National Health IT Board, and United Kingdom's National Collaborating Centre for Mental Health and Mental Health Network. Reference lists of included studies and documents were also searched.

Criteria for including studies and documents for this review
We restricted the study and government/organizational document inclusion to countries from the largest English speaking eHealth markets [22]. These countries were Australia, Canada, the Netherlands, New Zealand, the United Kingdom, and the USA. Studies of any design were eligible for inclusion.

Population of interest
Studies and government/organizational documents that included children and/or adolescents (0-18 years) as participants (studies only) or a population of interest (documents) were considered for inclusion. Government and organizational documents that focused on eMental healthcare technologies for all ages (including children and/or adolescents) were also eligible for inclusion.

eMental healthcare technology
Studies and government/organizational documents were eligible for inclusion if they evaluated/focused on eMental healthcare technology that met our definition: Internet-, computer-, or mobile-based programs and applications ('apps'). Studies and documents of eMental healthcare technologies focused exclusively on phone calls or teleconferencing were excluded from the review as these technologies did not meet our definition of eMental healthcare. Studies and documents that focused on eMental healthcare technology use with parents of children with a mental health need or pediatric healthcare professionals were eligible for inclusion.
Implementation findings, goals, and/or recommendations All outcomes, goals, and recommendations relating to implementation were considered. We used Proctor and colleagues' eight outcome categories for implementation research [23] as a framework to identify outcomes, objectives, goals, and recommendations of interest in the studies and documents. To be included, a study needed to evaluate and report on at least one of the eight categories, and a document needed to contain at least one goal and/or recommendation that related to a category. The eight categories are as follows: acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability [23]. These outcomes have shaped traditional mental health services integration in routine care [24][25][26][27][28] and provide a common taxonomy for examining eMental healthcare technology implementation. For the purpose of this review, the categories were defined as follows: acceptability, a measure of satisfaction with the technology (including attitudes, functionality, preferences, and user experience); adoption, the intention, initial decision, or action to try or employ technology (e.g., uptake and utilization); appropriateness, the perceived fit, relevance, usefulness/helpfulness, or compatibility of the technology for a given practice setting, professional, or user and/or perceived fit of a technology to address a particular issue or problem; cost, the financial impact of an implementation effort (including measures of cost-effectiveness or cost-benefit); feasibility, the extent to which a technology could be successfully used or carried out within a setting (including utility, compatibility, and barriers); fidelity, the degree to which a technology was implemented as it was intended such as adherence; penetration, the integration of a practice within a service setting and its subsystems (e.g., "spread" or "reach"); and sustainability, the extent to which a newly implemented treatment is maintained or integrated within a service setting's ongoing, stable operations [23].

Screening for eligibility
Studies and documents were organized and screened using EndNote X7.2.1. Studies and documents were first screened at the title and abstract level (stage 1 screening) to determine whether they met the inclusion criteria. At stage 1, two reviewers (NDG, AS) independently screened the title and abstract for the first 100 studies/ documents in the library and subsequently calculated inter-rater agreement with the kappa statistic [29]. The agreement was not sufficiently high (Cohen's kappa, κ = 0.70), and we determined that "implementation" was an unclear term to aid in determining whether studies and documents provided implementation information on implementation outcomes, objectives, goals, and/or recommendations. We introduced Procter's implementation outcomes framework [23] to the review protocol at this time, and two reviewers independently screened another 100 studies/documents in the library. This screening resulted in "almost perfect agreement" (Cohen's kappa, κ = 0.84) [30], which indicated consensus on the definition of implementation. The remaining studies/documents in the library were then divided in two with each reviewer taking a respective half to screen using the title and abstract. Any studies/documents where it could not be determined whether they met inclusion criteria using the article's title and abstract progressed to a review of the full-text (stage 2 screening) to determine eligibility. Any discrepancies were discussed between the reviewers and taken to a third party (ASN) if no agreement could be reached.

Data extraction
Data were extracted into a standardized form (Microsoft Excel; Microsoft, Redmond, Washington, USA). Extracted data were: (1) key article characteristics (e.g., author, date of publication, country); (2) study or document objectives; (3) technology type(s) (Internet-, mobile-, or computer-based) and services/treatments delivered; (4) details about research study design; (5) target population/study participants; and (6) implementation outcomes and findings from studies and implementation goals and/or recommendations from government/ organizational documents according to Proctor et al. (acceptability, adoption, appropriateness, cost, feasibility, fidelity, penetration, and sustainability) [23]. Included studies and documents were divided between two reviewers (NDG, AS) who extracted the data from their respective half and then checked the other's extraction for accuracy and completeness. Discrepancies were resolved by discussion and/or by contacting corresponding authors of included studies/documents for clarification.

Quality assessment
The quality of the research studies was assessed using the Mixed Methods Appraisal Tool (MMAT) [31]. The MMAT is applicable to quantitative, qualitative, and mixed methods studies. The scoring scale ranges from 0 (low quality) to 100 (high quality) and has been pilot tested for reliability in systematic reviews [32]. Ratings are specific to particular methodologies and are based on control of confounding factors, completeness of outcome data, minimization of selection bias, representativeness of sample, appropriateness of measures, response and withdrawal rates, appropriateness of study design to answer the research questions, and consideration of limitations. Two reviewers (NDG, AS) independently completed the MMAT for each included study and inter-rater agreement was considered "almost perfect" (Cohen's kappa, κ = 0.81) [30]. Discrepancies were resolved by a third party (ASN).
The quality of government and organizational documents was assessed using the Appraisal of Guidelines for Research & Evaluation II (AGREE II) [33]. AGREE II assesses six domains: scope and purpose, stakeholder involvement, rigor of development, clarity of presentation, applicability, and editorial independence. While originally intended for clinical practice guidelines, most domains are applicable to government and organizational documents. Two domains, "clarity of presentation" and "applicability," have criteria specific to guideline recommendations; however, because we were interested in document recommendations, these domains were applicable to our use. The AGREE II provided an overall quality score from 0% (low quality) to 100% (high quality), and a recommendation as to whether the document was recommended: (1) for use, (2) for use with modifications, or (3) not for use. Two reviewers (NDG, AS) independently completed the AGREE II for each document. Domain and overall quality scores were calculated by summing up the scores given by two reviewers [34]. Recommendations for document use were decided by consensus between the two reviewers. Discrepancies in recommendations for document use were resolved by a third party (ASN).

Data analysis
Data analysis followed two approaches. First, two reviewers (NDG, AS) conducted a narrative (descriptive) synthesis [35] to produce a summary of the research studies and government/organizational documents included in the review. This summary encompassed the five domains for which we extracted data. Second, implementation data from each research study and government/organizational document were grouped according to the eight implementation outcomes framework [23]. A cell remained empty if there was no relevant data pertaining to an implementation outcome. Data categorization was reviewed and discussed by NDG, ASN, and LW until consensus was achieved, and all data were coded into appropriate outcomes.
As a final step, a template approach to text analysis [36] was undertaken by two reviewers (NDG, AS). This approach involved bringing together the narrative synthesis and grouped implementation data under the implementation outcomes framework. This analytic step allowed the research team to identify the implementation foci of research studies as compared to government/organizational documents so that recommendations for implementation research, policies, and funding could be formulated.

Literature search and selection
As shown in Fig. 1, after the removal of duplicates, the literature search identified 3818 articles for screening; 3737 research articles and 81 government and organizational documents. A total of 3058 articles were excluded after screening the titles and abstracts. The full texts of the remaining 760 articles were reviewed. Of these, 29 articles were included in the review: 9 government/organizational documents and 20 research studies.

Research study and government/organizational document characteristics
As shown in Table 1, of the nine government and organizational documents, five focused primarily on eMental healthcare services for youth [17,[37][38][39][40] while the remaining discussed eMental healthcare for the general population, including youth [41][42][43]. Although some reports contained research elements (i.e., literature reviews, stakeholder interviews, surveys) [38-40, 42, 44], they were categorized as government/organizational documents given their affiliations. Four documents originated from Australia [37,38,43,44], two from the United Kingdom [39,42], two from Canada [17,40], and one from the Netherlands [41]. Following quality assessment, one document was found to be of poor quality To identify the barriers impacting eHealth readiness and adoption and explore how to minimize them 3. To identify eHealth enablers and explore how to apply them to drive adoption and effective usage -Government -Healthcare professionals ✓ (25%) and was not to be recommended for use [41], six documents were found to be of medium quality (range 58-83%) and were recommended for use given modification [17, 37-39, 42, 44], and two documents were found to be of high quality (92%) and were recommended for use without modifications [40,43] (see Additional file 3). Table 2 outlines the 20 research studies included in the review. Fourteen studies were from the USA [45][46][47][48][49][50][51][52][53][54][55][56][57][58], three from the United Kingdom [59][60][61], and one each from Australia [62], Canada [63], and New Zealand [64]. Across the studies, fourteen examined eMental healthcare technologies to be used by children and adolescents [45, 46, 48-51, 54-56, 59-62, 64], three examined technologies to be used by healthcare professionals when interacting with pediatric patients [47,53,63], and three examined technologies to be used by parents on behalf of their child [52,57,58]. One study was assessed as being of extremely poor quality, receiving a MMAT score of 0 [50], two studies were of poor quality and received a score of 50 [48,49], six were of good quality and received a score of 75 [47,51,53,56,58,61], and the remaining 11 were of excellent quality with a score of 100 [45,46,52,54,55,57,59,60,[62][63][64]]. An additional file shows details of the quality assessment (see Additional file 4). Table 3 presents the implementation outcomes and findings from research studies and implementation goals and recommendations from government and organizational documents, organized according to Proctor and colleagues' implementation outcomes.
Five government/organizational documents (56%) included goals or recommendations relating to the acceptability of eMental healthcare technology. Documents were mainly of medium quality (range 58-83%) [17,37,39,44]; although, one document was of high quality [43]. These documents described the need for better incorporation of consumer preference during design planning to ensure such services are user-centered and individualized [17,39,43,44] and emphasized the need to increase public awareness of the acceptability of eMental healthcare technology [17,37,43].

Adoption
Five studies (25%), of good or excellent quality, measured adoption of eMental health technology for pediatric mental healthcare [46,47,53,55,57]: pre-post measurements of treatment attendance [46], screening rates [47,55], number of intake appointments [57], and uptake into healthcare practice [53]. Studies showed that children and adolescents receiving the eMental healthcare technology demonstrated significantly higher rates of attendance [46], and electronic technologies were associated with improved clinician completion rates [57]; however, studies concerned with healthcare professional adoption showed moderate screening rates and uptake into practice [47,53,55]. Government and organizational documents did not address actual eMental healthcare technology adoption; although, documents did discuss the intent to use technology to provide mental healthcare.
Five government and organizational documents (56%) addressed appropriateness. One document was of high quality [40], while the remaining were of medium quality (range 58-83%) [17,38,42,44]. Documents considered appropriateness from the perspectives that (1) additional research and development should be conducted to assure healthcare quality and safety standards are not    [17,40,42,44], and (3) technologies need to be accessible to rural, regional, and indigenous communities [38,40] to ensure appropriateness to these populations.

Cost
Of the 20 research studies, one qualitative study (5%) of good quality considered cost issues from system and organization perspectives and described the need to address cost coverage in terms of who covers treatment costs and how to integrate third party payers [56]. All nine government and organizational documents (100%) cited cost as a major component of future eMental healthcare implementation efforts [17,[37][38][39][40][41][42][43][44]. One of these documents one was of low quality [41], four were of medium quality (range 58-83%) [37,38,42,44], and one was of excellent quality [40]. Across the government and organizational documents, start-up costs for developing or implementing technology into practice [17,37,38,40], reimbursement to consumers from health insurance companies [17,39,44], the costeffectiveness of blended care models [41], and billing requirements for healthcare professionals providing eMental healthcare services were discussed [17,39,44]. Overall, the government/organizational literature recommended establishing and evaluating a sustainable funding model to address high development and continuing maintenance costs [17,37,39,[42][43][44] and allocating more government funds to further support research and development [42].

Feasibility
Two studies (10%) quantitatively measured eMental healthcare technology feasibility [48,63]. These studies were of poor quality [48] and excellent quality [63]. The definition of feasibility did not vary between the studies. While one of the studies found eMental healthcare technology to be feasible [48], the other found that healthcare professionals perceived professional development and workload as feasibility challenges (for example, rate changes to workload may be required when adopting an eMental healthcare technology within a professional's clinical workflow) [63].
Seven government/organizational documents (78%) of excellent [40,43] and medium quality (range 58-83%) [17,37,38,42,44] described the need to address feasibility if implementation of eMental healthcare technologies are to be successful [17,37,38,40,[42][43][44]. Document recommendations included ensuring training and education programs for healthcare professionals and organizations as a means to ensure technology feasibility. That is, although a technology may be appropriate for a given setting, it may not be feasible to implement it if resources and training are not available [17,37,38,40,[42][43][44].

Fidelity
Three studies (15%) examined fidelity by quantifying adherence to the technology [48,57] and reasons for noncompletion [64]. Two of these studies were of excellent quality [57,64] and one was of poor quality [48]. Studies reporting on fidelity found either perfect [48] or good [57,64] adherence to the technology protocol, finding reasons for non-completion such as technical glitches and lack of time and/or interest [64].
Recommendations from seven government and organizational documents (67%) concentrated on the importance of fidelity in implementation evaluations [17, 38-41, 43, 44]. Of these documents one was of poor quality [41], four were of medium quality (range 58-83%) [17,38,39,44], and two were of excellent quality [40,43]. Government and organizational documents also described concerns about dropout and lack of follow-up with the use of eMental healthcare technologies [17,38], and the need for evaluations to determine if such technologies are being delivered as intended and are as innovative as described [17, 38-41, 43, 44].

Discussion
There is an increasing interest and sense of urgency from the perspectives of researchers, healthcare planners, and policymakers to use eMental healthcare technologies in pediatric mental healthcare to improve healthcare availability and access. To date, however, these stakeholders have tended to focus on different aspects of implementation. If effective eMental healthcare services are to become a core component of routine service delivery, these different areas of focus need to be identified so that alignment of research, policies, and funding can occur. We undertook a systematic review to identify research studies and government and organizational documents that describe eMental healthcare technology implementation in healthcare systems for children and adolescents and explore what areas of implementation researchers, healthcare planners, and policymakers have historically focused on. This approach allowed us to identify areas of focus in the current eMental healthcare landscape in terms of implementation outcomes among different stakeholder groups and to use these areas of focus to propose recommendations for implementation research, policies, and funding. The takeaway points of this review are as follows: Implementation foci differ between researchers, healthcare planners, and policymakers We found multiple differences in implementation outcome foci between government and organizational literature and research studies. Consistent with other reviews [18,65], the research studies in this review predominantly focused on patient and clinician outcomes (e.g., use, satisfaction, acceptability). Areas of focus in government and organizational reports were cost, penetration, and feasibility. Differences in underlying positions between stakeholder groups may lead to opposing criteria for successful eMental healthcare implementation and/or may jeopardize stakeholder engagement [66]. The task of disseminating evidence on the success of these initiatives does not occur in separate asocial and apolitical bubbles [67]. They are often produced by, and in turn feed back into, the political process of deciding priorities and allocating resources to pursue them.
Distinct knowledge holders often have differing mentalities; governing bodies are often highly incentivized by the lowest cost for the most efficient use of resources, while research tends to be incentivized by innovation and/or validating their work [68,69]. Thus, the differences in underlying priorities may reflect different values and goals of governing bodies and researchers. For example, due to underlying incentives, government and organization priorities may naturally gravitate towards long-term benefits of eMental healthcare (i.e., penetration and sustainability) as this aligns with the mandate of cost versus benefits. Given that eMental healthcare is a relatively new field, research may be appropriately focused on establishing an eMental healthcare innovation as acceptable and appropriate to users before studying other implementation constructs. As the field matures, there is an opportunity to study implementation outcomes, such as cost, penetration, and feasibility, that would be of interest and value to governments and organizations [23,70]. Moving forward, alignment of funding interests and partnerships between stakeholder groups may allow for the development of funding models to study outcomes that require longer-term investigation and substantial effort such as fidelity, penetration, and sustainability. Future research endeavors may also benefit from using a structured, theory-driven methodology to compile, evaluate, and integrate eMental healthcare information such as the Health Information Technologies-Academic and Commercial Evaluation (HIT-ACE) methodology [71].
A taxonomy for implementation outcomes is needed for the eMental healthcare technology field Consistent with the broader literature, this review reflects a nonstandard usage of terminology and lack of consensus on a taxonomy relating to implementation metrics [72][73][74]. For example, there was overlap and inconsistency in the operationalization and discussion of appropriateness and acceptability; adoption was often used interchangeably with uptake; fidelity was often referred to as adherence; feasibility described as acceptability; and sustainability was reflected in the literature by varying, but congruent definitions, such as maintenance, incorporation, and integration [23,[75][76][77][78].
Findings from a recent review to identify instruments to measure implementation in mental healthcare included an uneven distribution of instruments across implementation outcomes [79]. Although constructs such as acceptability and adoption are well recognized among instruments, other constructs are either underdeveloped or do not lend themselves easily to instrumentation, yielding few instruments to measure a range of implementation outcomes. More groundwork is needed to enable consistency in the definition and measurement of implementation constructs rather than considering them at an abstract level. This approach will help to align areas of focus and discussions between researchers, healthcare planners, and policymakers.

Limitations
This systematic review has several limitations. First, our research study search results may be limited given "behavioral health" or its derivatives were not used in our search string. While not all eMental healthcare studies will be indexed solely with these terms, there may be some that are. These studies would not have been included in our review. Additionally, while our search to identify unpublished research and research-in-progress was extensive, we did not search other databases, such as the NIH Reporter, which may have yielded additional eMental health technology studies. While some of the studies in the NIH Reporter may have been additionally registered in the registries we searched, some may not have been.
Another limitation is our decision to restrict inclusion of government and organizational reports from countries with known expertise and knowledge in the area of eMental healthcare. Thus, this review's findings may not generalize to low and middle-income countries where mental healthcare systems may be organized differently and technologies used in different ways. That none of our included government/organizational documents originated in the USA, while 15 of our research studies did, could display an unrepresented sample in the review, and may explain the lack of alignment between the groups of literature we studied. Further, not all of the government and organizational websites we reviewed may have made implementation information available on the web. It was not feasible for the research team to contact all organizations by email or phone to request information that may have been in paper-based form only.
That our review was limited to English language documents also limits the literature we included in the review, and thus the generalizability of the results. We also believe that there are eMental healthcare technologies being deployed in healthcare systems that have not been scientifically investigated; thus, important implementation data for these technologies would not be available for our review.
The decision to use Proctor and colleagues' implementation outcomes as a guiding framework was not applied a priori. Therefore, specific implementation outcome terms were not included in our search string and could have resulted in potentially relevant studies not being identified in the search results if they were indexed solely using implementation outcome terms. However, the search strategy was developed using terms specific to our population and interventions of interest. Studies of eMental healthcare technologies for children and/or adolescents are more likely to be solely indexed according to these terms and may or may not include indexing with implementation terms. Our descriptive approach to synthesizing implementation may also be criticized; however, in doing so, we believe we were able to identify an important perspective of how eMental healthcare is currently being discussed at the governmental/ organizational level alongside research developments.
Finally, the inconsistent use of terminology [72][73][74] across the literature in this review required us to make judgment calls regarding how to categorize implementation outcomes. There also remains conceptual ambiguity and overlap among implementation outcomes (e.g., acceptability and appropriateness), which could have resulted in some factors being arguably different constructs [23,79]. However, the difficulty in grouping these metrics within the implementation outcomes further confirms the lack of agreement regarding constructs hypothesized to affect implementation success and the identifiable measures of these constructs [80].

Conclusion
This systematic review identified differing implementation foci between research studies and government/ organizational documents, and a lack of consistent implementation taxonomies and metrics. These differences mean that the research evidence available to support eMental healthcare technologies for pediatric mental healthcare is not well aligned with the focus of governments and organizations. Using the results of this review as a guide, partnerships between researchers, healthcare planners, and policymakers may help to align implementation research, policies, and funding foci.