Skip to main content
  • Research article
  • Open access
  • Published:

Implementing and evaluating a regional strategy to improve testing rates in VA patients at risk for HIV, utilizing the QUERI process as a guiding framework: QUERI Series



We describe how we used the framework of the U.S. Department of Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI) to develop a program to improve rates of diagnostic testing for the Human Immunodeficiency Virus (HIV). This venture was prompted by the observation by the CDC that 25% of HIV-infected patients do not know their diagnosis – a point of substantial importance to the VA, which is the largest provider of HIV care in the United States.


Following the QUERI steps (or process), we evaluated: 1) whether undiagnosed HIV infection is a high-risk, high-volume clinical issue within the VA, 2) whether there are evidence-based recommendations for HIV testing, 3) whether there are gaps in the performance of VA HIV testing, and 4) the barriers and facilitators to improving current practice in the VA.

Based on our findings, we developed and initiated a QUERI step 4/phase 1 pilot project using the precepts of the Chronic Care Model. Our improvement strategy relies upon electronic clinical reminders to provide decision support; audit/feedback as a clinical information system, and appropriate changes in delivery system design. These activities are complemented by academic detailing and social marketing interventions to achieve provider activation.


Our preliminary formative evaluation indicates the need to ensure leadership and team buy-in, address facility-specific barriers, refine the reminder, and address factors that contribute to inter-clinic variances in HIV testing rates. Preliminary unadjusted data from the first seven months of our program show 3–5 fold increases in the proportion of at-risk patients who are offered HIV testing at the VA sites (stations) where the pilot project has been undertaken; no change was seen at control stations.


This project demonstrates the early success of the application of the QUERI process to the development of a program to improve HIV testing rates. Preliminary unadjusted results show that the coordinated use of audit/feedback, provider activation, and organizational change can increase HIV testing rates for at-risk patients. We are refining our program prior to extending our work to a small-scale, multi-site evaluation (QUERI step 4/phase 2). We also plan to evaluate the durability/sustainability of the intervention effect, the costs of HIV testing, and the number of newly identified HIV-infected patients. Ultimately, we will evaluate this program in other geographically dispersed stations (QUERI step 4/phases 3 and 4).

Peer Review reports


Over the past decade, in the developed world, the Human Immunodeficiency Virus (HIV) has been transformed from a fatal disease with rapid progression to clinical AIDS (Acquired Immune Deficiency Syndrome) and death – to a chronic illness that can be successfully managed through outpatient-based care over long periods of active life. However, advances in the effectiveness of treatment have been accompanied by new challenges for healthcare systems as they aim to provide adequate chronic illness care to large numbers of ambulatory HIV patients.

Unfortunately, the Centers for Disease Control and Prevention (CDC) estimates that approximately 25% of the nearly one million HIV-infected persons in the United States are not aware of their status [1]. Thus one of the key barriers to the receipt of care for HIV infection is under-diagnosis of this condition, particularly in the early, asymptomatic phases of illness when medical intervention is highly cost-effective [2, 3]. This article is one in a Series of articles documenting implementation science frameworks and approaches developed by the U.S. Department of Veterans Affairs (VA) Quality Enhancement Research Initiative (QUERI). QUERI is briefly outlined in Table 1 and is described in more detail in previous publications [4, 5]. The QUERI Series' introductory article [6] highlights aspects of QUERI related specifically to implementation science, and describes additional types of articles contained in the Series.

Table 1 The VA Quality Enhancement Research Initiative (QUERI)

The specific QUERI Centers are the operational arm of QUERI. Their establishment is determined by VA's Health Services Research and Development Service (HSR&D), in conjunction with clinical management. The existence of the HIV/Hepatitis QUERI Center, in part, reflects the fact that the VA is the largest single provider of HIV care in the United States, providing care for more than 20,000 veterans with HIV, or roughly 40% of the nation's HIV-infected veteran population [1, 7]. The original mission of the HIV/Hepatitis QUERI Center was to increase access to, and uptake of, evidence-based HIV care, and to improve the care that veterans receive for their disease and associated comorbid diseases. More recently, the scope of this Center has expanded to include veterans with chronic viral hepatitis.

Previous accomplishments of the HIV/Hepatitis QUERI Center include the development of effective decision support tools for HIV providers; analyses of the cardiovascular and cerebrovascular risk from long-term, highly-active antiretroviral therapy; determination of the scientific rational for early identification of HIV infection; development and assessment of strategies to improve medication adherence; and validation of the utility of rapid tests for diagnosis of HIV infection in the VA [813].

The purpose of this paper is to provide an illustrative case study that demonstrates work done by the HIV/Hepatitis QUERI Center, and to apply the QUERI 6-step/4-phase framework to the development and implementation of a program to improve rates of HIV diagnostic testing in VA medical care facilities.

The key issues addressed in this paper focus on how the QUERI framework guided our efforts to: 1) identify specific gaps in care, barriers to diagnosis, and the loci within the VA where interventions to improve HIV diagnostic testing would have the greatest impact; 2) develop an intervention to improve diagnostic testing; and 3) implement a staged-series of studies to evaluate the effectiveness of this program. We also discuss how we used known quality improvement strategies to change the behavior of primary care providers in regard to HIV testing.

In the following sections, we show how Steps 1–3 of the QUERI process were followed to demonstrate that undiagnosed HIV infection is a high-risk, high-volume clinical issue (QUERI step 1), that there are evidence-based recommendations for HIV testing (QUERI step 2), and that there are gaps in the performance of HIV testing within the VA healthcare system (QUERI step 3).

Based on this foundation, and with the support of theoretical and empirical considerations, we have developed and initiated a two-site (station) pilot project (QUERI step 4/phase 1) with the goal of refining our interventions to improve rates of HIV testing prior to launching a small-scale, multi-site evaluation in 5 separate stations (QUERI step 4/phase 2).


Step 1: Priority Condition/Issue: Is undiagnosed HIV infection a high-risk, high-volume clinical issue within the VA?

The observation by the CDC that 25% of HIV-infected patients in the United States do not know their status is of great relevance to the VA [1] because the VA treats more than 20,000 veterans with HIV per year [1]. If, as in the rest of the country, 25% of HIV-infected VA patients do not know their status, approximately 7,000 veterans are at risk of being diagnosed with HIV and treated only when they become symptomatic and severely immunosuppressed. Thus, since HIV patients benefit greatly from early diagnosis and treatment [2, 3], increased HIV testing in the VA can substantially reduce the number of newly diagnosed veterans who present with concurrent complications of profound immunodeficiency [3, 14] and extend survival for the average HIV-infected patient by 1.5 years [2, 3].

The decision by the HIV/Hepatitis QUERI Center to focus on HIV testing as a priority within HIV quality improvement was based on the above observations. This decision was approved by the nationally constituted executive committee of the HIV/Hepatitis QUERI Center. QUERI processes require that this prioritization needs to be re-justified on an annual basis through the development of a strategic plan that is reviewed and approved by VA/HSR&D Central Office appointees, representing both researchers and clinical management.

Step 2: Evidence-based Practices: Are there evidence-based recommendations for HIV testing?

The U.S. Preventive Services Task Force (USPSTF) gives a Grade A recommendation to HIV screening for all adolescents and adults who are at increased risk for HIV infection or who receive health care in a high-prevalence setting, such as where ≥ 1% of the patient population is known to be HIV-infected (Table 2) [15]. The VA has endorsed the USPSTF guidelines for HIV infection and has identified testing for HIV as being a high priority [16].

Table 2 US Preventive Services Task Force Grade A Recommendations for HIV Screening*

The cost-effectiveness of HIV testing in the VA is well established. As shown by a HIV/Hepatitis QUERI-affiliated investigator and others, for a population with a prevalence of HIV infection of ≥ 1%, the cost of one-time screening for HIV infection is $15,078 per quality-adjusted life-year gained [2]. Analyses that consider the relationship of diagnosis and treatment on HIV transmission show that the cost of routine HIV screening is < $50,000 per quality-adjusted life-year gained, unless the prevalence of HIV infection is <0.05% [2]. This level of cost effectiveness, which matches that of many well-accepted procedures such as performing colonoscopy for colorectal screening [17], provides a strong argument in favor of implementing HIV testing programs, especially as VA patients have rates of HIV much higher than the 0.05% lower bound of cost-effectiveness [18].

Step 3: Quality/Performance Gaps: Are there gaps in the performance of HIV testing in the VA?

Previous studies done by HIV/Hepatitis QUERI, the VA Public Health Strategic Healthcare Group (PHSHG, a dedicated national VA program office for guiding HIV care services), and other VA groups have shown that only 30 – 50% of VA patients with known, documented risk factors have undergone HIV testing [11, 16]. Furthermore, at the time of HIV diagnosis half of the veterans have advanced levels of immune suppression. These veterans have, on average, 3.7 years of VA care before their HIV is diagnosed [19].

To confirm and extend these data, we evaluated the rates of HIV testing in veterans seen in the five southern California and Nevada VA facilities (otherwise referred to as Veterans Integrated Service Network 22 or VISN 22). We found that between January 1999 and December 2004 only 30% of the 45,776 at-risk veterans (i.e., VA patients with positive laboratory tests or diagnostic codes for hepatitis, sexually transmitted diseases, and/or substance abuse) had been tested for HIV infection. The rate of testing for at-risk patients ranged from a low of 8% in primary care clinics to a high of 47% in substance abuse clinics. The low testing rates and large number of at-risk veterans in VA primary care clinics (nearly ten times as great as in substance use clinics due to the vast differences in the size of these clinic populations) pointed to the need for an intervention to focus on improvements in HIV testing performance in VA primary care clinics. To better understand the source of gaps in care and to discern facilitators that would improve current practice, we reviewed VA policies regarding HIV testing and surveyed providers' practices and attitudes regarding HIV testing at two VA facilities. Following, we present these findings.

Organizational factors

Public Law 100–322 requires that VA patients provide voluntary informed consent for HIV testing and that providers document pre- and post-test counseling [16]. In addition, many VA providers regard HIV testing and pre- and post-counseling to be the sole provenance of specially trained HIV counselors [20]. Furthermore, standard policy has been to require all patients with positive or negative test results to come back to clinic for face-to-face, post-test counseling. The post-test counseling appointment is problematic, as many VA providers do not have sufficient appointment slots to allow for timely in-person patient notification of test results (i.e., within 1–2 weeks of the test).

Provider willingness and ability to perform HIV testing

Surveys of a convenience sample of 30 VA primary care providers indicated that lack of knowledge of individuals' risk factors for HIV infection, the time requirements to fulfill necessary counseling processes [20, 21], and anxiety about post-test counseling patients who have positive test results [14, 22] were substantive barriers to ordering HIV tests.

Patient acceptance of HIV testing

A systematic review of 62 studies found that acceptance rates of voluntary HIV testing in the United Sates varies from 11% to 91% [23]. Importantly, this review found that higher acceptance rates were associated with confidentiality protections (strongly upheld by VA policy and procedures), as well as the provider's belief that testing would be beneficial.

Step 4: Identify and implement interventions to promote best practices

Here we describe our QUERI step 4 activities in further detail. We are now nearing the end of phase 1 of step 4, wherein we are conducting a two-station (two-site) pilot project.

Collaborating with clinical services to design an appropriate intervention

We explicitly sought broad institutional support for this project. This included the support of the VA PHSHG, and the VISN 22 Director, Quality Improvement Council, Clinical Practices Council, and the Clinical Performance Committee. VISN 22 leadership agreed to make HIV testing a performance monitor, to support installation of the HIV Testing Clinical Reminder, and to participate as full partners in enhancing station accountability. After obtaining national and regional support, we also made presentations to, and received support from the Medical Executive Committees, Chiefs of Staff, Ambulatory Care Leadership, primary care teams, and the HIV Coordinators at our two intervention stations. Receipt of all this support was greatly facilitated by the products generated from QUERI steps 1–3.

Development and initiation of a program to improve rates of HIV testing

Conceptual basis of the implementation interventions

As required by the QUERI process, we paid careful attention to the selection of a quality improvement model upon which to base our intervention program. We elected to base our program upon the Chronic Care Model (CCM). This decision was based on the previous, wide success of interventions based on the CCM precepts to improve clinical preventive care services [24, 25]. Further guidance for this implementation strategy was provided by Rogers, whose germinal work on diffusion promotes the use of opinion leaders or champions to facilitate change or innovation adoption, and highlights the importance of individuals' social networks, organizational leadership and structure [26].

Key components of the CCM include a clear definition of optimal care and enumeration of targeted patients, i.e., offering HIV-testing to at-risk patients; a road map for changing the system; and an effective improvement strategy [27]. Effective CCM implementation strategies also contain the following elements: decision support, clinical information systems, delivery system design, and patient self management [24, 2830]. Therefore, we developed an implementation strategy that uses: clinical reminders to provide decision support, audit/feedback as a clinical information system, and organizational change to achieve an appropriate delivery system design. These activities are complemented by academic-detailing and social marketing interventions to achieve provider activation to ensure that providers have the skills and motivation to improve their performance [24, 2931]. Both the CCM and the Institute for Healthcare Improvements Breakthrough Series have identified the need for provider activities to transform and sustain changes in group norms [24, 29, 30]. Thus, we chose to implement a multi-faceted provider activation program that includes academic detailing and social marketing [25]. Finally, we promoted HIV counseling to increase patient self-management by wide scale publicity of the HIV testing program in clinic waiting rooms and check-in areas.

Table 3 summarizes the relationships between the barriers to HIV testing, the content of the planned implementation program, and the relationship of each intervention to the elements of our CCM-based implementation strategy, which was augmented by academic-detailing and social-marketing interventions (provider activation). In the following section, we discuss the components of these interventions and the methods of delivery in more detail.

Table 3 Relationship between Identified Barriers, Elements of the Chronic Care Model and Implementation Strategy

Components of the implementation program

Decision Support

To leverage institutional resources, we implemented a real-time, electronic clinical reminder that had been developed by the VA PHSHG to identify veterans at higher than average risk for HIV infection – and to encourage providers to offer HIV testing to such individuals (Figure 1). Widely used to implement quality improvement, clinical reminders are well-suited for use in the VA because of the system-wide computerized patient record. The HIV Testing Clinical Reminder is triggered by prior evidence of infection by Hepatitis B or C infection, illicit drug use, sexually transmitted diseases (STDs), homelessness, and/or documented risk factors for Hepatitis C infection. All these data elements can be automatically extracted from the VA electronic medical record. The reminder is resolved (i.e., appropriately addressed) by: ordering an HIV test, recording the result of an HIV test performed elsewhere, indicating that the patient is not competent to consent to testing, or specifying that the patient refuses HIV testing. Once resolved, the reminder is no longer triggered.

Figure 1
figure 1

The HIV Testing Clinical Reminder is activated whenever a patient with HIV risk factors, who has not been previously tested for HIV, has an encounter with a healthcare provider. The format of the Reminder is as pictured.

The use of clinical reminders in individual patients, when combined with audit/feedback and organizational changes, has been shown to improve performance of vaccination, cardiovascular risk reduction, and breast and colorectal cancer screenings [9, 10, 28, 3133]. Electronic clinical reminders are a standard well-developed technology with which VA providers have great familiarity and have been shown to be well suited to improve performance of tasks similar in nature to HIV test ordering and counseling. Furthermore, our previous work has shown that use of clinical reminders contributes to 10–30% increases in the rates of appropriate clinical interventions in VA HIV-infected patients [9, 10]. Thus, this implementation intervention satisfies the FITT (fit between individuals, task and technology) framework for assessing the suitability of using this intervention in our strategy [34].

Clinical information system

We designed an audit-feedback system, wherein healthcare providers are informed of group performance in regard to HIV screening rates in at-risk patients. A meta-analysis of 85 trials demonstrates that the use of audit-feedback is effective in improving practice, especially when baseline adherence is low [35]. We have distributed audit-feedback reports (Figure 2) to senior medical center-level clinical managers, outpatient clinic managers, and primary care team leaders at the Los Angeles and the San Diego VA stations. The contents of the reports have been discussed during academic detailing visits to primary care team meetings, and in the social marketing campaign. Informal provider feedback regarding the content of these reports has been positive.

Figure 2
figure 2

Sample Audit/Feedback letter. The original report identifies the station and sub-stations by name rather than as "Station A" and sub-stations 1–11. The data provided in this letter indicate the number of persons with identified HIV risk factors who were tested for HIV or who were indicated to have refused HIV testing. No patient had previously been tested for HIV. These results, which represent incident HIV testing and refusal rates, differ from the data given in the results section that indicates the cumulative proportion of patients with identified HIV risk factors who had ever been tested. Achievement of 100% on the Y axis would indicate that all at-risk patients were offered HIV testing as denoted by records indicating that HIV testing was performed or refused by the patient.

Provider activation

We implemented a provider activation program that includes academic detailing and social marketing [25, 35, 36]. This approach recognizes that the engagement of providers and the use of multiple modalities are necessary to achieve and sustain the transformation of group norms and maximize quality improvement [24, 25, 29, 30, 35]. The goal of these activities was to engage providers and influence their attitudes, skills and habits regarding offering HIV testing to at-risk patients. We also used these interactions to reinforce more formal educational efforts. Particular attention was paid to: increasing providers' sense of responsibility for ordering HIV tests, emphasizing the use of streamlined pre-test counseling, reinforcing the use of telephonic notification of negative test results, and assuring providers of the availability of assistance for notification of persons with positive test results.

The academic detailing component of the project involves multiple presentations by clinical champions (physician and nursing staff), supplemented by project staff to the primary care team meetings and educational sessions. We have specifically targeted primary care clinic leaders as local, organizational opinion leaders [25]. We used social marketing with providers to reinforce the importance of changing their practice regarding HIV testing, and further motivate them to do so. Social marketing entails the development of a shared buy-in to the overall goal of the behavior change and is predicated on social exchange theory, which borrows from social and behavioral science doctrines in emphasizing the client/patient/provider as the starting point [37]. The social marketing element includes regular informal discussions of the basis for and benefits of increased rates of HIV testing by project staff during frequent ad hoc visits to the primary care clinic and presentations to sub-station and clinic leadership.

As per the precepts of social marketing theory [37], we have undertaken: audience segmentation for focused detailing to nurses, mid-level providers and physicians; channel analysis to optimize the setting and materials for these audience segments; goal orientation to keep stakeholders focused on why they are involved (i.e., emphasizing the differing tasks by nurses [pre-test counseling ] and physicians [order entry]); and process tracking to monitor progress and provide feedback for refinement and revision of strategies (i.e., through audit-feedback and through formative evaluation), which can be considered an intervention [38].

Based on survey responses by physicians, mid-level providers, nurses, and case managers, we also have developed provider education materials to supplement the face-to-face training conducted by project staff that focus on preparing providers to use the reminders effectively, making providers aware of HIV risk factors not captured by the reminder (i.e., multiple unprotected sexual contacts), and increasing provider comfort and abilities to provide pre- and post-test HIV counseling.

In our step 4/phase 1 project, all social marketing has been performed by senior project staff. Using the insights gained in the two-station pilot project, as we progress down the pipeline to a step 4/phase 2 small-scale, multi-station evaluation we will rely on a train-the-trainer model to activate local champions. Project staff will support the local champions with regular visits (monthly for the first 3 months of local implementation, and then quarterly), weekly telephone conference calls and e-mail support.

Delivery System Changes

We have strived to ensure that all providers are trained to use a recent revision of the VA HIV Consent Form [39]. This new document includes all the necessary elements of pre-test HIV counseling, and thus facilitates the consent process for healthcare providers who lack specific training regarding the performance of HIV counseling. We also have encouraged nurse-based rather than physician-based pre-test counseling [40]. Nurses perform many educational, health promotion, and disease prevention tasks as well as physicians [41]. Organizational changes that shift responsibilities from physicians to other personnel, most often nurses, are effective in improving preventive care. Moreover, we have encouraged providers to use a streamlined HIV counseling process that covers all the required elements of HIV pre-test counseling and reduces the time of pre-test counseling to 2–3 instead of 10–15 minutes, with some counseling lasting as long as 35 minutes [42]. Similarly, we have reduced the logistical challenges of post-test HIV counseling [16]. Given the gravity of the information, post-test counseling for persons with new positive HIV test results strongly warrants face-to-face counseling. In contrast, we have alerted providers that for patients with a negative result, post-test counseling can be very brief and can be done via the telephone [43]. To ensure compliance with post-test counseling requirements, we have distributed sample scripts for transmitting the results of the test.

In addition to being theory-informed, these interventions are informed by empirical evidence provided by studies in urgent care clinics, emergency departments, and STD (sexually transmitted disease) clinics that show increased testing rates and patient receipt of test results after implementation of structural changes, such as improved staff training in pre- and post-testing screening, introduction of streamlined counseling, and substitution of telephonic post-test counseling in place of a required return visit for face-to-face notification [43].


The intervention program is being put in place for one year in the primary care clinics of stations A and B in VISN 22. The three remaining stations in VISN 22 (stations C, D and E) served as controls. These facilities each provide care to 37,000 – 80,000 veterans per year. Facilities were assigned to the active or control arms by convenience. All facilities, except for one of the controls, consisted of an inpatient center plus one or more geographically dispersed sub-stations in which primary care and specialty services, including mental health and substance abuse treatment programs, were provided by academic staff physicians, post-graduate medical trainees, and mid-level providers. In addition, these facilities also provided primary care in other sub-stations staffed solely by non-academic physicians and mid-level providers. At the remaining control facility, care was provided only in outpatient sub-stations by providers who generally did not have an academic affiliation.

The decision support aspects of the intervention and policies regarding performance of HIV consenting and counseling have been implemented at all sub-stations and all clinics at the active facilities. However, the provider activation component of the program (i.e., academic-detailing and social marketing) has been fully implemented only in the primary care clinics at the two largest sub-stations at Facility A (out of a total of 11 sub-stations) and at the two largest sub-stations at Facility B (out of a total of 6 sub-stations); these sub-stations account for 46% and 69% of patients seen in primary care clinics at Facilities A and B, respectively. The other, smaller and geographically distant sub-stations differ in that e-mail and telephonic outreach largely replace personal outreach to promote academic detailing; all other tools (i.e., audit/feedback, provision of printed materials such as e-mail communications, pocket cards, posters and flyers, and removal of organizational barriers) are the same at all stations. The audit feedback program is directed at all providers in every primary care clinic.

Evaluation plan

The primary endpoints of this step 4/phase 1 two-station pilot project are the effect of the implementation program on the rates of resolution of the HIV clinical reminder and of HIV testing in patients with identified HIV risk factors. A multi-level, logistic regression analysis of the HIV testing rate will be done to adjust for the covariates at patient, provider, and sub-station levels – and for clustering.

We have obtained information regarding all inpatient and outpatient patient encounters within VISN 22 from a pre-established network database. For patients seen in outpatient clinics before and/or after the intervention, we obtained relevant laboratory tests, diagnosis codes, and health factors to determine if they were at increased risk for HIV.

Data regarding non-VA HIV testing, refusal of HIV testing, and incompetence for HIV testing were extracted from the standardized VA clinical reminder software package. Patients were defined as having been tested for HIV if there was documentation of HIV testing done within the VA healthcare system. Veterans were defined as having been evaluated for HIV if there was electronic documentation of prior HIV testing within the VA or elsewhere, patient's refusal to be tested, or patient's incompetence to consent for testing. Information regarding prior HIV testing within the VA was obtained through the VA electronic laboratory records, whereas information regarding outside testing, test refusal, and incompetence to consent was collected through responses to the HIV testing clinical reminder.

In addition to the data abstracted from the VISN 22 database, we also obtained a list of primary care providers classified into provider types (senior staff physicians, mid-level providers, physician assistants, and post-graduate medical trainees) from the primary care administration staff at Facilities A and B. The data on provider types were used to compare HIV testing and evaluation performance across different types of primary care providers.

As previously noted, we fully implemented the provider activation module only at the largest sub-stations at the intervention stations. This design allows us to assess whether this module, which is the most labor intensive component of our implementation strategy, independently contributes to improvement in the rates of HIV screening and testing. In addition, we are conducting formative evaluations to further refine our program and assess the organizational factors that determine the generalizability [38]. The overall aim is to better understand the influences that have an impact on the success of the implementation program by identifying contextually relevant factors (i.e., facilitators and barriers) and assessing the degree to which behaviors that led to improved testing performance become part of routine practice [38]. Semi-structured interviews with key informants will provide qualitative data regarding the effectiveness of the mode used for providing audit/feedback, the usefulness and usability of the testing reminder, and the efficiency of the consenting/counseling process. Interview questions will employ rapid ethnographic assessment methods to explore the ecological context of HIV testing [44].

To assess the degree to which behaviors leading to improved testing performance are institutionalized (i.e., become embedded in standard operating procedures), leading clinicians and administrators at each sub-station will complete a Level of Institutionalization survey. The instrument measures four sub-systems that support routine use of an innovation: production, where it must be integrated with other routine clinical services; maintenance, where employees must support it; supportive, where it must have a stable source of funding; and managerial, where it must be assigned to a specific service, staff must have written job descriptions, and performance is required to be measured and reported [45].

Finally, we are generating a comprehensive analysis of the workload and implementation costs of HIV screening and testing programs, using Business Case Modeling, a method for constructing data-driven models that forecast costs under varying specified conditions that support managerial or technical decision-making. This is warranted as the models of the cost-effectiveness of HIV screening do not address the upfront costs of implementing screening programs across differing clinic settings [2, 3, 46].


Preliminary unadjusted data show an increase in the cumulative rate of reminder resolution (i.e., rate at which the reminder was appropriate addressed) from 22% to 64% at Station A and from 33% to 70% at Station B during the first seven months of this project. Although the amount of salutary change varied across the geographically dispersed sub-stations of the two study stations, all sub-stations showed substantial increases in the rate of resolution of the HIV Testing Clinical Reminder. In contrast, no change in the reminder resolution rate was seen at the other VISN 22 stations where the intervention was not undertaken. Actual monthly rates of HIV testing (as opposed to other means of resolving the HIV Testing Clinical Reminder) increased 3–5 fold. In contrast, no change in HIV testing rates was seen in the control stations. As compared to at-risk patients seen at the control facilities, at-risk patients at the intervention facilities were more likely to be African American or to have a history of either illicit drug use or homelessness. Otherwise there were no significant differences among the demographics and risk factors for HIV risk factors of veterans who received care at the intervention versus the control facilities.

Of note, we found an unexpected high variation in the ratio of HIV test refusals to test performances with some sub-stations recording 10–20 times as many refusals as actual tests, whereas other sites had fewer refusals than tests. These results are at odds with studies that show higher rates of acceptance of HIV testing [23]. Our preliminary analyses suggest that some providers take more care than others in terms of informing patients of the benefits of knowing their HIV status. Furthermore, the rates of resolving the HIV clinical reminder and the ratio of HIV test refusal to performance appear to differ substantially among provider types (i.e., senior staff physicians vs. mid-level practitioner vs. post-graduate medical trainee), with trainees performing far less well than other providers.

Thus far, our preliminary formative evaluation also indicates that further work needs to be done to address station-specific barriers to HIV testing. Particular areas of concern include the development of procedures to address variances in the adoption of new technologies, such as the use of new, electronic paperless versions of the VA HIV consent form. Other important station-to-station differences involve workflow patterns. For example, station B routinely utilizes more nursing personnel for normal clinic routines. Therefore, their intake nurses were expected to share consenting and counseling responsibilities to promote efficient processing of at-risk patients. The process of addressing reminders at station A, on the other hand, is more physician-driven and requires targeting activation strategies to their different role and skill set. Our step 4/phase 1 two-station project also has allowed us to identify aspects of the HIV testing clinical reminder that lead to provider dissatisfaction. In particular, providers would like the reminder to clearly indicate what specific factor(s) triggered the HIV Testing Clinical Reminder, so that they can better counsel patients as to why HIV testing is relevant to the veteran's circumstances. Providers also have expressed a strong belief that homelessness, per se, is not a substantial risk factor for HIV infection, and that HIV testing may not be relevant to persons with a limited life expectancy. Finally, providers perceive the layout of the reminder to be unduly complicated.


This project has greatly benefited from practical guidance and consistent support of the VA QUERI program. For example, a step 3 study by Owens et al., showing that fewer than half of at-risk VA patients were tested for HIV, formed an important part of the basis of our rationale to proceed to step 4 [11]. These results were confirmed by further evaluations supported by core funding provided through the QUERI program. Moreover, the QUERI process guided our analysis of the importance of improving VA HIV testing rates, our assessment of the gaps in care, and the development of a multi-faceted project with broad-based institutional support to close this quality gap. Although the full analysis of our step 4/phase 1 two-station pilot project results is pending, and efforts to gauge the sustainability of our project are only just beginning, release of our preliminary results [47] has attracted considerable attention from other VA facilities, some of which are undertaking similar efforts to implement the electronic HIV Testing Clinical Reminder. The results of such implementations, which are being done without formal provider activation components, are not yet available. Specific issues of particular importance to our implementation program are discussed below.

Importance of leadership and team buy-in

The successful implementation of this project and the favorable preliminary results could not have been achieved without support at multiple administrative and clinical levels. The VA PHSHG provided invaluable support by promoting HIV testing within the VA, and by developing the HIV Testing Clinical Reminder. Their backing facilitated our ability to get the support of VISN 22 network leadership. In turn, this enabled us to elicit support for the institutionalization of the reminder, organizational changes, and implementation of the provider activation program at the individual stations where the step 4/phase 1 two-station pilot project was launched, and, subsequently to get the buy-in of individual primary team leaders. Ultimately, provider support is crucial. Acquiring this support required us to address issues of concern to providers, most notably facility-specific barriers to HIV testing, variances in the prevalence of HIV risk factors, issues related to the use of the reminder, and factors that contribute to variances in actual HIV testing versus refusal of HIV testing.

Station-specific barriers

Despite being a single healthcare system, there are substantial variances in local policies, processes, and institutional culture within the VA that must be addressed by station-specific customization of implementation projects. For example, differences between the processes of HIV consenting and counseling in stations A and B have required separate activation/education sessions with nursing staff regarding mandated requirements of the counseling/consenting process and the legal boundaries of their scope of practice. We also have found that further variations in training needs arise from differences in the mixtures of providers, such as the degree to which patient care is delivered by mid-level providers, post-graduate medical trainees, part-time physicians, contract employees, and full-time staff physicians. Furthermore, one station only allowed trained HIV counselors, who were positioned in key clinics and sub-stations, to order HIV tests or to perform pre- or post-test counseling. We needed to work with key stakeholders to change these policies. In addition, the rate of implementation of new technology varies. Thus, while electronic consent for HIV testing is available [48], acceptance of paperless consent requires concomitant local policy changes. Similar issues pertain to the implementation of rapid-testing for HIV infection – a procedure that has excellent patient acceptance in the VA [12].

Variations in the prevalence of HIV risk factors

Although individual providers generally have a panel size that is normalized to the time they spend in clinic, the proportion of veterans with HIV risk factors can vary dramatically. For example, a much higher proportion of patients trigger the HIV Testing Clinical Reminder in clinics that care for large numbers of veterans with drug use histories or hepatitis C infection. Providers in such clinics warrant attention to ensure that they do not feel overwhelmed by the burden of work precipitated by responding to these reminders.

Issues related to use of the reminder

As per the FITT model, reminders require customization to the needs of providers [34]. We have found that providers benefit from having ready access to the specific factor(s) that trigger the HIV Testing Clinical Reminder for an individual patient. In addition, it is important that the factors that trigger the Reminder be as closely linked to actual risks of HIV infection as possible. In this regard, we have found that the triggers for a diagnosis of homelessness are sufficiently non-specific (i.e., loss of housing for a few days) as not to warrant inclusion in the reminder. Similarly, the process triggered by the Reminder needs to be of potential benefit to the patient. Thus exclusion of persons with a very limited life expectancy is appropriate. Based on these findings, we are adopting a revision of the Reminder that addresses these issues.

HIV Test ordering versus HIV test refusal

The substation-specific variances in the ratio of HIV test orders to HIV test refusal was an unexpected finding. In response we are seeking to better understand the determinants of these behaviors. We anticipate that these will include provider comfort with HIV testing and site-specific barriers to HIV testing. We will also determine whether particular demographic factors, HIV risk factors or physician factors, are associated with patient refusal of HIV testing. Based on our findings, we will design a broader campaign of provider activation, such as reminding providers of patient acceptance of HIV testing [23] and delivery system changes to alleviate this discrepancy.

In summary, the strengths of our study include a quasi-experimental design that permits comparison of the effect of the implementation program on changes on HIV testing and evaluation rates – before and after the program is active and with control stations. This program relies heavily on the built-in quality improvement infrastructure in the VA, including the electronic medical record, clinical reminder software, and familiarity with performance measurements. Although unique aspects of the VA infrastructure will make it more difficult to generalize introduction of our process to healthcare systems that do not have these tools, such systems are becoming increasing common. Other limitations of our current findings include our incomplete qualitative process (or formative) evaluation to better understand the influences that impact the success of the program. In ongoing work we hope to identify contextually relevant factors (i.e., facilitators and barriers) and assess the degree that behaviors leading to improved testing performance become part of routine practice [38]. Areas of particular interest will be evaluating the contribution of nurse-based vs. physician-based HIV testing and evaluation, as well as the role of intensive provider activation, as this is the most costly and time consuming activity. Finally, although the overall cost-effectiveness of HIV screening is not in doubt, these models do not address variations in the complexity of screening programs across differing clinic settings, nor the costs of implementing a screening program [2, 3, 46].

Next steps

After completing our 12-month QUERI step 4/phase 1, two-station (site) pilot project, we will extend our work by performing a QUERI step 4/phase 2, small-scale, multi-site evaluation at all five VISN 22 stations. This will allow us to conduct a formal, multi-site formative and analytical evaluation, determine the sustainability of our project in our initial two-stations, determine the institutionalization of behaviors by our two-station pilot, and further refine and evaluate our implementation program.

If the phase 2 results are favorable, we plan to study the refined program in multiple geographic regions (QUERI step 4/phase 3), prior to a national roll-out of the program (QUERI step4/phase 4). We anticipate that with any widespread roll-out there will still be a need for the receiving facility to adapt/contextualize/customize the implementation program to maximize the likelihood of adoption. Finally, QUERI steps 5 and 6 call for documentation that best practices improve outcomes and that outcomes are associated with improved health-related quality of life. Therefore, of ultimate concern is whether increased rates of HIV screening and testing actually increase the rates of early HIV diagnosis, decrease HIV transmission and disease progression, and improve quality of life for HIV-infected and uninfected persons.


  1. VA HIV Report. 2005, 2: 1-4.

  2. Sanders GD, Bayoumi AM, Sundaram V, Bilir SP, Neukermans CP, Rydzak CE, Douglass LR, Lazzeroni LC, Holodniy M, Owens DK: Cost-effectiveness of screening for HIV in the era of highly active antiretroviral therapy. N Engl J Med. 2005, 352: 570-585. 10.1056/NEJMsa042657.

    Article  CAS  PubMed  Google Scholar 

  3. Paltiel AD, Weinstein MC, Kimmel AD, Seage GR, Losina E, Zhang H, Freedberg KA, Walensky RP: Expanded screening for HIV in the United States--an analysis of cost-effectiveness. N Engl J Med. 2005, 352: 586-595. 10.1056/NEJMsa042088.

    Article  CAS  PubMed  Google Scholar 

  4. McQueen L, Mittman BS, Demakis JG: Overview of the Veterans Health Administration (VHA) Quality Enhancement Research Initiative (QUERI). J Am Med Inform Assoc. 2004, 11: 339-343. 10.1197/jamia.M1499.

    Article  PubMed  PubMed Central  Google Scholar 

  5. Demakis JG, McQueen L, Kizer KW, Feussner JR: Quality Enhancement Research Initiative (QUERI): A collaboration between research and clinical practice. Med Care. 2000, 38: I17-I25. 10.1097/00005650-200006001-00003.

    Article  CAS  PubMed  Google Scholar 

  6. Stetler CB, Mittman BS, Francis J: Overview of the VA Quality Enhancement Research Initiative (QUERI) and QUERI theme articles: QUERI Series. Implementation Science. 2008, 3: 8-10.1186/1748-5908-3-8.

    Article  PubMed  PubMed Central  Google Scholar 

  7. Phillips B, Mole LA, Backus LI, Halloran JP, Chang SW: Caring for Veterans with HIV Disease- Fiscal Year 2002. 2003, Palo Alto, CA, Center for Quality Management in Public Health. Veterans Health Administration. Department of Veterans Affairs, 1-68.

    Google Scholar 

  8. Bozzette SA, Ake CF, Tam HK, Chang SW, Louis TA: Cardiovascular and cerebrovascular events in patients treated for human immunodeficiency virus infection. N Engl J Med. 2003, 348: 702-710. 10.1056/NEJMoa022048.

    Article  CAS  PubMed  Google Scholar 

  9. Patterson ES, Nguyen AD, Halloran JP, Asch SM: Human factors barriers to the effective use of ten HIV clinical reminders. J Am Med Inform Assoc. 2004, 11: 50-59. 10.1197/jamia.M1364.

    Article  PubMed  PubMed Central  Google Scholar 

  10. Patterson ES, Doebbeling BN, Fung CH, Militello L, Anders S, Asch SM: Identifying barriers to the effective use of clinical reminders: bootstrapping multiple methods. J Biomed Inform. 2005, 38: 189-199. 10.1016/j.jbi.2004.11.015.

    Article  PubMed  Google Scholar 

  11. Owens DK, Sundaram V, Lazzeroni LC, al. : HIV testing appropriateness and predictors of HIV infection in department of veterans affairs health care systems. Medical Decision Making. 2002, 22: 534-10.1177/027298902237969.

    Article  Google Scholar 

  12. Anaya H, Asch S: Improving HIV screening with rapid testing and streamlined counseling: 2006/8/17. Toronto, Canada, InternaionalAIDS Society; 2006:Abstract THLB0206.

  13. Groessl E, Goetz M, Looney D, Holodniy M, Ampel N, Glaser D, Gifford A: Implementing Pharmacist Support and PagerReminders for Antiretroviral (ARV) Adherence in Veterans: 2006/2/15. Volume Abstract 3084. Arlington, VA; 2006:92.

  14. Chou R, Huffman LH, Fu R, Smits AK, Korthuis PT: Screening for HIV: a review of the evidence for the U.S. Preventive Services Task Force. Ann Intern Med. 2005, 143: 55-73.

    Article  PubMed  Google Scholar 

  15. Screening for HIV: recommendation statement. Ann Intern Med. 2005, 143: 32-37. 10.1001/archinte.143.1.32.

  16. Perlin JB: Need for routine Human Immunodeficiency Virus (HIV) risk assessment and testing. Under Secretary for Health's Information Letter. 2005, IL 10-2005-017:

    Google Scholar 

  17. Stone PW, Teutsch S, Chapman RH, Bell C, Goldie SJ, Neumann PJ: Cost-utility analyses of clinical preventive services: published ratios, 1976-1997. Am J Prev Med. 2000, 19: 15-23. 10.1016/S0749-3797(00)00151-3.

    Article  CAS  PubMed  Google Scholar 

  18. Owens DK, Sundaram V, Douglass LR, al. : Seroprevalence of HIV infection at VA health care systems. Medical Decision Making. 2003, 23: 569-

    Google Scholar 

  19. Gandhi NR, Skanderson M, Concato J, Justice AC: Trends in healthcare utilization by HIV-positive veterans from 1998 to 2002: 2004/11/12. 2004, Tucson, AZ, []

    Google Scholar 

  20. Asch SM: Unpublished work. 2003

    Google Scholar 

  21. Centers for Disease Control and Prevention: Advancing HIV Prevention. Interim Technical Guidance for Selected Interventions. 2003, []

    Google Scholar 

  22. Kilmarx PH, Hamers FF, Peterman TA: Living with HIV. Experiences and perspectives of HIV-infected sexually transmitted disease clinic patients after posttest counseling. Sex Transm Dis. 1998, 25: 28-37. 10.1097/00007435-199801000-00007.

    Article  CAS  PubMed  Google Scholar 

  23. Irwin KL, Valdiserri RO, Holmberg SD: The acceptability of voluntary HIV antibody testing in the United States: a decade of lessons learned. AIDS. 1996, 10: 1707-1717.

    Article  CAS  PubMed  Google Scholar 

  24. Bodenheimer T, Wagner EH, Grumbach K: Improving primary care for patients with chronic illness: the chronic care model, Part 2. JAMA. 2002, 288: 1909-1914. 10.1001/jama.288.15.1909.

    Article  PubMed  Google Scholar 

  25. Thomson O'Brien MA, Oxman AD, Haynes RB, Davis DA, Freemantle N, Harvey EL: Local opinion leaders: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2005, CD000125-10.1002/14651858.CD000125.

    Google Scholar 

  26. Rogers EM: Diffusion of Innovations. 1995, New York, NY, The Free Press

    Google Scholar 

  27. Wagner EH: Chronic disease management: what will it take to improve care for chronic illness?. Eff Clin Pract. 1998, 1: 2-4.

    CAS  PubMed  Google Scholar 

  28. Renders CM, Valk GD, Griffin S, Wagner EH, Eijk JTM, Assendelft WJJ: Interventions to improve the management of diabetes mellitus in primary care, outpatient and community settings. Cochrane Database Syst Rev. 2005, CD001481-10.1002/14651858.CD001481.

    Google Scholar 

  29. Berwick DM: Developing and testing changes in delivery of care. Ann Intern Med. 1998, 128: 651-666.

    Article  CAS  PubMed  Google Scholar 

  30. Bodenheimer T, Wagner EH, Grumbach K: Improving primary care for patients with chronic illness. JAMA. 2002, 288: 1775-1779. 10.1001/jama.288.14.1775.

    Article  PubMed  Google Scholar 

  31. Hulscher ME, Wensing M, van der Weijden T, Grol R: Interventions to implement prevention in primary care. Cochrane Database Syst Rev. 2001, CD000362-

    Google Scholar 

  32. Asch SM, McGlynn EA, Hogan MM, Hayward RA, Shekelle P, Rubenstein L, Keesey J, Adams J, Kerr EA: Comparison of quality of care for patients in the Veterans Health Administration and patients in a national sample. Ann Intern Med. 2004, 141: 938-945.

    Article  PubMed  Google Scholar 

  33. Kawamoto K, Houlihan CA, Balas EA, Lobach DF: Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005, 330: 765-10.1136/bmj.38398.500764.8F.

    Article  PubMed  PubMed Central  Google Scholar 

  34. Ammenwerth E, Iller C, Mahler C: IT-adoption and the interaction of task, technology and individuals: a fit framework and a case study. BMC Med Inform Decis Mak. 2006, 6: 3-10.1186/1472-6947-6-3.

    Article  PubMed  PubMed Central  Google Scholar 

  35. Jamtvedt G, Young JM, Kristoffersen DT, O'Brien MA, Oxman AD: Does telling people what they have been doing change what they do? A systematic review of the effects of audit and feedback. Qual Saf Health Care. 2006, 15: 433-436. 10.1136/qshc.2006.018549.

    Article  PubMed  PubMed Central  Google Scholar 

  36. Thomson O'Brien MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL: Educational outreach visits: effects on professional practice and health care outcomes. Cochrane Database Syst Rev. 2000, CD000409-

    Google Scholar 

  37. Lefebvre RC, Rochlin L: Social marketing. Health Behavior and Health Education: Theory, Research, and Practice. Edited by: Lewis FM and Rimer BK. 1988, Jossey-Bass, 2nd

    Google Scholar 

  38. Stetler CB, Legro MW, Wallace CM, Bowman C, Guihan M, Hagedorn H, Kimmel B, Sharp ND, Smith JL: The role of formative evaluation in implementation research and the QUERI experience. J Gen Intern Med. 2006, 21 Suppl 2: S1-S8.

    Article  PubMed  Google Scholar 

  39. Pre-Test Counseling and Consent for HIV Testing. VHA-10-0121. 2005, []

  40. Robson J, Boomla K, Fitzpatrick S, Jewell AJ, Taylor J, Self J, Colyer M: Using nurses for preventive activities with computer assisted follow up: a randomised controlled trial. BMJ. 1989, 298: 433-436.

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  41. Gonzalez JJ, Ranney J, West J: Nurse-initiated health promotion prompting system in an internal medicine residents' clinic. South Med J. 1989, 82: 342-344.

    Article  CAS  PubMed  Google Scholar 

  42. Anaya HD, Golden J, Asch SM: Improving HIV Screening by Nurse Rapid Testing/Streamlined Counseling. 2006

    Google Scholar 

  43. Kendrick SR, Kroc KA, Withum D, Rydman RJ, Branson BM, Weinstein RA: Outcomes of offering rapid point-of-care HIV testing in a sexually transmitted disease clinic. J Acquir Immune Defic Syndr. 2005, 38: 142-146. 10.1097/00126334-200502010-00004.

    Article  PubMed  Google Scholar 

  44. Sobo E, Simmes D, Landsverk J, Kurtin PS: Rapid assessment with qualitative telephone interviews: Lessons from an evaluation of California's Healthy Families program & Medi-cal for children. J Evaluation. 2003, 24: 399-408.

    Article  Google Scholar 

  45. Goodman RM, McLeroy KR, Steckler AB, Hoyle RH: Development of level of institutionalization scales for health promotion programs. Health Educ Q. 1993, 20: 161-178.

    Article  CAS  PubMed  Google Scholar 

  46. Bozzette SA: Routine screening for HIV infection--timely and cost-effective. New England Journal of Medicine. 2005, 352: 620-621. 10.1056/NEJMe048347.

    Article  CAS  PubMed  Google Scholar 

  47. Goetz MB, Rossman B, Hoang T, Anaya H, Burgess J, Volpp BD, Osborn T, Bowman C, Gifford A, Asch S, QUERI-HIV VA: Implementationof a program to improve clinic - based HIV screening and testing: 2006/8/13. Toronto, Canada, International AIDS Society; 2006:Abstract WEPE0216.

  48. HIV Testing in the Department of Veterans Affairs. 2005, []

Download references


We acknowledge the contributions of Robert M. Smith, MD for his invaluable contributions to the development and implementation of the HIV testing clinical reminder. This work was supported by the U.S. Department of Veterans Affairs, Health Services Research and Development Service (Service Directed Project 06-001). The views expressed in this article are those of the authors and do not necessarily reflect the position or policy of the Department of Veterans Affairs.

Author information

Authors and Affiliations


Corresponding author

Correspondence to Matthew B Goetz.

Additional information

Competing interests

The author(s) declare that they have no competing interests.

Authors' contributions

MBG drafted the manuscript. CB, TH, HA, TO, AG and SA participated in the design of the study. TH performed the statistical analysis. All authors read and approved the final manuscript.

Authors’ original submitted files for images

Below are the links to the authors’ original submitted files for images.

Authors’ original file for figure 1

Authors’ original file for figure 2

Rights and permissions

This article is published under license to BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

Reprints and permissions

About this article

Cite this article

Goetz, M.B., Bowman, C., Hoang, T. et al. Implementing and evaluating a regional strategy to improve testing rates in VA patients at risk for HIV, utilizing the QUERI process as a guiding framework: QUERI Series. Implementation Sci 3, 16 (2008).

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: