OBJECTIVES: To evaluate the impact of clinician-targeted computer-generated reminders on compliance with HIV care guidelines in a resource-limited setting.
METHODS: We conducted this randomized, controlled trial in an HIV referral clinic in Kenya caring for HIV-infected and HIV-exposed children (<14 years of age). For children randomly assigned to the intervention group, printed patient summaries containing computer-generated patient-specific reminders for overdue care recommendations were provided to the clinician at the time of the child’s clinic visit. For children in the control group, clinicians received the summaries, but no computer-generated reminders. We compared differences between the intervention and control groups in completion of overdue tasks, including HIV testing, laboratory monitoring, initiating antiretroviral therapy, and making referrals.
RESULTS: During the 5-month study period, 1611 patients (49% female, 70% HIV-infected) were eligible to receive at least 1 computer-generated reminder (ie, had an overdue clinical task). We observed a fourfold increase in the completion of overdue clinical tasks when reminders were availed to providers over the course of the study (68% intervention vs 18% control, P < .001). Orders also occurred earlier for the intervention group (77 days, SD 2.4 days) compared with the control group (104 days, SD 1.2 days) (P < .001). Response rates to reminders varied significantly by type of reminder and between clinicians.
CONCLUSIONS: Clinician-targeted, computer-generated clinical reminders are associated with a significant increase in completion of overdue clinical tasks for HIV-infected and exposed children in a resource-limited setting.
- AMPATH —
- Academic Model Providing Access to Healthcare
- AMRS —
- AMPATH Medical Record System
- ART —
- antiretroviral therapy
- CDSS —
- clinical decision support systems
- EHR —
- electronic health record
- ELISA —
- enzyme-linked immunosorbent assay
- USAID —
- US Agency for International Development
- +ve —
What’s Known on This Subject:
Of more than 2 million children infected with HIV, almost 90% live in resource-limited settings where pediatric HIV care is often suboptimal. Implementing electronic health records with computerized decision support offers a potential tool for improving care.
What This Study Adds:
This randomized, controlled trial demonstrates that computer-generated clinical reminders can significantly improve clinician compliance with HIV care guidelines for children in a resource-limited setting. This intervention is scalable as developing countries implement electronic health record systems.
The World Health Organization estimates that 2.3 million children live with HIV worldwide,1 with almost 90% of affected children found in Sub-Saharan Africa.2 Few studies document the extent to which HIV care programs in these resource-limited settings are able to meet the pediatric care needs,3 but the limited evidence available suggests that there is significant room for improvement. Care programs report difficulties in establishing systems for early infant diagnosis of HIV, initiating antiretroviral therapy (ART) promptly, ordering baseline and follow-up laboratory investigations, detecting tuberculosis, maintaining prophylaxis for opportunistic infections, and managing child malnutrition.4–8 These challenges occur at multiple levels, ranging from individual clinical care systems to the country level.
Scalable and evidence-based approaches are needed to improve quality of care for HIV-infected children in resource-limited settings. Many of these settings are implementing electronic health records (EHRs) for this purpose.9 However, little rigorous evidence exists to demonstrate whether EHRs will effectively improve HIV care. In resource-rich settings, Clinical Decision Support Systems (CDSS), which use data stored in EHRs to provide care suggestions and reminders to clinicians, generally improve clinician behaviors and quality of care.10,11 For HIV care in the United States, computer-based alerts and reminders significantly improved clinicians’ adherence to HIV care guidelines for adults.12 In Kenya, a comparative study between 2 clinics showed that clinical summaries with computer-generated reminders improved clinicians’ compliance with CD4-testing guidelines for adults, but did not evaluate other aspects of HIV care.13 Rigorously controlled trials of CDSS’ effectiveness in resource-limited settings could not be found. Whether CDSS might address the particular challenges associated with pediatric HIV care or improve quality of care remains to be seen. In this clinic-level study, set in a resource-limited HIV care system in western Kenya, we used a randomized, controlled design to rigorously evaluate the impact of delivering computer-generated, patient-specific reminders to clinicians on their rates of compliance with pediatric HIV care guidelines.
This study was conducted in a large pediatric HIV clinic at the Moi Teaching and Referral Hospital in Eldoret, Kenya. This clinic is the referral site for the US Agency for International Development (USAID)-funded Academic Model Providing Access to Healthcare (AMPATH) in western Kenya.14 At the time this study was initiated (February 2011), this pediatric HIV clinic had enrolled 5140 HIV-exposed and HIV-infected children aged 0 to 14 years since its inception in 2003 and had started 1670 HIV-infected children on ART. HIV-infected children on ART are typically seen on a monthly basis in AMPATH, and those not yet meeting criteria to start ART are seen every 3 to 6 months. HIV-exposed children are followed monthly until 18 months of age. Those found to be uninfected are followed every 6 months until their fifth birthday.
Electronic Health Record System
Since 2004, USAID-AMPATH clinics have used the AMPATH Medical Record System (AMRS) to store comprehensive, longitudinal, electronic patient records for all enrolled patients.15 AMRS is the original implementation of OpenMRS, an open-source EHRs deployed widely in the developing world.16,17 Patient records in the system contain demographic information, historical and physical examination data, problem lists, medications, diagnostic test results, and visit data. Clinical information is stored as coded concepts (as opposed to free text) for easy retrieval and analysis.18 Clinicians caring for AMPATH patients do not enter data directly into AMRS but rather complete paper encounter forms (see Supplemental Appendix A). This is largely due to the lack of physical space and technical infrastructure to support computer access in every examination room. Clerks enter data from the completed paper forms into AMRS, with a random 10% subset of the data entered checked for accuracy. The encounter forms are then placed in the patient’s paper clinic chart, which is available to clinicians during patient care. Laboratory results are transferred electronically from the central laboratory information system into AMRS, with a paper copy of the same results placed into the patient’s chart.
Clinical Summaries and Reminders
We developed a module within OpenMRS that generated a patient-specific clinical summary. This summary, tailored for pediatric care, displayed selected information from the patient’s EHR to provide a quick reference to the most relevant data needed by clinicians (Supplemental Appendix B). The module also contained CDSS functionality that appended patient-specific care reminders to the bottom of the clinical summary.19 New summaries, with or without reminders, were generated for all patients in a portable document format every 4 days as part of a batch-generation process. Summaries could also be generated on demand as needed. For patients with a scheduled visit date, the generated summaries were typically printed by clinic intake personnel on the afternoon before the scheduled visit. Summaries for unscheduled patients were printed when the patient presented at the clinic. All summaries were attached to the relevant patients’ paper charts for clinicians to review during a patient’s clinic visit.
For this study, we implemented CDSS consisting of care reminders for overdue tests and treatments that were indicated based on standard pediatric HIV care protocols. These protocols were based on recommendations by the World Health Organization20 and the Kenya Ministry of Health.21 We implemented reminders to order overdue 6-week HIV DNA polymerase chain reaction tests, 18-month enzyme-linked immunosorbent assay (ELISA) antibody tests, CD4 tests, routine laboratory studies, and chest radiographs; to initiate ART; and to refer malnourished children for nutritional evaluation and assistance (Table 1).
Randomization and Study Population
We assessed the effect of the patient-specific, computer-generated pediatric reminders on compliance with pediatric HIV care guidelines by using a randomized, controlled trial. All patients, both HIV-exposed and HIV-infected, previously enrolled at the study site were randomly assigned to either the intervention or control group in a 1:1 ratio by using a 4-block randomization scheme. Study patients were HIV-infected or HIV-exposed children <14 years of age presenting for return clinical visits at the outpatient HIV clinic. New patients enrolling in the clinic during the 5-month evaluation were excluded from the study. Patients stayed in the same study group throughout the 5-month duration of the study. We randomized by patient instead of by clinician, because patients typically saw whichever clinician was first available at the time of their visit, and it was not possible to tell in advance which patient a clinician would see. We understood that this could sensitize clinicians to order the indicated care for control patients, which might bias our study against finding a significant effect for the reminders.
If an intervention patient was overdue for 1 or more of the care interventions described above, the reminder to complete that overdue task was printed at the bottom of the patient’s paper clinical summary when they presented to the clinic (Supplemental Appendix B). An example of a printed reminder was: “Please order Chest X-ray. +ve polymerase chain reaction or ELISA but no X-ray.” Clinicians were asked to document their response to each reminder within the clinical summary by selecting among 7 responses, namely: (1) ordered today; (2) not applicable (explain); (3) previously ordered; (4) patient allergic; (5) patient refused; (6) I disagree with reminder (explain); or (7) other (explain). Each type of reminder was assigned a unique number that appeared at the top of the printed patient clinical summary (Supplemental Appendix B: in this example, the number assigned for “order chest x-ray” reminder type is “6”). No more than 5 reminders were displayed for each patient per visit, so as not to overwhelm clinicians. Reminders were displayed in numerical order, with no preference or prioritization to any particular reminder. For intervention patients, if the recommended action was not taken, the reminder for that task would again be printed on the clinical summary at the next visit. Reminders were repeated at subsequent visits as long as the action remained overdue. For control patients, reminders were not printed on the paper summaries when the patients had overdue orders at any of the visits.
For both intervention and control patients, clinicians recorded actions taken as part of routine care (including test and treatment orders) on the paper encounter forms (Supplemental Appendix A: Sections 24–26). All clinical summaries and encounter forms were subsequently reviewed by 2 research assistants who extracted information on reminder type, study group, clinician, completion of targeted overdue tasks, and documented reasons in the clinical summaries when care suggestions were not implemented. Completion of overdue tasks was considered to have occurred if there was documentation of ordering an investigation, medication, or referral either within the patient’s paper chart or within the EHR. We also queried the EHRs 3 months after study closure (October 7, 2011) to determine if any other relevant results had returned.
Before our study, clinicians had received paper clinical summaries without any reminders on patients’ charts for ∼1 year. In the first 2 weeks of the training period (February 1–13, 2011), only reminder tracking numbers were printed on the top of each clinical summary for all patients. Clinicians were informed that these numbers were for internal use by the clinical decision support team. During the next 2 weeks (February 14–28, 2011), reminders were added to printed summaries, and clinicians were trained on how to respond to them in accordance with pediatric care protocols. Random selection of patients was done on February 28, 2011, and the study ran until August 8, 2011. The study was approved by the institutional review boards at Indiana University School of Medicine in Indianapolis, Indiana, and by the Institutional Review and Ethics Committee at Moi University School of Medicine, Kenya. The review boards did not require informed consent from patients or providers, because reminders have been considered routine components of care in the HIV clinics at AMPATH.
Continuous baseline variables are summarized by mean and SD, with age summarized by using median and interquartile range. Categorical variables are summarized by frequency and percentage. Student t tests were used to compare means for continuous variables. χ2 and Fisher exact tests were used to compare distributions for categorical variables. Demographic data used were obtained as part of the routine procedures for clinic registration and patient care.
The primary outcome of interest was the number of visits (inclusive) before a recommended action had been fulfilled and documented by a provider in the patient’s chart. Toward this goal, the analysis unit was the response to a specific recommended action at a particular visit, which was dichotomized as “Ordered” or “Not Ordered” at each visit. A care suggestion was considered ordered only when explicit documentation of the relevant order existed within the patient’s clinical encounter form, or when the requisite laboratory result was found within the EHR. We used a discrete survival model to test for differences in parameters that govern the cumulative rate of correcting overdue tasks for control and intervention groups (see Supplemental Appendix C for details of analysis). This was done for each type of reminder and for all reminders combined. Therefore, the primary hypothesis was tested through tests of equality of the model parameters between intervention and control groups. In our model, we included health care provider as a random effect.
All analyses were performed by using SAS 9.3 (SAS Institute, Cary, NC). A 2-sided P value of .05 was considered statistically significant. Bonferroni correction of the P values was made to adjust for multiple comparisons for the primary end point.
Patient and Provider Demographics
A total of 3993 HIV-positive or -exposed children were randomly assigned to the study. Of these, 1619 patients (41%) were eligible to receive at least 1 computer-generated reminder (ie, had an overdue order or referral) during the study period (Table 2). Eight of the 1619 were subsequently excluded from the analysis because a clinician involved in the study saw them. Median age was 6.17 years, with an interquartile range of 7.91 (p25 = 2.67, p75 = 10.58). Seventy-one percent of the eligible children were HIV-infected, and 784 (49%) were on ART. There were no statistically significant differences in the demographic characteristics between the control and intervention groups. A total of 1142 children (609 control and 533 intervention) were seen more than once during the study period.
Thirty clinicians saw the study patients during clinic visits, and, of these, 22 provided demographic information. The clinicians were primarily clinical officers (equivalent to physician assistants) (71%), but 1 attending physician (5%) and 5 nurses (24%) also provided care. The clinicians were mostly women (68%) and had worked with AMPATH for a mean of 52 months (SD 34) and at the study clinical site for a mean of 40 months (SD 34). Clinicians spent a mean of 4.4 days per week in the study site (SD 1.2).
Response to Care Reminders
In the intervention group, 825 patients (43%) had at least 1 unmet care protocol, resulting in computer generation and delivery of 2153 unique care reminders on their clinical summaries. In the control group, 786 patients (40%) had an unmet care protocol, generating 2843 unique reminders, but these were not printed on the clinical summaries. Table 3 shows the cumulative rates for correcting overdue clinical tasks, by visit, for control and intervention group patients. The intervention group had significantly higher rates of completing overdue clinical tasks at the first visit in the study period and across the subsequent visits. Over the course of the study, there was a fourfold increase in completing overdue clinical tasks when the patient with an overdue task had a reminder printed on their clinical summary sheet (68% completion in intervention group vs 18% in control group, P < .001) (Table 3).
Overdue clinical tasks were also corrected earlier when reminders were presented to clinicians. For the intervention group, the mean time from when a patient had an overdue clinical task to the completion of that task was 77 days (SE 2.4 days), whereas the mean time to correction was 104 days (SE 1.2 days) for the control group (P < .001 based on test for equality over strata) (Fig 1).
The impact of reminders varied by type of care suggestion (Table 3). Rates of correcting overdue clinical tasks were significantly greater for the intervention group compared with the control group in response to reminders for ordering HIV ELISA tests (P < .001); chest radiographs (P < .001); other tests including chemistries and blood counts (P < .001); and for making referrals for malnutrition (P < .001). Rates of initiating ART for qualifying children were higher in the intervention group than in the control group, but this difference did not reach statistical significance (79% vs 62%, P = .28).
Response rates to clinical reminders also varied by clinician. Of the 30 clinicians who saw patients during the study period, 8 clinical officers saw >50 reminders for intervention patients. The cutoff of 50 reminders was chosen to identify providers who had enough exposure and familiarity with the reminder system beyond the learning phase. Among these 8 clinical officers, rates of correcting overdue clinical tasks in response to a reminder ranged from 11% to 32% per encounter. Seven of the 8 clinical officers had significantly higher rates of correcting overdue clinical tasks when reminders were presented in comparison with when there were no reminders (P < .001). The 1 clinician who did not have a significant increase still had higher rates of correction per encounter with presentation of reminders (4.6%–11.2%, P = .0621) (Table 4).
When reminders did not result in performing the suggested task during a visit for an intervention patient, the following reasons were provided by the clinicians: test was previously ordered (28%); clinician planned to order at next visit, patient refused, or other deferral not specified (25%); or provider disagreed with reminder or considered it not applicable (13%). In 13% of encounters where a reminder for an intervention patient did not result in correction of the overdue task, no reason was provided as to why the reminder was not followed.
In this randomized, controlled trial of CDSS to improve pediatric HIV care in a resource-limited setting, we found that including patient-specific, computer-generated reminders on a paper-based clinical summary significantly improved adherence to pediatric HIV care protocols. Presenting the clinicians with these reminders also significantly decreased the time to correction of overdue clinical tasks, with the observed improvements seen throughout the duration of the study.
These findings could have particular clinical importance for the pediatric HIV population, in which early mortality and morbidity are high. Early diagnosis of HIV and prompt initiation of ART can reduce children’s high mortality by as much as 76%,22 but our findings suggest that many of the recommendations for pediatric HIV care are not performed in a timely fashion. Clinicians may have difficulty adhering to care protocols because they care for large numbers of patients, they might not recall guideline recommendations, or they cannot readily determine whether a given patient needs a particular test or therapy. A CDSS intervention that addresses these challenges and significantly improves the quality of pediatric HIV care could yield real clinical improvements.
Despite the improvements in care practices for the intervention group, the response rates did vary by type of reminder. This suggests that clinicians found some reminders more relevant or critical. In addition, reminders never resulted in perfect compliance with care protocols for reasons similar to those found in other settings.23 The recommended orders were sometimes appropriately deferred because they were not applicable, the EHR data generating the reminder was deficient, or because other patient factors made it appropriate to wait. Allowing clinicians to give reasons for not following the reminder reveals whether the continued noncompliance with protocols is appropriate. Although reminders did not result in perfect adherence with care protocols, significant improvements in care practices were seen across a variety of reminder types, ranging from test ordering to referral patterns. The findings suggest that CDSS can have broad applicability across multiple care scenarios.
This study underscores that completeness and quality of data within an EHR are crucial for the successful implementation of CDSS. Without access to complete and high-quality data, CDSS cannot offer relevant, timely, and accurate reminders. For example, in 28% of the clinical encounters where clinicians were given a reminder for an overdue task but did not complete this task at that visit, clinicians reported that the test, medication, or referral had previously been ordered. In these cases, the relevant information was likely not reflected in the EHR, suggesting a breakdown in the system connecting the clinician, patient, laboratory, and EHR. Such data quality issues deserve additional attention, because clinicians can become indifferent to reminders that are consistently inaccurate.24
This study has some limitations. It was conducted in a single HIV clinic with a limited number of clinicians, potentially limiting its generalizability. Nonetheless, these constraints enabled conduct of a rigorous, randomized, controlled trial to provide high-quality evidence. In addition, the study included a large sample of 1611 children, and this may improve its applicability to other resource-limited settings. Another limitation is that the study was conducted in a setting where patients sometimes saw different clinicians on different visits, and the study findings might not necessarily translate to institutions where clinicians routinely follow up the same patients. It should also be emphasized that computer-generated reminders are only as good as the rules programmed into the computer. Although some health care systems may not have a robust EHR in place, the OpenMRS system used at the study site has been widely implemented in over 20 African countries and identified for broad rollout across countries like Kenya and Rwanda. The widespread and expanding use of OpenMRS means that our findings will be directly translatable to a large number of sites. In fact, our CDSS system has now been rolled out to all 51 AMPATH clinical sites. We hope to evaluate the impact of CDSS on clinical outcomes in the future.
This randomized, controlled trial provides robust evidence that patient-specific, computer-generated clinical reminders are associated with improved completion of overdue tasks relevant to pediatric HIV care in a resource-limited setting. This technology could be widely applied to improve compliance with care protocols in other resource-limited settings.
We thank our research team, Nyoman Ribeka, and AMPATH patients and providers.
- Accepted November 19, 2012.
- Address correspondence to Martin C. Were, MD, MS, Regenstrief Institute, Inc, and Indiana University School of Medicine, 410 W 10th St, Indianapolis, IN 46254. E-mail:
Dr Were conceived of and designed the study, implemented the acquisition of data, supervised the analyses, drafted and revised the article, and approved the final article as submitted; Dr Nyandiko was involved in the design and implementation of the study and contributed to the critical revisions of the manuscript, and approved the final article as submitted; Dr Huang designed the data collection instruments and supervised the acquisition and cleaning of data in Kenya, she also participated in the interpretation of data and critical revisions of the article, and approved the final article as submitted; James Slaven conducted the initial analyses, critically revised the manuscript, and approved the final article as submitted; Dr Shen supervised the design and conduct of the analyses, including their revisions, critically revised the manuscript, and approved the final article as submitted; Dr Tierney participated in the design and concept of the study, including implementation of the decision support framework within the study site, and he contributed to the critical revision of the article and approved the final article as submitted; and Dr Vreeman partnered with Dr Were in the conceptualization and design of the study, she supervised the implementation in the pediatric clinic in Kenya, evaluated the data collection and analyses, drafted portions of the article, revised it critically, and approved the final article as submitted.
This trial has been registered at clinicaltrials.gov (identifier NCT01235247).
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose
FUNDING: This work was supported primarily by Abbott Fund and CDC grant R18 HK000058. The work was also supported in part by the joint support for the USAID-AMPATH Partnership from the US Agency for International Development (USAID). Dr Were is supported by a grant from Abbott Fund and RWJF Grant RWJ68522 as a Harold Amos Medical Faculty Program Scholar. Dr Vreeman is supported by a grant from the National Institute for Mental Health (NIMH) (1K23MH087225-01). The contents of this study are the sole responsibility of the authors and do not necessarily reflect the views of the funders or institutions.
- ↵Antiretroviral Therapy for HIV Infection in Infants and Children: Towards Universal Access. Geneva, Switzerland: World Health Organization; 2010
- ↵UNAIDS. 2010 Report on the global AIDS epidemic: The Joint United Nations Programme on HIV/AIDS. Geneva, Switzerland: UNAIDS; 2010
- Alemayehu YK,
- Bushen OY,
- Muluneh AT
- Mamlin BW,
- Biondich PG,
- Wolfe BA,
- et al
- Were MC,
- Mamlin BW,
- Tierney WM,
- Wolfe B,
- Biondich PG
- World Health Organization
- ↵Guidelines for Antiretroviral Drug Therapy in Kenya 2005. Nairobi, Kenya: National AIDS and STI Control Program, Ministry of Health; 2005
- ↵Perna G. Clinical alerts that cried wolf. As clinical alerts pose physician workflow problems, healthcare IT leaders look for answers. Healthc Inform. 2012;29(4):18, 20
- Copyright © 2013 by the American Academy of Pediatrics