BACKGROUND AND OBJECTIVES: Timely provision of developmental services can improve outcomes for children 0 to 3 years old with developmental delays. Early Intervention (EI) provides free developmental services to children under age 3 years; however, data suggests that many children referred to EI never connect to the program. We sought to ensure that 70% of patients referred to EI from an academic primary care clinic serving a low-income population were evaluated within 120 days of referral.
METHODS: Recognizing that our baseline system of EI referrals had multiple routes to referral without an ability to track referral outcome, we implemented a multifaceted referral process with (1) a centralized electronic referral system used by providers, (2) patient navigators responsible for processing all EI referrals submitted by providers, and (3) a tracking system postreferral to facilitate identification of patients failing to connect with EI.
RESULTS: The percentage of patients evaluated by EI within 120 days increased from a baseline median of 50% to a median of 72% after implementation of the systems (N = 309). After implementation, the centralized referral system was used a median of 90% of the time. Tracking of referral outcomes revealed decreases in families refusing evaluations and improvements in exchange of information with EI.
CONCLUSIONS: Rates of connection to EI improved substantially when referrals were centralized in the clinic and patient navigators were responsible for tracking referral outcomes. Knowledge of EI intake processes and relationships between the practice and the EI site are essential to ensure successful connections.
- EI —
- early intervention
- EMR —
- electronic medical record
- PN —
- patient navigator
- QI —
- quality improvement
Standardization of developmental screening in young children has improved identification of developmental delays and increased opportunities for referral to therapeutic services such as early intervention (EI).1–6 EI provides affordable, evidence-based physical, cognitive, social, and adaptive therapies for children <3 years of age who are experiencing or are at risk for developmental delays.7 EI services have been shown to improve developmental outcomes,8–10 mitigate secondary behavioral complications, and increase caregiver confidence.9,11,12 However, in the literature, it has been demonstrated that the majority of children eligible for EI are not served by the program.13,14 Under-diagnosis of developmental delay, differences in referral practices among providers, and logistical barriers in connecting to EI can all hinder children and their families from receiving the therapies they need.13,15,16
In several recent studies, researchers show that a gap exists between identification of developmental delay in the primary care setting and initiation of EI services, with authors in 2 recent studies citing that only 30% to 51% of patients referred to EI underwent evaluation.1,15 Known barriers to connection for families include not understanding the referral process, concerns that the child or family will experience stigma if delays are revealed, and logistical barriers to obtaining services.16,17 Because minimizing long-term disability depends on both appropriate screening as well as connection to effective developmental services, it is essential that developmental screening in primary care be bolstered by systems that facilitate referral completion and follow-up.
Although the primary care practice is not ultimately responsible for conducting EI evaluations, it plays a key role in activating families to pursue an evaluation, connecting families to EI, and helping families overcome any barriers to referral completion. Despite this, there are few published models in which systems that effectively connect primary care patients with EI programs are described.18–21 Baseline analysis in our clinic revealed that only 50% of patients referred to EI underwent evaluation. Thus, we engaged in a quality improvement (QI) study in which we designed a new EI referral process with (1) systems to better activate families to connect with EI, (2) clear referral pathways agreed on by EI sites, and (3) a postreferral tracking system to identify those failing to connect. The overall goal of this system redesign was to ensure that 70% of patients referred to EI were evaluated by the program.
Setting and Patient Population
This QI initiative was implemented at an academic hospital-based primary care clinic that cares for ∼16 000 patients, 17% of whom are <3 years of age and potentially eligible for EI services. Families reside primarily in urban neighborhoods, and 68% are Medicaid insured. The pediatric provider team consists of 22 attending physicians, 4 nurse practitioners, and 66 resident physicians. Developmental screening is standardized within the primary care practice; the Parent Evaluation of Developmental Status is administered at every well-child visit for patients aged 6 months to 4 years, and the Modified Checklist for Autism in Toddlers is completed at 18 and 24 months. At baseline, these developmental screens were performed at over 95% of eligible visits. Twenty percent of well-child visits at our clinic are billed as having a developmental-behavioral concern, which is consistent with findings that children living in poverty experience the highest rates of disability.22
The EI program in Massachusetts is funded and administered by the Department of Public Health. There is no central referral system. Rather, the state is divided into catchment areas, and each program manages its own intake and evaluation processes. Sixty percent of our patients live in the city of Boston; in Boston alone, there are 6 EI programs, of which 5 have overlapping catchment areas. There is a state registry of EI participants, but children are entered into the database after an intake visit has been scheduled rather than at the time of referral receipt, and the registry is not publicly accessible.
Designing the Intervention
A multidisciplinary improvement team (physicians, patient navigator [PN], social worker, project manager, and research assistant) was formed to decrease the gap between primary care EI referral and EI evaluation. A clinic-wide provider focus group was held at the onset of the project to identify and map baseline EI referral routes. Next, we surveyed the literature and performed key informant interviews to identify drivers of successful connection to EI (Fig 1). The improvement team met regularly to create an “ideal state” flowchart of the referral and follow-up process, identify potential system changes to be tested through Plan-Do-Study-Act cycles, discuss ongoing implementation of systems changes, and review data to assess the efficacy of the tests of change.
Components of the Intervention
During the planning phase, the improvement team found that the clinic did not uniformly promote the EI program, used multiple routes to refer patients to EI, and lacked a system for tracking referrals after they were made. In response, we designed a new process with a focus on 3 of the 4 key drivers of successful EI referral:
Patient and provider activation: Improvement team members met with local EI staff to review eligibility criteria and learn best practices in motivating families to connect with EI. The improvement team then developed an EI brochure available in multiple languages to better educate families on EI’s services and evaluation process (Supplemental Figs 5 and 6).
Centralizing and tracking referrals through an EI registry: Referral routes to connect patients with EI at baseline ranged from calling the relevant EI site, giving the family the telephone number, and/or asking support staff (eg, PNs) to assist with referrals. To streamline referral routes, we encouraged use of an electronic order form within our electronic medical record (EMR) to direct the referral from the provider to the team PN. PNs were asked to process all EI referrals and track their outcomes in a newly developed EI registry. The team worked to define potential referral outcomes other than evaluation (eg, lost to follow-up). These outcome categories were tracked over time to identify additional avenues for process improvement.
Partnership with EI sites: Given the independent nature of EI sites in Massachusetts, intake processes differ considerably across sites. After our initial meetings with our local EI sites, we worked directly with the EI site to customize information flow appropriately to each EI site. All exchanged information was tracked in the EI registry.
Study of the Intervention
QI measures were developed to assess project performance throughout the initiative. The primary outcome measure was the percentage of patients referred to EI each month who received an EI evaluation within a timely manner, defined as within 120 days of referral. The goal of 70% for this measure was chosen because it represented an improvement of 1 SD (1 σ) from our baseline data. We measured provider adherence to the new referral system by examining the percent of EI referrals each month made by using the computerized referral system within the EMR. Finally, to identify opportunities for ongoing improvement, we tracked the status of each referral at the 120-day mark. The rate of completion of developmental screening for children 6 months to 3 years old was followed as a balancing measure to ensure that the new EI referral process did not impose a burden on providers and thus discourage screening. Data were drawn from the EMR and an excel registry maintained by the PNs. This study was approved by the Boston Children’s Hospital institutional review board.
Our primary outcome and process measures were followed on run charts and statistical process control charts, which allowed us to assess system stability and identify special cause variation and process shifts.23 We used t tests, χ2 testing, and multivariate logistic regression to identify demographic and clinical predictors of successful evaluation.
Between December 2014 and December 2015, 309 children were referred to EI from our primary care clinic. Sixty-four percent were boys, and 77% were publicly insured (Table 1). Sex, caregiver English language proficiency, race and/or ethnicity, and age at referral did not have a significant association with timely referral. However, those who were privately insured were more likely to have timely evaluation compared with those with public insurance (82% vs 67%, P = .01). Of patients referred to EI in our cohort, 59% had a concurrent or previous diagnosis of speech delay, 13% had a diagnosis of motor delay, and 30% had a diagnosis of a significant medical need (such as prematurity, technology dependence, or another chronic disease). A previous diagnosis of “developmental delay” or “motor delay” was associated with higher rates of timely evaluation compared with not having these diagnoses (P = .03 and .01, respectively). In multivariate logistic regression, after controlling for age at referral, sex, English language proficiency, and race and/or ethnicity, only motor delay and insurance status remained significant in the model, with those with motor delay more likely to have a timely evaluation (odds ratio: 2.82 [95% confidence interval: 1.02–7.79]), and those with public insurance were less likely to have a timely evaluation (odds ratio: 0.44 [95% confidence interval: 0.21–0.90]).
The percentage of patients evaluated by EI within 120 days of referral (primary outcome) rose from a median of 50% in the baseline period to a median of 72% during the intervention period (Fig 2) and was in statistical control throughout the study period. The rate of evaluation increased from 56% to 93% because the EI registry was trialed by 1 PN and then spread to the other PNs (December 2014 to February 2015). The rate of evaluation declined in association with an unexpected 4-day downtime in the EMR, as well as with local EI staffing shortages, but these did not meet criteria for special cause variation.
We conducted a series of Plan-Do-Study-Act cycles regarding communication with EI sites. First, we trialed different strategies to communicate with EI sites to obtain follow-up information on evaluation status. Although we hypothesized that communication would be best if each PN was responsible for communication with 1 of our top 3 EI sites, referral completion did not increase during this trial (March 2015 to June 2015). Rather, when PNs managed the referrals they themselves had placed, rates of evaluation improved. Next, we varied how frequently and with what systems we communicated with the EI sites. An initial trial of using a phone call every 2 weeks directly to the referral coordinator revealed good results with some sites but not with others. Rates of evaluation increased from 61% to 78% after we tailored communication systems to the preferences of individual sites (July 2015 to December 2015).
In our redesigned system, providers were asked to use an electronic order to alert the PN about the EI referral; this electronic order had been available before the intervention but was not consistently used (Fig 3). Clinic-wide education to standardize the use of the electronic order began in February 2015. Improvements in use of the electronic system were seen after the initiation of reminder e-mails sent to providers who missed opportunities to use the electronic order. Sustained gains in use of the order were seen with ongoing provider education, such as neon slips on EI brochures reminding providers to submit an electronic order (May 2015), presentation of EI QI data at a clinic staff meeting (July 2015), and a resident preclinic conference focused on developmental screening (August 2015). As a result, referrals initiated through an electronic EI referral system rose from a median of 71% to 94%, allowing PNs to better capture and track the population needing to connect with EI. Clinic processes to promote universal developmental screening were in place before this intervention. In both the baseline and intervention phases, 95% of patients aged 6 months to 3 years were screened for developmental delay. With these high and stable screening rates, it is suggested that the EI referral process was not sufficiently burdensome as to dissuade practitioners from screening for delay.
Despite our redesigned system, there remained patients who did not receive an evaluation within 120 days of referral. The improvement team worked with PNs to characterize the reasons why evaluation had not occurred, identifying 4 main categories: (1) family refusal; (2) patient was lost to follow-up; (3) evaluation had not occurred, but the referral was still “pending” at the EI site; or (4) we were unable to obtain information from the EI site regarding referral outcome. Cases that were pending generally represented those in which 1 EI site had to refer to another EI site given lack of ability to accommodate a particular need or family language. In Fig 4, we show the outcome of referrals by month examined at the 120-day mark. The percent of patients refusing evaluation after the initial referral decreased after the rollout of our clinic-designed EI brochure in May 2015, which included space for providers to remind the patient why they had been referred.
Despite well-described barriers in connecting primary care patients to EI programs, our process change resulted in a median of 72% of patients referred receiving an evaluation within 120 days, which represented both substantial improvement from our baseline as well as a higher rate of connection than that seen elsewhere in the literature.1,19 We attributed success of the intervention to a few key system changes. First, the use of an EMR-based electronic referral system, even in the absence of electronic referrals directly to EI, helped to consolidate responsibility to the PN, ensuring that referrals were made correctly and allowing referrals to be tracked. This finding echoes researchers in earlier studies who showed that communicating referrals directly to EI, rather than asking families to call, resulted in higher rates of connection.15 Secondly, the creation and use of an EI registry allowed PNs to carefully track the patient population and helped to facilitate follow-up for patients in situations in which logistical barriers may have prevented completion of the evaluation. In earlier studies, researchers have shown that logistical barriers in communicating with EI are high enough that only “very motivated” families are able to overcome them.16 In this case, PNs seemed to help families overcome some logistical barriers to allow evaluation to occur, a finding endorsed in 1 previous study.20 Finally, the registry also served as a measurement tool, allowing our QI team and the PNs themselves to understand whether changes were working as hoped.
We identified several systemic barriers to success. For one, EI is a voluntary program by design. Parents’ concerns about involvement with the program (based on a variety of factors, including disagreement with the medical team’s perception of developmental delay, fear of stigmatization, or belief that EI workers are connected with child protective services) have been well documented in other studies.15,16 These factors may account partially for the differential rates of evaluation we saw among our publicly and privately insured patients. Our project included efforts to educate families on EI and prepare providers to address families’ potential concerns; we saw a corresponding decrease in refusals connected to this work. However, respecting family decisions and preference in this process is 1 reason that our goal was not for universal evaluation. We also recognize that many children who initially appeared to be developmentally delayed at the time of the referral may have made gains postreferral, thus contributing to the percentage of families who refused EI evaluation.
Challenges in information sharing with individual EI sites were a hurdle that our system redesign only partially addressed. Given that there is no centralized EI referral system in Massachusetts, our project required communication with each individual EI site. Additionally, we found EI intake processes varied substantially for different sites. Knowledge of the EI sites’ individual intake systems and relationships with key personnel at the sites were paramount in facilitating successful information flow. For example, some sites were easily accessible by telephone, whereas others preferred e-mail–based or online communications. Some sites had a central intake coordinator, whereas other sites shared this work among multiple people, meaning our staff had to communicate with up to 10 EI workers to learn about the status of referrals on our patients. When we could not work out a communication system, as was the case with 1 large agency in our system, we were unable to help intervene when patients were unable to connect.
It should be noted that the regulatory body for EI in Massachusetts does not require EI sites to report patients as having been referred until the agency has a conversation directly with the family to establish their interest. Although this policy is patient- and family-centered, it does not allow the state to track those families referred by primary care who do not connect with the EI program. Centralized EI referral systems for the state might facilitate both easier referrals for families and an opportunity for the state to monitor the volume of patients referred to EI who never connect to the program.
Our study had several limitations. First, our findings do not represent our entire patient population because some children who previously connected to EI through an inpatient or NICU referral were not included in this analysis. Secondly, because this is a QI study, system changes we found to be associated with improved outcomes are not necessarily generalizable to sites with different staff or patient characteristics. For example, our intervention relied on PNs, which many clinics may not employ. However, the nature of the intervention that emerged was focused on referral tracking and follow-up, suggesting that this type of intervention could be done in other practices by administrative staff. Finally, we did not directly measure provider or family experience with these system changes.
With improved developmental screening procedures enhancing recognition of delays, effective coordination of EI referrals is an essential function of pediatric primary care. Systems built to improve EI referrals may act as a template for other high-risk community-based referrals, such as those for mental health services. Further study is needed to understand how the techniques used here (using local referral “experts,” creating registries to track referrals, and building strong relationships with community providers) can be of assistance for other essential community-based referrals.
- Accepted April 2, 2018.
- Address correspondence to Kathleen Conroy, MD, MS, Division of General Pediatrics, Department of Medicine, Boston Children’s Hospital, Hunnewell Ground, 300 Longwood Ave, Boston, MA 02115. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Funded by the Academic Innovations Collaborative, Center for Primary Care, Harvard Medical School.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Guevara JP,
- Gerdes M,
- Localio R, et al
- King TM,
- Tandon SD,
- Macias MM, et al
- Earls MF,
- Hay SS
- Schonwald A,
- Huntington N,
- Chan E,
- Risko W,
- Bridgemohan C
- Hix-Small H,
- Marks K,
- Squires J,
- Nickel R
- Duby JC; American Academy of Pediatrics Council on Children With Disabilities
- Bailey DB Jr,
- Hebbeler K,
- Spiker D,
- Scarborough A,
- Mallik S,
- Nelson L
- National Research Council
- Institute of Medicine
- Shonkoff JP,
- Hauser-Cram P
- Karoly LA,
- Greenwood PW,
- Everingham SS, et al
- Rosenberg SA,
- Robinson CC,
- Shaw EF,
- Ellison MC
- Magnusson DM,
- Minkovitz CS,
- Kuhlthau KA,
- Caballero TM,
- Mistry KB
- Houtrow AJ,
- Larson K,
- Olson LM,
- Newacheck PW,
- Halfon N
- Provost L,
- Murray S
- Copyright © 2018 by the American Academy of Pediatrics