BACKGROUND AND OBJECTIVES: Previous studies have documented poor rates of delivery of preventive services, 1 of the core services provided in the primary care medical home setting. We aimed to increase the reliability of delivering a bundle of preventive services to patients 0 to 14 months of age from 58% of patient visits to 95% of visits. The bundle includes administration of routine vaccinations, offering influenza vaccination, completed lead screening, completed developmental screening tool, screening for maternal depression and food insecurity, and documentation of gestational age.
METHODS: The setting was 3 academic pediatric primary care clinics that serve 31 000 patients (>90% Medicaid). Quality improvement methodology was used and key driver diagram was determined. Patient “Ideal Visit Flow” and the Responsible, Accountable, Support, Consulted, and Informed Matrix were developed to drive accountability for components of the ideal flow. Plan, Do, Study, Act cycles were used to develop successful interventions. The percent of patients seen who received all bundle elements for which they were eligible was plotted weekly on a run chart, and statistical process control methods were used to determine a significant change in performance.
RESULTS: The preintervention percentage of patient visits ages 0 to 14 months receiving all preventive service bundle elements was 58%. The postintervention percentage is 92%.
CONCLUSIONS: Innovative redesign led to improvement in percentage of patients age 0 to 14 months who received the entire preventive services bundle. Key elements for success were multidisciplinary site-specific teams, redesigned visit flow, effective communication, and resources for data and project management.
- CCHMC —
- Cincinnati Children’s Hospital Medical Center
- EHR —
- electronic health record
- IT —
- information technology
- MA —
- medical assistant
- QI —
- quality improvement
- RASCI —
- Responsible, Accountable, Support, Consulted, and Informed
More than 10 years ago, the Institute of Medicine called attention to the suboptimal performance of the United States health care system, concluding: “Current care systems cannot do the job. Trying harder will not work. Changing care systems will.”1 In pediatrics, a 2007 publication revealed that patients receive only 41% of indicated preventive services.2 Past efforts to improve quality in pediatric primary care have focused mostly on individual services, such as improving immunization rates3 or lead screening.4–6 Yet, pediatric practices continue to struggle with comprehensively delivering preventive services in light of lengthy health maintenance guidelines and short visits.7–12 The challenges of delivering comprehensive well child care underscore the need for true system transformation rather than intense focus on a single measure. In other areas of medicine, “bundle measures” have been used to measure whether every patient receives every service every time.13–15
We identified 1 previous study of a bundle measure in pediatric primary care. This bundle included services related only to physical health (immunization, lead screening, anemia, and tuberculosis) and was implemented in nonacademic practices.16 This 2004 study demonstrated that it is possible to improve care across multiple measures through system improvement. However, changes in the scope of well child care in the past decade call for a more broadly defined measure that includes services beyond screening for physical illness and exposures.17
Locally, our medical center was redesigning primary care, with a global aim to improve health promotion, chronic disease outcomes, and patient and family experience, while decreasing cost of care at a population level. The intervention described in this article focuses on our goal to improve health promotion. The scope of this quality improvement (QI) study was to improve delivery of preventive services, including assessment of development and family well-being, during visits at our 3 primary care centers in the first 14 months of life.
We aimed to measure and improve a “preventive service bundle” of 5 indicators of promoting an infant’s physical health, cognitive and social development, and family well-being. We used an all-or-none bundle measure to obtain a more accurate picture of the quality of care that individual patients received18 during office visits. We theorized that, by monitoring performance on a bundle measure and using the data to drive small tests of change, we could optimize performance during each stage of an office visit (eg, registration, intake, order entry). We could then clearly define roles, standardize processes, and transform our system to achieve highly reliable preventive service delivery.19 We aimed to improve the percentage of visits at which patients received all bundle elements for which they were eligible from 58% to 95% within 1 year.
Cincinnati Children’s Hospital Medical Center (CCHMC) has 3 primary care centers involved in primary care redesign. Sites 1 and 2 (urban) and site 3 (suburban) are the medical homes for #x223c 18 000, 7000, and 6000 patients, with #x223c 37 000, 14 000, and 15 000 visits, respectively, per year. The payer mix is uniform across the centers: 90% Medicaid, 3% private insurance, and 7% self-pay/uninsured. All 3 sites train large number of learners. All sites have been using Epic, an electronic health record (EHR), since May 2011. In 2012, CCHMC and the Division of General and Community Pediatrics invested $250 000 to conduct a large-scale primary care redesign that aimed to improve health promotion and prevention, outcomes for children with chronic diseases, patient and family experience, and decrease cost of care. This article focuses on 1 phase of the redesign and describes the QI tools and processes used to improve health promotion and prevention for infants in our patient population. This study was granted exemption by the CCHMC Institutional Review Board.
Study Population and Outcome
We sought to design a comprehensive, efficient, and effective preventive service delivery system. We focused on the first 14 months of life when opportunities are greatest for engaging families in primary care.20
The following services were selected for inclusion in the bundle of preventive services that all infants should receive during the first 14 months of life: administration of routine immunizations, seasonal influenza vaccination offer, lead screening, standardized developmental screening using Ages and Stages Questionnaire, and screening for bio-psychosocial risk factors including gestational age, parental depression, and food insecurity. These services were chosen by the primary care redesign steering committee, explained below, based on American Academy of Pediatrics recommendations,21 with minor modifications.
A team structure was developed to support all phases of the primary care redesign. Some of the team’s time focused on preventive services for infants. The team included the following: (1) a steering committee that provided overall guidance and consisted of a project sponsor, project leader, physicians, patient services manager, business director, QI consultant, and advisors; (2) a project manager that guided resources to achieve specific goals; (3) support teams that provided content expertise on family engagement, training, finance, resident education, QI data analysis, information technology (IT), and communication; and (4) site-based teams at each of 3 primary care clinics spearheaded the work while participating in a hands-on QI training program. Participating physicians did not have extra protected time. Nurses and medical assistants (MAs) received payment for the hours they spent in these planning and training activities. At a staff kick-off meeting, the project vision, mission, and goals were reviewed and feedback was collected. An office system inventory22 was used to provide a baseline assessment of our preventive services delivery system. Our redesign aligned with the National Committee for Quality Assurance/Patient Centered Medical Home requirements.23
Development of Key Driver Diagram and Interventions
Improvement work began in June 2012. Preventive service support and site-based teams met throughout the planning and implementation phases to develop a theory of change, plan tests of change, and review data. The Model for Improvement was used.24 This model is based on 3 fundamental questions: (1) What are we trying to accomplish? (2) How will we know that a change is an improvement? (3) What changes can we make that will result in improvement? The team organized its theory of change by using a key driver diagram25 (Fig 1). Interventions were tested by using Plan, Do, Study, Act cycles.24
Optimizing patient visit flow was expected to improve preventive service bundle delivery. The project manager, consultant, and site-based teams mapped the detailed existing patient visit flow for each clinic. Clinic staff roles and responsibilities were identified by using a Responsible, Accountable, Support, Consulted, and Informed (RASCI) Matrix. RASCI is an acronym derived from the 5 responsibilities most typically used: Responsible, Accountable, Support, Consulted, and Informed. This matrix describes the participation by various roles in completing tasks26 and allows for prioritization of work. The current process map and RASCI matrix were used to develop ideal patient visit flow with the goal of optimizing each care team member’s role to maximize individuals’ training and skills. Roles were redefined and reassigned, and Plan, Do, Study, Act cycles tested integrating reassignments into visit flow. In addition, the IT team optimized the EHR to display patients’ preventive service needs on the patient schedule. This information was referenced by providers and staff during relevant steps in the ideal flow.
Several key changes were made to visit flow. First, 2 types of “huddles” were implemented. The nurse manager huddled with nurses and MAs in the morning to discuss potential flow challenges that day. In addition, at the beginning of each morning, afternoon, and evening clinic session, each provider huddled with their assigned MA to review the preventive service needs of each scheduled patient. Second, registration staff preassembled standardized, age-specific packets that included developmental and social screening forms.27,28 These were distributed at registration for parents to complete before rooming. Third, much of the information-gathering and EHR documentation for well visits was shifted from providers to intake staff. This ensured screening questions were documented and allowed physicians to focus the visit on areas of concern. Fourth, intake staff preordered immunizations and screening tests for providers to review and sign. Finally, the patient discharge process was standardized and included a staff member checking that all appropriate preventive services had been delivered. Clinic start time was not changed, and the number of visit was not reduced during ideal flow implementation.
Methods of Evaluation
A measure of the reliability of daily preventive service delivery was created, with the visit as the unit of analysis. This measure calculated the percentage of 0- to 14-month-old visits during which the patient received all elements of the bundle for which they were eligible that day. More details on the measure are provided in a separate publication.29 Figure 2 summarizes expectations for preventive service delivery at each age. For example, any visit for a 9-month-old patient would require administration of any overdue immunizations, developmental screening, lead testing, and influenza vaccine offered if seasonally appropriate. Although ill visits were included in our study, if a provider decided to defer immunizations because of acute illness and documented this with an appropriate diagnosis code, this was not counted as a failure.
Performance data on bundle delivery were retrieved directly from clinical documentation in the EHR. Reports on this measure were constructed through an iterative process. An automated data report was initially formulated, then compared with manual chart reviews to ensure interrater reliability and to identify unique situations for which decision rules needed to be created. The automated report was then modified and the process was repeated. In its final form, the report was automated with quality checks done manually by a data manager who followed a list of decision rules.
Daily data were collated into weekly rates and plotted on a run chart. Standard probability-based rules were used for interpretation of run charts.30 In accordance with these rules, the median was recalculated when a “shift” of 8 consecutive points above or below the median was observed. Initially, chart audits were performed by the data manager for all failures, and appropriate timely feedback was provided to staff and providers. Weekly data were shared with all.
Throughout the QI process, qualitative data were collected from front-line staff regarding barriers to implementing interventions and impact of interventions on clinic flow, patient care, and resident education. QI consultants were on-site during implementation to shadow patients through visits and collect feedback that was used to improve processes. As a balancing measure, patient cycle time (time from patient sign-in to discharge) was collected pre- and postinterventions.
Over an 8-week baseline period (June through July 2012), the entire bundle of preventive services for which patients age 0 to 14 months were eligible was delivered at 58% of visits. Postintervention results revealed a median performance of 92%, achieved by May 2013 and sustained over 1 year (Fig 3).
Interventions fell into 2 major categories: structural and functional. Dates of intervention implementation are annotated on the run chart (Fig 3). Structural interventions included team development, shared vision, expert consultation, QI training, divisional support, and project and data management. Functional interventions included prioritization of work, optimizing the EHR, previsit planning, improving communication, and implementation of ideal visit flow. Interventions were temporally associated with shifts on the run chart (Fig 3). Cycle time remained the same pre and postinterventions (average 75 minutes).
Qualitative data collected from front-line staff indicated that early in our testing, a majority of failures were due to family refusal of preventive services or failure to document services that were completed or deferred due to acute patient illnesses. In addition, more failures happened on evenings or weekends, and when transient providers worked in our clinics. There was no difference in failures by age group. Failures were addressed by having QI consultants coach providers on proper documentation, observe and evaluate flow, and provide feedback and recommendations to optimize the ideal flow map. A separate in-depth study looked at the effects of primary care redesign on residents’ educational experience. It is presented in a separate publication currently under review. Overall, residents had improved perceptions of patient flow and physician/nonphysician teamwork.
Our bundle delivery rate was sustained at a median of 92% over a 1-year period. We planned for sustainability by developing a standardized orientation and training process for all newly hired staff and providers. We also performed frequent assessment and retraining of existing staff and providers to ensure the ideal flow protocol was being followed.
Between June 2012 and June 2013, we increased delivery of a preventive services bundle from 58% of visits to 92% of visits for patients 0 to 14 months of age, and we sustained the improvement for over 1 year. Interventions that coincided temporally with improvement in bundle delivery included prioritization of work, previsit planning, preclinic huddles, and implementation of ideal flow. Through improvement on the bundle measure, we now reliably address infants’ physical health, cognitive and social development, and family well-being during primary care visits. We have also transformed our system to create effective mechanisms for communication among staff members and more failsafe processes for identifying patients’ health maintenance needs.
Our study was strengthened by rigorous use of QI methods, with on-site QI consultants who ensured fidelity to the interventions. We maximized the sustainability of our intervention by studying our system under various conditions, including during evening and weekend hours and clinic sessions staffed by transient providers (moonlighters and learners). By collecting performance data directly from the EHR, we were able to include all visits in our data. Because we audited charts and coached providers on proper documentation, we are confident that our data closely reflect actual preventive service delivery.
Our findings support those of other studies in which system-based improvements resulted in improved preventive service delivery. Shaw et al6 and Young et al31 showed that practices who set goals and used QI methods achieved improvement in at least 1 preventive service. In accordance with recommendations by Solberg et al,32 we used strategies that considered multiple characteristics of participating clinics. Our findings were consistent with those of Bordley et al,5 who achieved improvement in immunization rates, anemia, and lead screening through implementation of previsit planning, risk assessment forms, provider prompts, and redistribution of responsibilities among office staff. Given findings by Shojania et al33 that point-of-care EHR reminders alone achieved small effects, and Lanham et al34 that relationships and respect among office staff contribute to primary care quality and success of improvement initiatives, our initial structural interventions (eg, formation of site-based teams with nonphysician team leaders) likely played a key role in laying the groundwork for our success.
To our knowledge, ours was the second study to measure delivery of a “bundle” of preventive services in pediatric primary care. Margolis et al16 measured a bundle of 4 preventive services (immunizations, and screening for tuberculosis, anemia, and lead) at 18 nonacademic practices and achieved improvement from 7% to 34%. Because Margolis et al16 used a patient-level measure rather than a visit-level measure, we cannot directly compare our success. However, our findings build upon the work by Margolis et al16 by expanding the bundle to include services related to developmental and social well-being and also demonstrate that it is possible to achieve significant sustainable improvement in large academic-affiliated pediatric practices. Expansion to this setting is important in ensuring high quality care for low-income children, given the large percentage of publicly insured patients who are served by academic-affiliated safety net hospitals.35
Our study had some limitations. Because we did not consider decisions to defer immunizations because of illness a failure to deliver indicated services, our data do not capture potential missed opportunities to vaccinate in situations where the patient is ill but immunization deferral is not clinically indicated.36 Additionally, because we are no longer performing chart audits on all failures, it is possible that some preventive services are being delivered but not documented; therefore, our actual performance may be better than what is shown.
In considering whether our findings are generalizable to other practices, we acknowledge context-specific factors that may have contributed to our success including the financial support for external consultation, data analysis, a project manager, QI consultants, and QI methodology training for front-line staff. We believe these resources were integral to our success given the size and complexity of our settings. However, because the fund was not entirely used for this particular phase of the redesign and the support staff had multiple other job responsibilities, it is difficult to quantify the exact cost of our preventive service improvement. It is notable that our improvements were achieved without hiring extra staff or reducing patient volume. We believe that small and less complex practices can learn from our experience to implement some changes with minimal project management resources. Universal strategies from our work that may be applied without a lot of further testing include the following: (1) developing EHR-based prompts about services due; (2) creating standardized packets for each patient age with appropriate developmental and social screening forms,27,28 which can be distributed by registration staff and completed by parents during waiting time; (3) shifting some history-taking and EHR documentation to intake staff to maximize the time and skills of all clinical team members. Also, new health care models with per-member per-month payment systems may soon incentivize investment in personnel to facilitate improvement work and quality measurement because funding these activities may provide returns on investment by improving outcomes for patients for which the health system is accountable.
Reaching our original goal of 95% would require higher reliability interventions, such as increased automation of tasks.32 We will continue to work closely with our IT team to implement these types of changes. Future directions include improving preventive service delivery on a population level and measuring preventive service bundles for other patient age groups.
Using QI methods, we achieved and sustained a preventive service bundle delivery rate of 92% of primary care visits for patients 0 to 14 months of age. The bundle included administration of all routine immunizations, offering influenza vaccine to eligible infants, and screening for lead, developmental delay, gestational age, food insecurity, and parental depression at appropriate intervals. Our bundle is the first, to our knowledge, to include preventive services in multiple domains of child and family well-being. Our study demonstrates the dramatic increase in reliability that is possible with system transformation, even in complex settings serving low-income populations.
- Accepted July 13, 2015.
- Address correspondence to Zeina Marcho Samaan, MD, Cincinnati Children’s Hospital Medical Center, ML 2011, 3333 Burnet Ave, Cincinnati, OH 45229. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: This study was conducted with internal funding from the Cincinnati Children’s Hospital Medical Center and the Division of General and Community Pediatrics.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Crossing the Quality Chasm
- Fiks AG,
- Grundmeier RW,
- Biggs LM,
- Localio AR,
- Alessandrini EA
- Shaw JS,
- Wasserman RC,
- Barry S, et al
- Belamarich PF,
- Gandica R,
- Stein RE,
- Racine AD
- Margolis PA,
- Lannon CM,
- Stuart JM,
- Fried BJ,
- Keyes-Elstein L,
- Moore DE Jr
- Committee on Psychosocial Aspects of Child and Family Health
- ↵Bright Futures/American Academy of Pediatrics. Recommendations for preventive pediatric healthcare. Available at: http://brightfutures.aap.org/pdfs/AAP%20Bright%20Futures%20Periodicity%20Sched%20101107.pdf. Accessed July 23, 2015
- ↵Commonwealth Fund. Office Systems Inventory. Available at: www.commonwealthfund.org/usr_doc/office_systems_inventory.pdf. Accessed July 23, 2015
- The National Committee for Quality Assurance-
- Langley G,
- Nolan K,
- Norman C,
- Provost L,
- Nolan T
- Project Management Institute.
- Beck AF,
- Klein MD,
- Kahn RS
- Squires J,
- Bricker D,
- Potter L
- Perla RJ,
- Provost LP,
- Murray SK
- Young PC,
- Glade GB,
- Stoddard GJ,
- Norlin C
- Shojania KG,
- Jennings A,
- Mayhew A,
- Ramsay C,
- Eccles M,
- Grimshaw J
- Cunningham P,
- May J
- Copyright © 2016 by the American Academy of Pediatrics