A Quality Improvement Collaborative to Improve Pediatric Primary Care Genetic Services
OBJECTIVE: To investigate if a national pediatric primary care quality improvement collaborative (QIC) could improve and sustain adherence with process measures related to diagnosis and management of children with genetic disorders.
METHODS: Thirteen practices in 11 states from the American Academy of Pediatrics’ Quality Improvement Innovation Networks participated in a 6-month QIC that included regular educational opportunities, access to genetic professionals, and performance feedback. The QIC identified 11 aims related to improving diagnosis and management of children with genetic disorders. The practices evaluated adherence by reviewing patient records at baseline, monthly for 6 months (active improvement period), and then once 6 months after the QIC’s conclusion to check for sustainability. Random intercept binomial regression models with practice level random intercepts were used to compare adherence over time for each aim.
RESULTS: During the active improvement period, statistically significant improvements in adherence were observed for 4 of the 7 aims achieving minimal data submission levels. For example, adherence improved for family histories created/maintained at health supervision visits documenting all components of the family history (6% vs 60%, P < .001), and for patients with specific genetic disorders who received recommended care (58% vs 85%, P < .001). All 4 of these aims also demonstrated statistically significant improvements during the sustainability period.
CONCLUSIONS: A national QIC reveals promise in improving and sustaining adherence with process measures related to the diagnosis and management of genetic disorders. Future research should focus on patient outcome measures and the optimal number of aims to pursue in QICs.
- AAP —
- American Academy of Pediatrics
- ICD-9 —
- International Classification of Diseases, Ninth Revision
- QIC —
- quality improvement collaborative
- QuIIN —
- Quality Improvement Innovation Networks
- RC —
- Genetics and Newborn Screening Regional Collaboratives
Primary care pediatricians are often responsible for identifying, referring patients to specialists for definitive diagnosis, and then managing the health care of patients with genetic disorders.1 Although early diagnosis and intervention, and continued health supervision remain crucial for improved outcomes in patients with genetic disorders,2–5 many pediatricians express uncertainty regarding their ability to care for this complex and heterogeneous patient population.1,6 Only 49% of pediatricians agree that they feel competent in providing health care to patients related to genetics.7 Practitioners identify a number of challenges in diagnosing and caring for pediatric patients with genetic disorders,4,5,7–9 including inadequate time, education, and genetic-focused resources.10–12 Additionally, many pediatricians report the inconsistent application of best-practice recommendations for a cornerstone of genetic disease assessment: creating family histories.7,12–15 Finally, although many genetic conditions have specific health supervision guidelines,16–24 it is unclear how often these guidelines are followed by pediatricians or how best to ensure guideline implementation.
In recent years, quality improvement collaboratives (QICs) have revealed success in improving adherence to guidelines and patient outcomes.25–30 A QIC conducted by the American Academy of Pediatrics’ (AAP) Quality Improvement Innovation Networks (QuIIN) increased the number of pediatric practices with at least 70% of newborn screens documented and communicated to families from 27% to 67%.28 Other, noncollaborative based quality improvement studies, many of which were conducted in adults, suggest computerized family history tools can improve family history creation,31–34 and standardizing family history practices leads to improved identification of patients at increased risk of serious diseases.32,33,35 It remains unclear if quality improvement methodology, and QICs in particular, can improve both diagnosis and ongoing care management of patients with genetic disorders in primary care settings.
We hypothesized that a national pediatric primary care QIC could improve adherence with process measures for diagnosis and care management of children with genetic disorders, and sustain these changes after the QIC’s conclusion.
This quality improvement initiative was conducted from March 2013 through March 2014 with the AAP’s QuIIN, a national group of practicing pediatricians interested in quality improvement. QuIIN is made up of over 300 ambulatory care practices in 46 states, ranging from single practitioner private practices to large academic institutions with over 80 physicians. Participation in QuIIN projects is voluntary and projects typically involve 10 to 15 participating practices with 3 to 5 quality improvement aims. A survey of QuIIN practices was conducted as a needs-assessment before the project, suggesting appreciable variation in care provided to children with genetic disorders.7 Requests for participation in this project were solicited from the QuIIN list serve, and practices were required to complete an application on baseline practice characteristics. All practices that completed the application were invited to participate (N = 13). Participation allowed practitioners to obtain both continuing medical education credit and American Board of Pediatrics Maintenance of Certification Part IV credit.
Figure 1 summarizes the intervention timeline. First, an expert group of geneticists, primary care practitioners, quality improvement specialists, QuIIN staff, and health services researchers met for almost 1 year before the first QIC learning session to develop quality improvement aims, measures, and tools for practices. The group met twice face to face and held monthly conference calls during this period. Based on expert consensus, data in the literature and the needs assessment described above,7 the QIC expert group focused on improving 2 domains of genetic care for children: (1) diagnosis and (2) management. As QuIIN previously conducted a newborn screening QIC,28 this was not a focus of this project. The 11 aims for this project are described in Table 1.
Each practice was required to identify a core improvement team, led by a physician and including at least 2 other practice members, who could be physicians, nurses, or office staff. Practices collected baseline data for the QIC aims and completed preassessments of the current state of their practice in caring for children with genetic disorders. In March 2013, core improvement teams participated in a 2-day, face-to-face learning session led by the expert group. This learning session covered the rationale behind this project, quality improvement methodology, genetic knowledge for practitioners, collaborative team sharing of current successes and challenges, tools to improve processes of care, family history evaluation practice sessions, and time for teams to develop local tests of change and 60- to 90-day aim statements. Additionally, the QIC partnered with the Genetics and Newborn Screening Regional Collaboratives (RCs) to increase partnership between practices and genetic professionals at the community level. Each RC identified a local genetic professional (geneticist or genetic counselor) to support and mentor each of the participating practices around the provision of genetic services.
During the 6-month active improvement phase, teams participated in monthly QIC webinars, phone or e-mail contact with their RC genetic mentor, data collection, and data feedback on how the practice was performing. Teams could also access an e-mail list serve to ask questions or solicit best practices. Teams then participated in a second face-to-face learning session on similar topics as the first learning session, as well as on sustainability and the project’s overall impact on patients across practices. After the second learning session, teams were no longer exposed to QIC interventions, although the list serve remained open but with appreciably less activity. In March 2014, 6 months after the second learning session, teams were asked to submit a final round of sustainability data.
This project was approved by the AAP’s institutional review board.
Data were collected through the AAP’s Quality Improvement Data Aggregator, a web-based data collection and analysis program for QICs, and SurveyMonkey. Both electronic and paper record review methodology were used to collect local data, based on the functionality of each practice’s electronic health record. For the 4 aims related to the diagnosis of children with genetic disorders, adherence was measured by the presence of specific items within patient records, both paper-based and electronic, such as the use of all predefined components of a multigenerational family history. Practices were instructed to review the first 10 patient records meeting inclusion criteria in a given month and no limitations were put on the number of records per provider or per clinic location reviewed. Practices were able to enter more or less than 10 records monthly. For some aims, inclusion criteria for the aim were dependent on the record successfully meeting the previous aim. For example, to be eligible for the aim, “Current family histories are discussed with patient/family, both positives and negatives,” the record had to achieve “Family histories are created/maintained at health supervision visits documenting all components” at that visit.
For the 7 aims related to the management of children with genetic disorders, the expert group created a list of International Classification of Diseases, Ninth Revision (ICD-9) diagnosis codes that would help identify patients with probable genetic disorders to be entered into a practice registry, and also provided a shorter list of genetic disorders that have existing disorder-specific health supervision guidelines or require an emergency plan (Supplemental Information). These lists were only used to define the inclusion criteria for the 7 aims related to the management of children with genetic disorders. Practices were encouraged to use these complete lists but were able to create local versions more pertinent to their clinic setting. Once created, practices used these registry lists to identify patients with genetic disorders and then reported on the number of these patients who met QIC aims monthly, regardless if they were seen in the practice that month. Practices were asked to self-identify, in conjunction with RC input, patients who required palliative care discussions. Given the challenges practices faced in working on 11 aims simultaneously, they were surveyed at the conclusion of the active improvement period regarding how hard they worked to improve each of the 11 aims (“a lot,” “a little,” or “did not work on”).
Finally, in an effort to expand quantitative findings and gain deeper insights into each practice’s learning and implementation process, a qualitative component was included in the study to reflect a mixed-methods design. Researchers conducted semistructured exit interviews of all participating practices in the spring of 2014 by using an interview protocol that targeted questions on changes made and sustained, characteristics leading to success, barriers to change, and ability to spread improvements. These interviews were conducted by 2 trained qualitative researchers via telephone, lasted ∼45 minutes each, notes were typed in real time, and participants were given the discussion guide in advance.
The unit of analysis was the patient visit for aims related to the diagnosis of children with genetic disorders, and the unit of analysis was the patient for aims related to management of children with genetic disorders. Throughout the article, a patient’s “record” refers to paper and electronic documentation at a specific visit for the diagnosis aims, and all paper and electronic documentation for the management aims. Due to concerns that analyzing results from aims with minimal data submission would lead to inaccurate conclusions, we did not analyze aims that in aggregate had a median of less than 5 records submitted per practice per month (N = 4). This was a posthoc decision made during data analysis. Adherence to the aim was expressed as an indicator that the patient record achieved the aim. For each practice and month, we aggregated the patient visit or patient data and calculated the number of patient records achieving the aim (binomial numerator) out of the total number of patient records reviewed (binomial denominator). Random intercept binomial regression models where then used to model the monthly adherence over time. Defining our outcome in this way accounted for variation over time and across practices in the number of records reviewed. We hypothesized a priori that percent adherence with each aim would increase over time during the active improvement period and would potentially continue to improve after the conclusion of the active improvement period; this was accounted for in the model by including a linear spline with a knot at the last month of the active period. Models took into account the potential correlations of patient measures within practices by including practice level random intercepts. From the fit of the model, monthly adherence was estimated and Wald’s tests were used to compare adherence at baseline to adherence at the end of the active improvement period (6 months) to test for effectiveness of the QIC. Adherence at the end of the active improvement period was also compared with adherence at the end of the sustainability period (12 months) to test for sustainability. For the 5 aims related to management, we acknowledge that these data denominators use nonindependent lists of patients from month to month. The analysis accounts for clustering among patient visits at the same practice, but does not account for the potential that additional correlation may exist for patients with multiple visits. A sensitivity analysis was performed for each model including a variable for how “hard” a practice worked on a given aim. Missing data were assumed to be missing at random and our methods are valid under this assumption. Qualitative data were analyzed by using thematic analysis. Themes and patterns from each interview were extracted into main themes for a given topic across all practice responses. Particular attention was paid to affordances and constraints within the change process and extrapolations for other quality improvement efforts. No additional qualitative software was employed.
Twelve pediatric practices and 1 family medicine practice in 11 states serving ∼130 000 pediatric patients annually participated in the QIC. Practice demographics are presented in Table 2. The QIC had representation from large and small practices, urban and suburban practices, and practices with and without previous quality improvement knowledge.
There was appreciable variability in the number of records reviewed for adherence and patients included in registries per practice per month (Table 1). Four aims were excluded from analysis because the median number of records reviewed or patients included in registries per practice per month were less than 5 (these analyses are available upon request). For the 2 analyzed aims related to the diagnosis of children with genetic disorders, median records reviewed ranged from 9 to 10 per practice per month. For the 5 analyzed aims related to the management of children with genetic disorders, median patients in practice registries ranged from 16 to 94.5 per practice per month (Table 1). The median number of patients in practice registries is particularly variable because not all patients with genetic disorders had health supervision guidelines or required transition of care discussions. Three teams did not submit sustainability data.
Adherence with 4 of the 7 aims had significant improvements comparing baseline to the end of the active improvement period using the fit of the random intercept binomial regression models: “family histories are created/maintained at health supervision visits documenting all components” (6% vs 60%, P < .001), “patients with genetic disorders have up to date age-appropriate health supervision visits” (70% vs 77%, P < .001), “patients with specific genetic disorders that have existing disorder-specific health supervision guidelines16–24 receive the specified care” (58% vs 85%, P < .001), and “patients with genetic disorders have next steps of care and planned follow-up” (14% vs 47%, P < .001; Table 3). All of these aims also demonstrated statistically significant improvements in adherence using the fit of the random intercept binomial regression models comparing the end of the active improvement period to the end of the sustainability period, suggesting improvements were sustained and even increased. Numerical improvements in adherence for 2 other aims did not reach statistical significance comparing baseline to the end of the active improvement period. Contrary to our hypothesis, 1 aim demonstrated a statistically significant reduction in adherence with the aim comparing baseline to the end of the active improvement period: “patients with genetic disorders are offered genetic services at least initially” (38% vs 18%, P < .001).
The 2 aims with the highest percent of practices reporting they worked a lot on the aim were family histories are created/maintained at health supervision visits documenting all components, (85% worked on a lot) and patients with genetic disorders have up to date age-appropriate health supervision visits (62% worked on a lot). On the contrary, the 2 aims with the highest percent of practices reporting they did not work on the aims were “patients with genetic disorders who require an emergency plan to prevent catastrophic illness have a plan requested from the specialist and placed in the chart, at least annually” (46% did not work on), and “patients with genetic disorders have a discussion about palliative care at least annually when appropriate” (46% did not work on). Both of these aims had less than 5 median records submitted per practice per month. Including the variable for how hard a practice worked on a given aim, did not appreciably change the effect size or statistical significance for any aim.
Qualitative exit interviews of participating practices suggested 4 key learning points: (1) practice commitment to change, buy-in from staff, and having a champion for change were the most noted characteristics contributing to the success of project implementation, and when 1 of these facets were missing, change was more difficult to implement; (2) although practices noted barriers to change and that the project required a great deal of work, most indicated they would encourage other practices to pursue these changes; (3) participation helped practices “think genetically,” which increased the identification of patients with genetic disorders and improved the quality of patient care; and (4) electronic health record systems were limited in their ability to incorporate family history information.
This national QIC in 13 pediatric and family medicine primary care practices demonstrated statistically significant improvement in 4 of 7 process aims related to the diagnosis and management of children with genetic disorders. Moreover, all 4 of these aims revealed sustained improvement 6 months after the conclusion of the QIC. Primary care practices, ranging from rural, private-practice single practitioner sites to urban, university affiliated multipractitioner tertiary care clinics, were able to share, collaborate, and work to improve genetic-related care for their patients and families.
Pediatricians are tasked with diagnosing and managing an increasing array of patients during shorter appointment times. QIC projects can help busy ambulatory practices deliver best-practice care for all patients and speed adoption of health supervision guidelines.28 By identifying gaps in current patient care processes,7 and then using “all teach, all learn” methodologies, this QIC improved the care delivered to patients. A toolkit derived from this collaborative learning process is available for download.36 Interested providers should examine this toolkit that provides step-by-step instructions for achieving these aims and lessons learned from this QIC. These data suggest any primary care practice, regardless of setting, patient demographics, or patient volume, can augment existing genetic health care services. Also, these data suggest that genetic care may not be different from other pediatric care processes, which can be improved with local, context-specific resources, sharing challenges and successes with like-minded practices, and QIC methodology.
It is unclear why some aims were more successfully implemented than others during this QIC, with 1 aim demonstrating a decreased percent of records achieving the aim. It is possible that these aims were more difficult to implement, not as much of a priority for practices, or more challenging to measure. The authors also hypothesize that simultaneously presenting 11 aims for improving care, although comprehensive, posed too demanding a task for practices over the 6-month time period allotted. It is unclear why adherence with the aim of patients with genetic disorders are offered genetic services at least initially decreased from baseline to 6 months (38%–18%, P < .001) but statistically significantly improved from 6 months to 12 months (18%–36%, P < .001). We hypothesize that practices’ increased focus on providing other genetic services reduced their focus on the offering of genetic services or its documentation, leading to an initial reduction with eventual rebound when practices were more familiar with other genetic services. Based on exit interview results, the authors note similarities between reported markers of success in this project and previous models of quality improvement success.37 Additionally, we suggest effort is needed to improve electronic health record family history taking, with a focus on family participation in building the family history and the tools used to collect it.
Over the past decade, the traditional notion of genetic disorders as rare entities has undergone a significant shift toward the realization that in the aggregate genetic conditions are common. Although individually rare, a growing number of more than 4200 disorders are known to be caused by alterations of single genes.38 Recent evidence suggests that individuals in the general population may carry an average of 3 recessive mutations for childhood disease.39 Approximately 2% to 3% of newborns are affected by major congenital anomalies that collectively account for ∼20% of deaths in infancy, another 5% to 6% of the population will be diagnosed with a genetic condition by early adulthood, and up to one third of the population will develop a condition with a genetic component by age 60.40 It is in this context that pediatricians are faced with the challenging responsibility of integrating rapidly expanding resources for genetic medicine into diagnosis and care management. For practitioners to take part in the evolving translation of genomics into clinical medicine, core competencies need to be developed and implemented into a practice flow. This QIC was an attempt to develop a guiding framework for process changes aimed at improving the care of patients with genetic disorders in the setting of busy primary care practices. The 2 domains of improvement focused on during the project, diagnosis and management, reflect previously proposed genomic competencies for primary care providers,41 including evaluation and diagnosis, genetic testing and risk communication, and management and coordination of care. These data could help to prioritize benchmarks for quality care delivered to children with these disorders.
We acknowledge limitations in this research design. First, data were collected and submitted by practices without systematic, centralized data checking for appropriate coding. We created multiple opportunities for practices to learn and re-learn about data collection and QIC measure definitions at learning sessions and webinars. Although obvious errors in rates or numbers entered were brought to practices’ attention, we cannot speak comprehensively to data accuracy. Similarly, for aims related to the management of children with genetic disorders, practices were allowed to create their own lists of patients included in a registry and it is unclear how consistent these denominator definitions were applied across sites. Related to this, some measures had few practices reporting patients meeting criteria for inclusion in the denominator, leading to process measures with variable numbers of data points and the exclusion of 4 aims with less than 5 median records submitted per practice per month. Questions to practices on how hard they worked on each aim were not validated and may not reflect the quality of care received by patients. This is exemplified by 1 practice’s qualitative response that, “The items we did not work on were either already in place or not relevant to our patients.” This could explain why this variable did not appreciably change model effect sizes or statistical significance in the sensitivity analyses. Three practices did not submit sustainability data at the 12-month time point. It is unclear if data from these practices would have decreased the number of aims demonstrating sustainability. The practices included in this project were self-selected by interest in the topic and interest in quality improvement in general. The results obtained therefore, may be more challenging to generalize to practices without genetic and/or quality improvement interest. Despite this, we believe having motivated, “innovator” practices test, develop and demonstrate what is possible in quality improvement, allows other pediatric practices to more quickly adopt and adapt important quality improvement interventions. Six of the 13 practices reported they were “somewhat” or “not knowledgeable” about the Model for Improvement, suggesting that previous quality improvement experience may not be a prerequisite to complete similar projects. Finally, observed, nonaggregated data from practices demonstrate continued local variability in process measures, suggesting practices could still improve and these data may not represent a true benchmark for what is possible.
A national QIC reveals promise in improving process measures related to the diagnosis and management of children with genetic disorders in primary care settings. Future research should focus on sustainability of interventions past 12 months, patient outcome measures, the applicability of lessons learned through dissemination of the QICs genetic toolbox,36 and the optimal number of aims to simultaneously pursue in QICs.
The authors gratefully acknowledge the effort of the 13 practices involved in this QIC, who worked tirelessly to improve care for their patients: All About Children Pediatrics, ALL Pediatrics, Children’s Hospital of Pittsburgh Primary Care Center, CMC Myers Park Pediatrics, Fair Oaks Health Center Pediatrics, Iron Horse Pediatrics, Nassim and Associates, Northampton Area Pediatrics, Pediatric and Adolescent Health Partners, Prattville Pediatrics, University Pediatric Clinic, Vanguard Medical Group, and YouthCare Pediatrics. We also acknowledge Meggen Kaufmann and Caprice A. Knapp, PhD for their contributions with the qualitative analysis.
- Accepted August 17, 2015.
- Address correspondence to Michael L. Rinke, MD, PhD, Department of Pediatrics, Children’s Hospital at Montefiore, 3415 Bainbridge Ave, Rosenthal 1, Bronx, NY 10467. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Support for the quality improvement collaborative discussed in this article was provided by the Genetics in Primary Care Institute (GPCI), a cooperative agreement between the American Academy of Pediatrics and the US Department of Health and Human Services, the Health Resources & Services Administration, the Maternal & Child Health Bureau (federal award number UC7MC21713). Dr Tarini was supported, in part, by a K23 Mentored Patient-Oriented Research Career Development Award from the National Institute for Child Health and Human Development (K23HD057994).
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Kemper AR,
- Uren RL,
- Moseley KL,
- Clark SJ
- Silversides A
- Department of Health
- Committee on Genetics
- Trotter TL,
- Martin HM
- Hersh JH; American Academy of Pediatrics Committee on Genetics
- Trotter TL,
- Hall JG; American Academy of Pediatrics Committee on Genetics
- Bull MJ; Committee on Genetics
- Frías JL,
- Davenport ML; Committee on Genetics and Section on Endocrinology
- Committee on Genetics
- McCandless SE; Committee on Genetics
- American Academy of Pediatrics Committee on Genetics
- Hersh JH,
- Saul RA; Committee on Genetics
- Section on Hematology/Oncology Committee on Genetics
- American Academy of Pediatrics
- Miller MR,
- Griswold M,
- Harris JM II, et al
- Miller MR,
- Niedner MF,
- Huskins WC, et al; National Association of Children’s Hospitals and Related Institutions Pediatric Intensive Care Unit Central Line-Associated Bloodstream Infection Quality Transformation Teams
- Hinton CF,
- Neuspiel DR,
- Gubernick RS, et al
- Hulscher ME,
- Schouten LM,
- Grol RP,
- Buchan H
- Schouten LM,
- Hulscher ME,
- van Everdingen JJ,
- Huijsman R,
- Grol RP
- Acheson LS,
- Zyzanski SJ,
- Stange KC,
- Deptowicz A,
- Wiesner GL
- Qureshi N,
- Armstrong S,
- Dhiman P, et al; ADDFAM (Added Value of Family History in CVD Risk Assessment) Study Group
- American Academy of Pediatrics Quality Improvement Innovation Network, Genetics in Primary Care Institute
- Kaplan HC,
- Provost LP,
- Froehle CM,
- Margolis PA
- ↵Online Mendelian Inheritance in Man OM-NIoGM. Available at: http://omim.org/. Published 2012. Accessed November 14, 2014
- Bell CJ,
- Dinwiddie DL,
- Miller NA, et al
- Scott J,
- Trotter T
- Copyright © 2016 by the American Academy of Pediatrics