Quality Improvement Measures in Pulse-Oximetry Newborn Heart Screening: A Time Series Analysis
- Bethann Mangel Pflugeisen, MS, MEda,
- Paul J. Amoroso, MDa,
- Diane Zook, BSb,
- Karl F. Welke, MDc,
- Anne Reedy, MBAa, and
- Matthew V. Park, MDd
- aMultiCare Institute for Research and Innovation, Tacoma, Washington;
- bMary Bridge Children’s Hospital and Health Center, Tacoma, Washington;
- cChildren’s Hospital of Illinois and the University of Illinois College of Medicine, Peoria, Illinois; and
- dNorthWest Congenital Heart Care, Pediatrix Cardiology, Tacoma, Washington
BACKGROUND AND OBJECTIVES: The use of pulse-oximetry screening to detect critical congenital heart defects in newborns has gained national and international momentum in the past decade. Our hospital system began screening in 2008. Since then, our program has undergone leadership changes and multiple quality improvement interventions. The aims of this study are to evaluate the evolution of our pulse-oximetry program and to provide insights from lessons learned over the course of a long-standing program.
METHODS: We reviewed 6 years of screening data and evaluated trends of missed screens, false-positives, protocol violations, and parental decline of screening. We implemented 3 quality improvement interventions (change in protocol, redesign of an electronic medical record documentation system to autocalculate results, and transition from research to standard-of-care) and reviewed the impact of a rigorous quality assurance review process. We used linear regression and statistical process control charts to evaluate the data.
RESULTS: A total of 18 363 newborns were screened; we identified 5 critical cases. We observed a significant decrease in missed (P < .001) and false-positive (P = .03) screens over time but found no significant trend in the rate of percentage of protocol violations (P = .26) or decline of screening (P = .99). Each metric showed behavior attributable to at least 1 quality improvement intervention.
CONCLUSIONS: We established a sustainable pulse-oximetry screening program in our community hospital system, and the screening has now become routine. The quality of our screening was influenced by choice of screening protocol, rigor of quality assurance reviews, and the process used to interpret screening results.
- quality improvement
- critical congenital heart disease
The use of pulse-oximetry screening to detect cyanotic critical congenital heart defect (CCHD) in asymptomatic infants first appeared in the medical literature in the early 2000s.1–3 Since then, the technique has been widely studied3–14 and momentum has built worldwide to provide screening for eligible newborns.15,16 It is estimated that 29.5% of newborns with an asymptomatic, screening-detectable CCHD receive a diagnosis >3 days after birth, after the onset of symptoms or death, and that many of these infants could benefit from routine screening before discharge from their birth hospital.17 Infants with undiagnosed CCHD are at increased risk of morbidity and mortality, which can be mitigated or avoided if the condition is detected and managed early in the child’s life.5 CCHD screening with the use of pulse-oximetry is estimated to cost ∼$14 per newborn, with screening for 1 infant taking an average of 9 (±3.4) minutes and posing minimal burden to staff.18
In January 2011, a national workgroup convened to develop strategies for the implementation of safe and efficient pulse-oximetry screening.19 Later that year, CCHD screening was added to the Recommended Uniform Screen Panel,20 and in early 2012 the protocol developed by the workgroup was endorsed by the American Academy of Pediatrics (AAP).21 Despite these efforts, screening protocols currently in use vary. Some of this variation may be related to the rapid implementation of screening programs over the past 5 to 10 years. A recent report from the Centers for Disease Control and Prevention indicated that only one-third of screening hospitals in Georgia follow the AAP protocol.22 Although New Jersey and Tennessee have mandated screening, neither uses the AAP protocol.23,24 The Centers for Disease Control and Prevention recently launched an initiative to evaluate data from screening programs across the country with the goal of using computer simulation to create a new, data-driven algorithm.25
The MultiCare Health System’s (MHS) Tacoma General Hospital was the first hospital in Washington State to establish an institutional review board (IRB)–approved screening program. Screening to detect CCHD in newborns before hospital discharge and the onset of symptoms began in February 2008 as a research study to evaluate the effectiveness of pulse-oximetry screening in the early detection of CCHD and to prevent these healthy-appearing infants from being discharged from the hospital with an undiagnosed condition.
Members of our team served on the workgroup that developed the AAP-endorsed protocol. In May 2012 we restructured our study to align with the AAP protocol and expanded our research program to Good Samaritan Hospital in Puyallup, Washington. In April 2013, pulse-oximetry screening was adopted as standard-of-care and parents were no longer required to provide informed consent for screening; pulse-oximetry screening for wellborn neonates is now part of the routine discharge process from our birthing facilities.
During the 6 years of our program we screened >18 000 newborns (Fig 1). Three quality improvement interventions were enacted: a change in screening protocol, modification of an electronic medical record (EMR) documentation flowsheet to include autocalculation of results, and transition from research to standard-of-care. In addition, a rigorous quality assurance (QA) regimen was established. The program has proven to be sustainable throughout changes in primary leadership.
The primary aim of this analysis was to evaluate the evolution of our pulse-oximetry program with respect to the following metrics: missed screens, protocol violations, false-positives, and parental decline of screening. To accomplish this aim, we sought to answer 2 main questions: (1) did the incidence of each metric decrease over the lifetime of the screening program and (2) did relevant interventions impact the metrics in a meaningful way? A secondary aim was to provide insights from lessons learned over the course of our long-standing program.
Approval for the research program was granted by the MultiCare IRB in October 2007; screening began in February 2008. Parents of newborns underwent an informed consent process before opting to participate in the research study, which included pulse-oximetry screening for their infant and granting permission for the study team to review the child’s chart and/or contact the family for up to 1 year.
Four weeks before the commencement of the study, education and training were provided to ∼70 staff members who would perform the screening. The company that manufactures the pulse-oximeters used at our site provided in-service training as part of this mandatory training. Physicians from the pediatric heart center team provided education on congenital heart defects, transitional circulation, pulse-oximetry principles, the screening protocol, study documentation practices, pediatric echocardiography, and pediatric cardiology resources for follow-up in the event of a positive screen. Written competencies were completed by staff and assessed by unit educators. Clinical staff hired to the birth center after implementation of the screening program were trained on the protocol by existing staff.
During implementation, MHS and community physicians with privileges to see newborns in our hospital were invited to attend 1 of 3 educational seminars about the screening program and protocol. Subsequently, several Continuing Medical Education sessions, Grand Rounds, and other formal talks have been given by the principal investigator. Although it can be difficult to train busy physicians, we believe the effort to reach the providers was beneficial because the screening program has been well received and feedback from these providers about the education sessions has been overwhelmingly positive.
Study consent forms were created and translated from English to the community’s 6 most prevalent languages. The forms were reviewed with the staff who would consent patients, but extensive training on the informed consent process was not provided. Brochures were created that provided information about the prevalence of heart defects, symptoms of congenital heart defects, screening procedures, and answers to frequently asked questions. Family members were provided with a brochure and a copy of the consent form before screening. A Web site was developed to provide information on the screening, downloadable educational brochures, information on regional pediatric cardiac providers, and other relevant links.
The screening protocol set in place in February 2008 was developed in collaboration with our system’s pediatric cardiac surgeons and referring pediatric cardiology practices after a comprehensive literature review. This protocol was based primarily on 2 published studies that used postductal (foot) pulse oxygen saturation (Spo2) levels to evaluate the neonates.1,2 Our protocol allowed up to 1 repeat screen, performed immediately, in the event of a saturation <95%. In May 2012 we transitioned to the AAP-endorsed screening algorithm, which requires calculation of the difference between right hand (preductal) and foot Spo2 levels and allows up to 3 repeat screens at 1-hour intervals; this protocol remains in use today. See Fig 2 for details and comparisons of the 2 protocols.
The study team conducts weekly, monthly, and quarterly QA data reviews. Weekly reports that include screening results, hours-of-age at screening, and other related variables are generated by the Epic (Epic Systems Corporation, Verona, WI) EMR report writer for all infants admitted to MHS birth centers. These reports are monitored for screening errors, including missed screens, improper charting, or protocol violations (including failure to repeat a screen or perform an echocardiogram when indicated). Screening errors are immediately reported to birth center nurse managers who follow-up directly and in person with the nurse responsible for performing the screen. If echocardiography is ordered, imaging results are reviewed. Monthly summaries of screening outcomes are maintained, and quarterly cumulative reports are distributed to birth center nurse managers.
The QA reviews do not include follow-up on screened infants with the goal of identifying false-negative results. Although certainly important, such review is beyond the capacity of our program. In 2012 our program underwent major leadership changes, with personnel transitions in both the principal investigator and project manager roles. Between August and December 2012 the QA review was not performed. All QA reviews (weekly, monthly, and quarterly) were reimplemented by January 2013.
In May 2012, IRB approval was granted for screening per the AAP protocol.19 Training on the new procedure was provided for staff at both hospitals (∼50 Good Samaritan Hospital and 70 Tacoma General Hospital staff) and new consent forms were generated. The AAP protocol is more complex than our postductal protocol, with up to 3 screens and determination of the absolute difference between the hand and foot Spo2 levels as an evaluative measure. The existing documentation flowsheet used to track screening results in the child’s EMR was modified to enable recording of both metrics, allow for the possibility of an equivocal result, and allow documentation of a third screen if needed. Once the revised documentation flowsheet was available in Epic, screening under the new protocol began immediately and no further screens were performed with the use of the postductal protocol.
After screening under the AAP protocol was underway we recognized the increased opportunity for human error under this protocol. Our study team enlisted the support of EMR analysts to enhance the recently modified documentation flowsheet. The redesign resulted in a flowsheet that autocalculates the hours-of-age at screening and the difference between the pre- and postductal Spo2 levels, autopopulates the screening outcome, and creates a best-practice alert in the event of a positive screen. The modified flowsheet also added CCHD screening to the discharge checklist. The revised flowsheet went live in the EMR in September 2012. See Supplemental Figs 4–10 for screenshots of the flowsheets for both protocols.
In May 2013 pulse-oximetry screening was designated standard-of-care in our hospital system and parental consent was no longer required. Although parents may still opt out of the screening, infants are routinely screened during discharge procedures. Because of the standard-of-care designation, screening is established in any new facility acquired by our health system without obtaining IRB approval.
We used linear regression to assess overall trends related to each metric, considering P values <.05 to represent a statistically significant trend. We used statistical process control chart methodology to evaluate the relationship between each of the metrics and relevant interventions. Proportion control charts (P-charts) with varying subgroup sizes and lower/upper control limits (LCL/UCL) set at ±3 SDs from the center line were created for the percentage of missed screens, protocol violations, false-positives, and parental declines for the entire study period. LCLs with a negative value were set to zero. Points outside the control limits were considered to represent quarters in which the process was not in statistical control; the variation seen in those quarters is understood to be ascribable to a special cause rather than random/natural process variation. Plots were evaluated for seemingly nonrandom trends in the data. All analyses were conducted in the R statistical computing environment (R Core Team, Vienna, Austria).
Between February 2008 and May 2012, 10 535 newborns were screened under the postductal protocol. A total of 165 (1.5%) of those newborns screened positive; 4 cases of CCHD were identified. Between May 2012 and January 2014, 7828 newborns were screened under the AAP protocol. In this group, second screens were performed on 70 newborns, and 15 had a third screen. There were 15 positive screens (<1%); 1 case of CCHD was identified. All infants in both cohorts with a positive/failed screen received echocardiography. Table 1 provides details of screening results.
Missed screens showed a statistically significant decrease over the lifetime of the screening program (P < .001). The percentage of missed screens was outside of the UCL during 4 of the first 5 quarters of the program, ranging from 6.9% to 9.8% of eligible infants missed. After the protocol transition and implementation of the modified EMR documentation flowsheet, the program experienced a steady downward trend of missed screens. The missed screen rate fell below the LCL for the first time during the quarter in which screening transitioned to standard-of-care. The percentage of missed screens remained below the LCL, and at or below 1%, since the transition (Fig 1A, Table 2).
The percentage of protocol violations during the 6 years of screening did not change significantly (P = .26). The only quarter in which the percentage of protocol violations was outside the UCL was that in which we implemented the protocol change (Fig 3B), with 1.1% of screens in that quarter violating protocol. Although the EMR documentation flowsheet had been modified to allow for recording both the hand and foot measurements at the time of the protocol change in quarter 2 of 2012, the autocalculation features were not available within the flowsheet until quarter 3 of 2012. Once the autocalculation features were implemented, the percentage of protocol violations fell back within the control limit range and subsequently remained near the center line (0.18%).
The percentage of false-positives shows a significant decreasing trend over the lifetime of the program (P = .03), but a clear demarcation exists at the protocol change. A distinct upward trend exists in the percentage of false-positives between the start of the program and the protocol change (Fig 3C). The process fell outside the UCL twice in that time period, with 2.2% and 2.8% of screens showing false-positives. After the protocol change, the false-positive rate dropped sharply and remained <0.5%
No significant trend was observed in the percentage of parental declines (P = .99), although this metric also shows an upward trend until the quarter of the protocol change (Fig 3D). In that quarter the percentage of declined screens increased above the UCL to 3.9%, remained outside the UCL the following quarter (at 3.2%), but dropped steadily until falling below the LCL the quarter after the screening became standard-of-care. Parents of only 3 infants declined screening in the last 3 reported quarters.
We reviewed data for the months during which QA reviews were not conducted. Missed screens occurred in 3.7% of eligible patients, slightly more than the overall average of 3.5% missed. Protocol violations occurred in 0.39% of screens during that time, which is double the overall average. Of parents, 2.8% declined screening during this time, compared with an average of 1.7%. There were no false-positive screens in these months.
This 6-year review of a pulse-oximetry screening program to detect CCHDs provides a unique perspective on the evolution of a screening program with insights on quality improvement interventions and lessons learned. Analysis of the percentage of missed screens, protocol violations, false-positives, and parental decline of screening with respect to 3 quality improvement interventions revealed that each metric was substantially influenced by at least 1 intervention. A thorough QA review of screening results performed at regular intervals has proven to be critical to the integrity of our program.
Upon implementation of the AAP protocol, the false-positive rate decreased drastically. The additional third screen and evaluation of saturation levels in the hand and foot likely contributed to the lower false-positive rate. Under the AAP protocol, 15 of 19 patients with equivocal second screens were rescreened and 9 of these patients (60%) had a negative final screen. Fifty patients evaluated with the AAP protocol had a postductal saturation <95% but ultimately screened negative due to a preductal saturation ≥95% and within ±3% of the postductal value. These screens would have been positive under our postductal protocol or the protocol currently in use in New Jersey, which requires pre- and postductal saturations ≥95%.
Redesign of the EMR flowsheet led to fewer missed screens and protocol violations. A recent study by Oster et al26 showed a significant difference in correct interpretation of screening results when using a computer-based tool as opposed to manual interpretation of results using the AAP algorithm. Health care providers in the study correctly identified only 65.4% of failed screens through manual interpretation. Facilities without an EMR or resources to build an autocalculating system can use a free online tool to autocalculate and interpret the results of screening per the AAP protocol.27
The increase in parental declination of screening at the protocol shift could have been a result of the mechanism of consent, because nursing staff were required to consent patients to a more complex algorithm using a longer consent form. This finding suggests that the informed consent process itself is important to compliance and also suggests that consent forms should be crafted to minimize attention to standard-of-care aspects of the process and instead focus clearly on additional activities for which consent is required (ie, use of data or permission to contact the family in the future).
The increase in the percentage of protocol violations during the months of missed QA review emphasizes the importance of a routine QA system. QA reviews enable rapid response in the event of a missed screen, potentially avoiding further misses. More than 400 infants are born in our system each month, but review of the screening data requires only 15 to 20 minutes per week. An additional 45 to 60 minutes each quarter are used to maintain the master data set and distribute reports to birth center staff. This amount represents a small time commitment with important benefits because we are able to perform immediate reviews of the screening results and complete longitudinal assessments of our program.
The longevity of our screening program and its transition from research to standard-of-care, despite major changes in program leadership, shows that a CCHD screening program using pulse-oximetry is sustainable over time and that screening can become embedded in the hospital’s routine newborn care. Our program has been successful due to continuing process improvement efforts and careful monitoring of our program.
A limitation of this study is our inability to follow-up with patients or to perform chart reviews to determine a false-negative rate. A recent article by Singh et al28 shows the additional benefit of pulse-oximetry screening in identifying noncardiac conditions that require medical attention, such as pneumonia or sepsis. Although chart review for all infants on whom an echocardiogram was performed is beyond the scope of this report, future review of these charts could be warranted to further explore the benefits of pulse-oximetry in identifying such conditions before discharge.
CCHD screening with the use of pulse-oximetry is becoming widely implemented and is now mandated in several states and countries.15,16,23 Our long-term experience with pulse-oximetry screening suggests that cost-effective screening can be implemented with minimal burden to hospital staff using the AAP screening protocol. In addition to education and training, autocalculation of screening outcomes, EMR documentation and reporting, and routine QA reviews are important components for an efficient and sustainable screening program.
We acknowledge Dr Ronald Woods, who established the pulse-oximetry screening program at Tacoma General Hospital. We also acknowledge the birth center families, MultiCare nursing staff, cardiac sonographers, newborn care providers, and staff pediatric cardiologists whose passion for this project has made it a success. In addition, we thank the anonymous reviewer who suggested the use of control chart methodology for analysis of our data.
- Accepted August 26, 2014.
- Address correspondence to Matthew V. Park, MD, NorthWest Congenital Heart Care, 314 Martin Luther King Way, Suite 303, Tacoma, WA 98405. E-mail:
Ms Pflugeisen supervised data collection from 2012 to the present, acted as coinvestigator from 2012 to the present, performed data analyses, and drafted the final manuscript as submitted; Dr Amoroso contributed to the study design and critically reviewed the manuscript; Ms Zook acted as study coordinator from 2008 to 2011 and as coinvestigator in 2012, oversaw program implementation and education at Tacoma General and Good Samaritan Hospitals, supervised data collection and performed data analysis, critically reviewed the manuscript, and reviewed the final manuscript; Dr Welke acted as principal investigator from 2011 to 2012 and reviewed the final manuscript; Ms Reedy contributed to the study design, provided programmatic oversight from 2011 to 2012, and reviewed the final manuscript as submitted; Dr Park acted as principal investigator from 2012 to the present, conceptualized modifications to the study protocol, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by the Fraternal Order of Eagles.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Copyright © 2015 by the American Academy of Pediatrics