Abstract
OBJECTIVE: To describe the variability across hospitals in diagnostic test utilization for children diagnosed with community-acquired pneumonia (CAP) during emergency department (ED) evaluation and to determine if test utilization is associated with hospitalization and ED revisits.
METHODS: We conducted a retrospective cohort study of children aged 2 months to 18 years with ED visits resulting in CAP diagnoses from 2007 to 2010 who were seen at 36 hospitals contributing data to the Pediatric Health Information System. Children with complex chronic conditions, recent hospitalization, trauma, aspiration, or perinatal infection were excluded. Primary outcomes included diagnostic testing, hospitalization, and 3-day ED revisit rates across hospitals. We examined variation in diagnostic testing among hospitals by using multivariable mixed-effects logistic regression.
RESULTS: A total of 100 615 ED visits were analyzed. Complete blood count (median: 28.7%), blood culture (27.9%), and chest radiograph (75.7%) were the most commonly ordered ED diagnostic tests. After adjustment for patient characteristics, significant variation (P < .001) was found for each test examined across hospitals. High test-utilizing hospitals had increased odds of hospitalization compared with low-utilizing hospitals (odds ratio: 1.86 [95% confidence interval: 1.17–2.94]; P = .008). However, differences in the odds of ED revisit between the low- and high-utilizing hospitals were not significant (odds ratio: 1.21 [95% confidence interval: 0.97–1.51]; P = .09).
CONCLUSIONS: Emergency departments that use more testing in diagnosing CAP have higher hospitalization rates than lower-utilizing EDs. However, ED revisit rates were not significantly different between high- and low-utilizing EDs. These results suggest an opportunity to reduce diagnostic testing for CAP without negatively affecting outcomes.
- children
- clinical practice variation
- diagnostic tests
- emergency medicine
- health resources
- health services research
- hospitalization
- logistic models
- physician practice patterns
- pneumonia
- radiography
- utilization
- APR-DRG —
- All Patient Refined Diagnosis Related Group
- CAP —
- community-acquired pneumonia
- CBC —
- complete blood count
- CHA —
- Children’s Hospital Association
- CT —
- computed tomography
- ED —
- emergency department
- ICD-9-CM —
- International Classification of Diseases, Ninth Revision, Clinical Modification
- LOS —
- length of stay
- PHIS —
- Pediatric Health Information System
What’s Known on This Subject:
There is wide variation in testing and treatment of children hospitalized with pneumonia. Limited data are available on diagnostic testing patterns and the association of test utilization with disposition outcomes for children with pneumonia evaluated in the emergency department (ED).
What This Study Adds:
Significant variation exists in testing for pediatric pneumonia. EDs that use more testing have higher hospitalization rates. However, ED revisit rates were not significantly different between high- and low-utilizing EDs, suggesting an opportunity to reduce testing without negatively affecting outcomes.
Community-acquired pneumonia (CAP) is the most common serious bacterial infection in children.1–5 Despite its high incidence and morbidity, there is a paucity of data regarding the ability of laboratory testing to diagnose pneumonia, differentiate etiology, and predict clinical course.6 For example, although white blood cell count and C-reactive protein are 2 commonly obtained laboratory markers obtained in pediatric infection, they have only fair specificity and poor sensitivity in the diagnosis of bacterial pneumonia; the degree of elevation does not distinguish bacterial from viral infection.7,8 There is wide variability in hospitalization rates and diagnostic testing among hospitalized children with CAP, highlighting the fact that with this lack of evidence comes the potential for variation in resource utilization.9–11
Studies in adults with CAP suggest that despite widely variable processes between hospitals and providers, outcomes do not differ substantially.12–14 Pediatric studies in other diseases also have demonstrated variation in emergency department (ED) testing that is associated with increased resource utilization, including hospital admission.15,16 Although 2 previous studies have documented variation in chest radiography and antibiotic utilization in the ED for CAP,17,18 no studies, to our knowledge, have examined the association between diagnostic testing and ED disposition decisions. The objectives of the current study were to describe the variability across hospitals in diagnostic test utilization for children presenting to the ED with CAP and to determine if test utilization is associated with the disposition decision.
Methods
Study Design and Data Source
This multicenter, retrospective cohort study included ED visits of children diagnosed with CAP. Data were from the Pediatric Health Information System (PHIS), an administrative database of 43 not-for-profit, tertiary care pediatric hospitals in the United States affiliated with the Children’s Hospital Association (CHA; Shawnee Mission, KS). Data quality and reliability are assured through a joint effort between CHA and participating hospitals. Hospitals provide discharge/encounter data, including demographic characteristics, procedures, and diagnoses in International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM), format; 42 of these hospitals also submit resource utilization data (eg, pharmaceuticals, imaging, laboratory). Data are de-identified, but encrypted medical record numbers permit identification of patients across multiple visits to the same hospital.19 The current study excluded 6 hospitals because ED or resource utilization data were not available, leaving 36 hospitals in our cohort. This study was reviewed and approved by the institutional review board of The Children’s Hospital of Philadelphia.
Study Population
Inclusion Criteria
Patients between 2 months and 18 years of age who were diagnosed with CAP from July 1, 2007, to June 30, 2010, were eligible for inclusion. If patients had multiple visits for CAP in a 28-day period, only the initial visit was included to best capture the utilization on initial presentation for each single episode of CAP. To minimize within-patient clustering, if a patient had multiple distinct episodes of CAP resulting in multiple visits for pneumonia over the 3-year study period, a single initial visit, representing a distinct episode of CAP, was randomly selected for inclusion. Overall, 90.7% of patients had a single visit in the study period, 7.9% had 2 visits, and 1.4% had ≥3 visits.
Definition of Pneumonia
A previously validated algorithm was used to identify patients with CAP.20,21 Patients were considered to have CAP if they met either of the following criteria: an ICD-9-CM primary diagnosis code indicating pneumonia (codes 480–483 and 485–486), empyema (510), or pleurisy (511.0, 511.1, or 511.9) or a primary diagnosis of a pneumonia-related symptom ICD-9-CM code (eg, fever, cough) (Supplemental Information 4) and a code for pneumonia, empyema, or pleurisy in any other diagnosis position.20
Exclusion Criteria
Patients with complex chronic conditions, as defined by Feudtner et al,22 were excluded because these patients may have unmeasured covariates not reflective of the general population and may therefore warrant a different approach. To exclude patients with hospital-acquired pneumonia, we excluded patients hospitalized within 30 days preceding the ED visit. We also excluded patients with codes indicating trauma (518.5, 800–999) or aspiration pneumonitis (506–508). To minimize the inclusion of patients with perinatally acquired and neonatal hospital-acquired infections, we excluded patients with diagnoses associated with pregnancy and delivery (640–679, 760–779). Patients seen initially at another institution and directly transferred were also excluded because they may have had testing performed before arrival.
Diagnostic Testing
Laboratory measures included complete blood count (CBC), blood chemistries, inflammatory markers (C-reactive protein and erythrocyte sedimentation rate), coagulation studies, viral studies, and blood cultures. Imaging studies included chest radiography, ultrasonography, and computed tomography (CT) scans. All tests were determined with PHIS-specific Clinical Transaction Classification codes. Only measures obtained on the first hospital day were included in the analyses to capture those tests performed in the ED.
Outcome Measures
Outcomes included hospital-level rates of variation in diagnostic testing, hospital admission rate on initial ED visit, and ED revisit rate within 3 days after the index ED visit in those initially discharged. As a secondary analysis, we examined the hospital length of stay (LOS) in those patients who were initially admitted. A short LOS was defined as ≤2 days.
Covariates
Patient-level covariates included age, gender, race/ethnicity, primary source of payment, and admission season and year. Hospital-level covariates included geographic location, average hospital daily census, ED annual volume, percent uninsured, and severity. Hospital-level severity was determined by generating an average of each hospital’s All Patient Refined Diagnosis Related Group (APR-DRG) severity score for all patients seen at that hospital during the study period. APR-DRG severity scores represent illness severity and risk of death for hospitalized patients.23 Because these scores are not validated for outpatients, we did not apply them at the ED visit level but rather used them at the hospital level as a proxy for overall hospital severity of illness for all diagnoses.
Statistical Analysis
To explore variation across hospitals in diagnostic testing, unadjusted distributions for each test were determined by calculating the rate of subjects at each hospital who received the test and summarizing these rates across hospitals. Adjusted rates were obtained by adjusting hospital-level testing rates for patient-level characteristics; we used a mixed-effects logistic regression model for the subject-level binary outcome of test use (eg, blood cultures, yes/no), adjusted for patient age, race/ethnicity, year and season of presentation, and insurance status, with hospital-specific random intercepts. Each random intercept represented the degree to which a hospital’s test use departed from what would be expected, on average, for a hospital with a similar case mix. This model was used to estimate population-averaged rates of testing expected at each hospital based on its patients’ characteristics. These expected rates were compared with observed rates at each hospital, and an adjusted rate was obtained by standardizing the unadjusted rate by this ratio of observed to expected rates of testing.24
Hospitals that were high or low utilizers of testing were identified by detecting outliers in the random effects distribution.25 A mixed-effects logistic regression model was fit for each diagnostic test with hospital-specific random intercepts. For each hospital, a test statistic was calculated as the ratio of the estimated random intercept to its estimated SE. Each hospital was assigned a rank from low to high based on ascending values of this statistic for each test. Pairwise Pearson correlation coefficients of these ranks were determined to explore the correlation between the utilization of tests. The sum of ranks across tests was obtained for each hospital, and an overall utilization rank was assigned. Hospitals with ranks in the upper tertile were defined as “high” utilizers, and those in the lowest tertile were defined as “low” utilizers.
Mixed-effects logistic regression models with hospital-specific random intercepts were used to determine the association of test utilization with each binary outcome for patient disposition. In these models, utilization of each test was decomposed into a within-hospital and between-hospital term. By simultaneously estimating the effects associated with average hospital-level utilization and patient-level utilization, we avoided the potential for ecological bias resulting from assuming a single aggregate utilization measure.26 Finally, mixed-effects logistic regression models were built with utilization tertile as the primary independent variable to examine the association of overall utilization with disposition outcome.
To account for potentially important confounders not included in the analyses, several sensitivity analyses were performed. Because APR-DRG severity scores cannot be accurately used for individual ED patient visits, we used the average of APR-DRG severity scores for all patients seen at each hospital as a proxy for hospital-level overall severity and examined these across hospitals. To assess the potential association between severity and utilization, a linear regression model, including overall hospital APR-DRG score and utilization tertile, was used. Because patients with asthma or bronchiolitis may warrant a different management approach, we repeated analyses excluding these patients. A sensitivity analysis was also performed to assess the extent to which an unmeasured factor could alter the results.27
All analyses were performed by using Stata 12.1 (Stata Corp, College Station, TX). Graphics were generated by using R 2.15.0 (R Foundation for Statistical Computing, Vienna, Austria).
Results
During the study period, 100 615 ED visits for CAP were eligible for inclusion across the 36 hospitals. The median age was 3 years (interquartile range: 1–6). Fifty-four percent of subjects were male. Thirty-four percent of patients were non-Hispanic white, 27% were non-Hispanic black, and 20% were Hispanic. Forty percent of patients had government insurance. Thirty-six percent of ED visits occurred in winter. Twenty-six percent of the subjects were hospitalized initially; 6.4% of patients initially discharged from the ED returned to the ED within 3 days (Supplemental Information 5).
Variation in Diagnostic Testing
Unadjusted and adjusted summary statistics for each diagnostic test are presented in Table 1. CBC, blood culture, and chest radiograph were the most frequently ordered tests. Significant variation was found between hospitals for each diagnostic test in both unadjusted and adjusted analyses. After adjustment for patient characteristics, CBC, blood culture, and inflammatory markers demonstrated the broadest range of variation across hospitals.
ED Utilization of Diagnostic Tests for CAP Across 36 PHIS Hospitals
Hospital-Level Utilization of Diagnostic Testing
Figure 1 illustrates both the individual utilization ranking for each test according to hospital and overall utilization tertiles by using the sum of the ranks. Hospitals with high overall utilization were often high utilizers of multiple individual tests, as indicated by the darker shaded boxes. The most highly correlated tests were CBC with blood culture (r = 0.83) and CBC with chemistries (r = 0.78).
Diagnostic test utilization by hospital. For each diagnostic test listed on the horizontal axis, hospitals were ranked according to their utilization (see the Statistical Analysis section). A shaded box represents the utilization rank for each hospital; darker shading indicates higher utilization. For each hospital, overall utilization was calculated as the sum of utilization ranks across diagnostic tests. Hospitals were ordered according to their overall utilization rank, from low to high, on the left vertical axis. Tertiles of overall utilization were used to classify overall utilization as low, moderate, or high (as indicated on the right vertical axis). CRP, C-reactive protein; CXR, chest radiograph; ESR, erythrocyte sedimentation rate; US, ultrasound.
Utilization of Diagnostic Testing and Patient Disposition
With the exception of chest radiography, all diagnostic tests were highly associated with increased odds of hospitalization (Table 2). The highest odds of hospitalization were seen with ultrasound, CT, coagulation studies, CBC, and chemistries. Receipt of CBC, blood culture, chemistries, and inflammatory markers on initial ED visit were all associated with ED revisit and subsequent hospitalization within 3 days.
Adjusted Mixed-Effects Logistic Regression: Association of Utilization With Hospitalization and ED Revisit
As overall utilization rank increased, the rate of hospitalization increased (Fig 2). Hospitalization rate ranged from 17.6% to 67.6% in the high-utilization tertile compared with 10.4% to 38.2% in the low-utilization tertile. In mixed-effects regression analyses, high-utilizing hospitals had an 86% increased odds of hospitalization compared with low-utilizing hospitals (Table 3). For patients hospitalized, there was no difference in the odds of a short LOS in the high-utilization tertile compared with the low-utilization tertile (odds ratio: 0.96 [95% confidence interval: 0.78–1.17]; P = .7).
Overall utilization and disposition outcomes. Individual hospitals are represented by shaded circles; darker shading indicates increased overall utilization. A, Hospitalization rate according to overall utilization. B, Three-day ED revisit rate according to overall utilization.
Overall Utilization and Association With Hospitalization and ED Revisit
The rate of ED revisit within 3 days, however, was similar between low-utilizing (range: 5.3%–10.5%) and high-utilizing (range: 5.3%–11.1%) EDs. There was no significant difference in the odds of ED revisit between the low- and high-utilization tertiles.
Sensitivity Analyses
To examine overall hospital-level severity, we examined average APR-DRG severity scores across hospitals and found that the average APR-DRG severity scores, which ranged from 1 (least severe) to 4 (most severe), ranged from 1.1 to 1.35 across all hospitals in our cohort. In addition, there was no association between utilization tertile and overall hospital APR-DRG severity score (low-utilization, reference group; medium-utilization coefficient: 0.03, P = .2; high-utilization coefficient: 0.04, P = .1). There was no difference in the results when patients with asthma and bronchiolitis were excluded. Several scenarios to evaluate the impact of an unmeasured confounder were examined. Assuming that the prevalence of the unmeasured confounder was 50% greater in high-utilizing hospitals compared with low-utilizing hospitals, there would need to be more than double the risk of hospitalization given the unmeasured confounder at the high-utilizing hospitals to attenuate the odds ratio for hospitalization to 1. When the prevalence of the unmeasured confounder was increased to >50%, similar results were obtained, suggesting that the unmeasured confounder would need to have a high prevalence to attenuate the results observed in these analyses.
Discussion
In this multicenter study of children with CAP, there was significant variation across EDs in the use of diagnostic testing. Certain EDs were consistently high utilizers of diagnostic tests for CAP. High utilization was associated with increased odds of hospitalization. Although it might be expected that hospitals which test less might “miss” cases and thus have a higher ED revisit rate, our results demonstrate that low utilization was not associated with increased revisit rates. This finding suggests that high-utilizing hospitals may be able to decrease utilization and hospitalization without overlooking children who warrant hospital admission.
Substantial variation in ED use of diagnostic testing was present despite adjustment for patient characteristics. These results build on those of previous studies demonstrating significant variation in testing and antibiotic use among hospitalized children with CAP10 and variation in chest radiograph and antibiotic use in the ED.17,18 In general, the degree of variation in a care process is associated with the level of uncertainty surrounding patient outcomes.28 The first US guidelines for the management of pediatric CAP were released in 2011.6 Although the lack of these guidelines at the time of our study may explain some of the variability we observed, these guidelines also illustrate the ambiguity surrounding diagnostic testing for pediatric pneumonia. Of the 27 recommendations regarding diagnostic testing, almost one-half (including the use of the CBC) are based on low-quality evidence.6 The degree of elevation of the white blood cell count does not reliably predict bacterial etiology or development of a severe course.7,29,30 Despite the paucity of evidence on the utility of CBC in CAP, it was the most frequently ordered laboratory test in our study, and there was substantial variation in its utilization.
Certain institutions are consistently high utilizers of diagnostic tests. Certain tests, such as coagulation studies, ultrasound, and CT scans, would likely only be ordered in sick patients or in patients with a specific indication. Utilization ranks for these tests seem random and do not correlate with overall hospital utilization. The reverse is observed in tests in which the results do not usually affect clinical outcome in routine CAP, such as CBC, blood cultures, chemistries, and inflammatory markers. For these tests, utilization is highly clustered and strongly correlated with overall hospital utilization. These same tests individually were associated with increased odds of ED revisit across hospitals, whereas the overall measure of utilization (eg, high-utilizing versus low-utilizing hospitals) was not associated with ED revisit. This increase in odds of hospitalization and revisit for these individual tests that are frequently ordered together, without a difference in ED revisit based on overall utilization, suggests that patients with a more severe presentation (at all hospitals) receive these tests and are therefore more likely to return to the ED.
Test results may be driving disposition decisions in high-utilizing hospitals. There is precedent for results of ED testing to influence the decision to hospitalize.15 ED discharge is less likely if the physician is faced with abnormal test results.31 Although it is possible that low-utilizing hospitals might overlook children who warrant hospitalization as a result of less frequent testing, our results demonstrate no association between test utilization and ED revisit rates. Taken together, these findings suggest that low-utilizing hospitals are not discharging patients from the ED who actually warrant admission at initial presentation.
The current study used administrative data and thus has several limitations. First, tests and outcomes may be miscoded, resulting in misclassification bias. If present, miscoding would likely be random and nondifferential, resulting in bias toward the null. This limitation is curtailed by use of a validated algorithm as well as through coding consensus conferences held by the CHA that would minimize, although not completely eliminate, this issue. Second, because PHIS comprises freestanding children’s hospitals, our results may not be generalizable to all hospitals caring for children with CAP. No study has examined variation in testing and treatment of pediatric pneumonia at nonchildren’s hospitals; however, there has been documentation of variation in antibiotic prescribing in the office/outpatient setting.5 Given these data and what we know about variation of care, we would have no reason to suspect that the extent of variation would be any less at nonchildren’s hospitals. Third, PHIS does not include information on indications for tests or threshold for admission across hospitals. We attempted to account for this lack of data by excluding patients with complex chronic conditions who may have other indications for testing. In addition, if there were differences in test indications or admission thresholds, these would likely become significant after adjusting for case mix, which we accomplished by using mixed-effects regression models, thus minimizing the influence of these factors in our analyses. Furthermore, limiting testing to performance on the day of the visit may minimize the true rate of diagnostic testing, particularly for hospitals that have a high patient burden in the evening and overnight hours. To account for this limitation, we repeated analyses for each diagnostic test, including testing on both hospital days 1 and 2 without substantial differences in our results. Fourth, we do not have test results, and it is therefore impossible to know precisely the effect of the result of a test on the disposition decision. However, the variation and association of utilization with disposition were significant after controlling for patient-level and hospital-level covariates in a statistical model that accounted for clustering within hospitals.
Finally, the PHIS database does not include a robust measure of illness severity or initial presenting symptoms. It is possible that individual patient severity may confound the relationship between utilization and disposition. Furthermore, spectrum bias may be present if certain hospitals care for more severe patients overall. We addressed these limitations in several ways. We included as a covariate in the regression models each hospital’s average overall APR-DRG severity score as a proxy for overall severity level at the hospital level. To further explore the relationship between severity and utilization, we performed additional analyses that found no association between overall hospital APR-DRG severity score and utilization tertile. In addition, the narrow range of overall APR-DRG scores across hospitals indicates minimal differences in overall severity of illness. The lack of a difference in short-stay hospitalizations suggests that high-utilizing hospitals are not seeing more severe patients overall. Finally, in exploring the effects of an unknown confounder on our results, we found that such a confounder (eg, severity) would have to be highly prevalent to attenuate the results.
Conclusions
The significant across-hospital variation in diagnostic tests performed in the ED for childhood CAP illustrates the need to improve the quality of care provided. If overutilization can be diminished, there is the potential to decrease unnecessary hospitalizations, decrease costs, prevent unnecessary hospital-acquired infections, and potentially improve short-term quality of life in children with CAP. The results of this study argue for a national quality improvement effort to better understand utilization of resources in this common pediatric disease. Future research should seek to better understand motivation for testing in CAP and to prospectively evaluate the effects of testing on disposition decision-making and clinical outcomes.
Footnotes
- Accepted May 17, 2013.
- Address correspondence to Todd A. Florin, MD, MSCE, Division of Emergency Medicine, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Ave, ML 2008, Cincinnati, OH 45229. E-mail: todd.florin{at}cchmc.org
Dr Florin conceptualized and designed the study, analyzed and interpreted the data, drafted the initial manuscript, and approved the final manuscript as submitted; Dr French was involved in the design of the study, supervised the data analysis and interpretation, reviewed and revised the manuscript, and approved the final manuscript as drafted; Drs Zorc and Alpern were involved in the design of the study, participated in the interpretation of the data, reviewed and revised the manuscript, and approved the final manuscript as drafted; and Dr Shah supervised the conceptualization and design of the study, participated in the interpretation of the data, reviewed and revised the manuscript, and approved the final manuscript as drafted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: Supported by a pilot grant from The Center for Pediatric Clinical Effectiveness at The Children’s Hospital of Philadelphia.
COMPANION PAPERS: Companions to this article can be found on pages 229, 245, and 369, and online at www.pediatrics.org/cgi/doi/10.1542/peds.2013-0359, www.pediatrics.org/cgi/doi/10.1542/peds.2012-2830, and www.pediatrics.org/cgi/doi/10.1542/peds.2013-1569.
References
- Copyright © 2013 by the American Academy of Pediatrics