Objective. Pediatric resident physicians’ clinical diagnostic accuracy of otitis media is unknown. We attempted to correlate the clinical examination of pediatric house staff with pediatric otolaryngologists and tympanometry.
Methods. Pediatric residents evaluated patients who were scheduled in the pediatric acute care clinic and completed a provider examination form detailing their otoscopic findings, interpretation, and treatment plan. Patients were then immediately reevaluated by a pediatric otolaryngologist using an identical form. Tympanometry was also performed by a pediatric audiologist. We used κ statistics to calculate correlation of clinical findings and interpretation.
Results. A total of 103 patients consented for the study; 70 patients were examined by 27 different pediatric residents with 43 patients (86 ears) examined by all 3 providers. Correlation of clinical findings between all pediatric residents and the pediatric otolaryngologists was a κ statistic of 0.30 (fair agreement). The individual diagnostic finding with the greatest correlation was tympanic membrane erythema (κ statistic: 0.40 [fair agreement]), and the worst correlate was tympanic membrane position (κ statistic: 0.16 [slight agreement]). Resident interpretation and tympanometry yielded a κ statistic of 0.20 (slight agreement), and the otolaryngologist interpretation and tympanometry yielded a κ statistic of 0.32 (fair agreement).
Conclusions. Otitis media is the most common disease seen by practicing general pediatricians, but there is a paucity of formalized resident education. We demonstrated only a slight to moderate correlation between the clinical diagnostic examinations of pediatric residents and pediatric otolaryngologists and tympanometry.
Acute otitis media (AOM) is the most commonly diagnosed disease in childhood and a primary factor in increased antibiotic resistance.1 Variations in diagnostic criteria2 and antibiotic therapy3 of AOM exist among physicians. No association has been found between diagnostic variation and various primary care residencies or the length of time in practice. In fact, primary care physicians show as much variation in AOM diagnosis as those trained in surgical subspecialties.2 Accurate diagnosis requires differentiating normal ears from AOM, symptomatic disease that requires treatment with antibiotics, and otitis media with effusion (OME), asymptomatic disease with persisting middle ear effusion (MEE) generally considered not to require antibiotics.
An expert panel recently reviewed the problem of misdiagnosis of AOM in infants and young children and recommended that pediatric training programs emphasize otoscopic skills development.4 Many pediatric residency programs acknowledge a lack of formalized AOM education,5 and at present there are no data regarding pediatric residents’ clinical accuracy in diagnosing AOM. We investigated pediatric residents’ clinical diagnostic accuracy of otitis media compared with the clinical examination of a pediatric otolaryngologist, with both compared with tympanometry. We used κ statistics to calculate correlation between the individual providers’ clinical examination findings and final interpretation.
Site and Recruitment
This prospective study was performed at Lucile Packard Children’s Hospital in the Stanford University Medical Center in Palo Alto, California, from January 2 to June 29, 2001. The study site was the Lucile Packard Children’s Hospital Acute Care Clinic (ACC), which serves patients for acute visits from a variety of socioeconomic classes with approximately 800 patients seen each month, including 60 to 80 diagnoses of AOM per month. Health care providers in the ACC are Stanford University pediatric (postgraduate year [PGY]-1 through PGY-3) residents as well as PGY-1 Stanford University emergency medicine residents and Stanford University medical students, all supervised by attending university faculty pediatricians. The Stanford University pediatric residency program consists of 59 residents, 29 of which rotated through the ACC during the study period. Institutional review board approval and pediatric resident consent were obtained before study initiation.
Eligible study participants all were children ages 3 months to 16 years with scheduled acute visit appointments in the ACC. Exclusion criteria included children with tympanostomy tubes in place or middle ear abnormalities (eg, cholesteatoma) that might complicate diagnosis. Pediatric residents or attending physicians obtained informed consent from patients or their parents at the completion of the scheduled acute visit. Only patients who were examined by a pediatric resident, not an emergency medicine resident or a medical student, were asked to participate in the study. Each pediatric resident participated in this study on a voluntary basis; there were no refusals.
Pediatric residents completed an examiner form (Fig 1) that described their otoscopic physical findings, final interpretation, and treatment plan after examining a child and before presenting the patient to the attending pediatrician. Study patients then proceeded to the nearby pediatric otolaryngology clinic for reevaluation and tympanometry. Pneumatic otoscopy with the same equipment (Welch Allyn 20000 Otoscope; Welch Allyn, Skaneateles Falls, NY) was performed by 1 of 2 board-certified pediatric otolaryngologists. Tympanometry (Welch Allyn GSI-33 Middle Ear Analyzer) was performed on each patient by 1 of 5 licensed pediatric audiologists. Otoscopists and audiologists were blinded as to the results of either pneumatic otoscopy or tympanometry as well as the resident examination findings. Pneumatic otoscopy and tympanometry were performed in a random order on each patient.
We used the clinical diagnostic examination of the pediatric otolaryngologist as the standard, with tympanometry as an additional evaluation tool. The otoscopic diagnosis of AOM was symptomatic children (fever, otalgia, irritability, etc.) with an MEE as diagnosed by pneumatic otoscopy. A chart review confirmed symptomatic patients. MEE was diagnosed as impaired or lack of tympanic membrane (TM) movement. OME was diagnosed as MEE by the same otoscopic criteria in an asymptomatic child. Tympanometry curve results were classified according to Jerger6 as types A, B, or C, with types A and C curves interpreted as no MEE and type B as predictive of a MEE. We used a cutoff value of −150 daPa to distinguish a negative pressure C curve from a normal A curve.
We used the κ statistic to analyze correlation among the pediatric residents, pediatric otolaryngologists, and tympanometry. The κ statistic is a measure of correlation from 0 to 1, with 0 being random and 1 showing perfect correlation. Additional classifications for interpreting the correlation of the κ statistic are as follows: <0, poor; 0 to 0.20, slight; 0.21 to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; and 0.81 to 1.0, almost perfect correlation.7
Enrollment totaled 103 patients, 70 of which (140 ears) were examined. The remaining 33 patients who consented never arrived at the otolaryngology clinic (19) or decided not to participate after consent was obtained (14). Each of the 70 examined patients was seen by a pediatric resident, but not all were seen by the pediatric otolaryngologist (18) or pediatric audiologist (9) because of occasional operating room emergencies and other constraints; 43 patients were examined by all 3 providers, and statistical analysis was performed only on those 43 patients.
Twenty-seven of the 29 different residents in the ACC during the study period examined the 43 patients. The 27 residents were divided among the 3 training years as follows: 8 PGY-1, 7 PGY-2, and 12 PGY-3 residents. Pediatric otolaryngologists performed pneumatic otoscopy versus visual otoscopy a higher proportion of the time (1.0; 95% confidence interval: 0.95–1.0) than pediatric residents (0.79; 95% confidence interval: 0.67–0.87). The age ranges of children examined (Table 1) were 4 to 165 months of age, with a mean age of 41.4 months. This yielded 25.6% of patients who were younger than 1 year and 44% who were younger than 2 years.
The diagnoses of the 43 patients (86 ears) examined by the 27 pediatric residents and the 2 pediatric otolaryngologists are listed in Table 2. The 43 patients were examined by residents in the following levels of training: 15 PGY-1, 10 PGY-2, and 18 PGY-3. The correlation between the final interpretation of the pediatric resident ear examination to the pediatric otolaryngologist’s examination (Table 3) was fair (κ statistic: 0.30). This finding was similar for the residents with each of the 2 individual otolaryngologists (κ statistics: 0.28 and 0.31). Analyzing the individual PGY levels of pediatric residents revealed that the PGY-2 residents had the best correlation to the pediatric otolaryngologists (κ statistic: 0.47). Separating the patients into categories of <2 years of age and >2 years of age (Table 4) yielded κ statistics of 0.25 and 0.33, respectively, demonstrating that age did not significantly affect diagnostic correlation. We also collapsed the data into 2 simpler categories: 1) an abnormal ear examination that comprised those interpreted as either AOM or OME and 2) a normal ear examination, eliminating for that analysis the ears labeled “unable to tell” or “ retracted TM, no disease.” This compared clinical determination of the presence or lack of an effusion, respectively. With this broader and dichotomous diagnosis definition (Table 5), the correlation between clinicians was still only fair (κ statistic: 0.30 increased to 0.38).
Tympanometry curve results were similarly collapsed into 2 categories: mobile (curves A and C, normal ear and negative pressure, respectively) and flat (curve B) for comparison with the physician diagnoses collapsed into the previous 2 categories, abnormal (AOM or OME) and normal. There was slight correlation (Table 6) between the residents (κ statistic: 0.20) and a fair correlation between the otolaryngologists (κ statistic: 0.32) and tympanometry. An additional analysis of physician diagnosis of TM mobility, indicative of an effusion and the best known predictor of otitis media, versus tympanometry yielded a fair correlation between pediatric residents (κ statistic: 0.25) and a fair correlation between otolaryngologists (κ statistic: 0.30) to tympanometry.
Providers did not select antibiotic treatment choices or duration of therapy from options on the examiner form but were instead allowed to write in their choice. Pediatric residents’ choices included 10 different antibiotic regimens, whereas otolaryngologists’ responses were simply written as “treat” or “none.” Therapy was given for every patient with AOM by both pediatric residents and otolaryngologists; the otolaryngologists treated 21 of 52 patients seen, whereas the residents treated 31 of the 70 patients seen. However, although the otolaryngologists did not treat patients diagnosed with OME, the residents treated 9 of 15 patients in whom both ears were diagnosed as OME or 1 ear had OME and the other was normal.
Otitis media is the most common problem in general pediatric practice, and its diagnosis and treatment account for >$5 billion in health care expenditures per year.8 Despite the great prevalence of this disease, accurate diagnosis remains difficult. This study compared the correlation between the clinical examination of pediatric residents at a major university teaching hospital with the clinical examination of a pediatric otolaryngologist; both were compared with tympanometry. Assuming that residents could diagnosis AOM after visualizing an obvious bulging TM, we were curious as to the finer distinction of the presence or absence of an effusion to discriminate AOM or OME, distinguished by symptomatology, from a normal ear. With κ statistics, our study demonstrated a slight to fair correlation between the 2 clinical examinations, as well as both examinations with tympanometry.
The signs and symptoms of AOM are protean; no specific symptom is a reliable predictor of AOM.9 The most reliable signs of AOM are impaired mobility, cloudiness, and bulging of the TM.10 The significance of a red TM is controversial because erythema may occur in the presence of fever or screaming.11 The presence of an MEE is fundamental to the development of AOM, and the lack of an effusion makes the diagnosis of AOM impossible. Eustachian tube dysfunction results in decreased ventilation and a negative pressure in the middle ear as the vascular mucosal lining of the middle ear cavity absorbs air in a closed system and creates a vacuum. The negative middle ear pressure reverses the normal flow of secretions, and mucus, replete with bacterial flora from the nasopharynx, enters the normally sterile middle ear.12
Experts have specifically underscored the diagnostic value of pneumatic otoscopy,4 stating that the correct use of pneumatic otoscopy will result in more accurate diagnosis of middle ear pathology and more appropriate antibiotic prescribing.13 Pneumatic otoscopy can be a difficult skill that relies on the expertise of the operator; in 1 study, the clinical otoscopic skills of 23 general practitioners were found to be no better than those of inexperienced medical students.14 Pneumatic otoscopy use in clinical practice remains suboptimal, despite its generally acknowledged diagnostic value in AOM and guideline recommendation for diagnosis of OME.15 One survey of general pediatricians found that 42% do not use pneumatic otoscopy routinely, whereas only 21% always use pneumatic otoscopy.16 Pneumatic otoscopy among residents in our study was 79%. One explanation may be the emphasis placed on pneumatic otoscopy in our residency program or the Hawthorne effect of doing a study.
Some authors propose that otitis media management guidelines should emphasize implementing better pneumatic otoscopy training for clinicians to reduce the overdiagnosis and cost of AOM,8 including training for pediatric residents.17 There are several reports of otitis media education in otolaryngology training programs. Wormald et al18 reported an educational program in which 10 otolaryngology residents who were shown 30 otoscopic photographs and given structured examination instruction demonstrated an improvement in diagnosis with a fall in overall error rate from 57% to 31%. Silva and Hotaling19 described a 4-month clinical and didactic training program for 4 PGY-2 otolaryngology residents with a curriculum of standard readings; clinical training in a pediatric otolaryngology clinic; and pneumatic otoscopy, otomicroscopy, and myringotomy sequentially on patients scheduled for myringotomy and pressure equalizing tube placement. Kaleida and Stool20 later included private pediatricians and pediatric nurse practitioners with otolaryngology residents and pediatric otolaryngology fellows for certification as validated otoscopists for clinical research. Participants were asked to evaluate patients immediately before myringotomy on a weekly basis and required an average of 3.6 months to achieve a sensitivity of 80% and specificity of 70% accuracy.
Although some could argue that competencies can be measured without a formal curriculum, this assertion does not satisfy the need to verify adherence to a common set of minimum standards for skills and proficiencies.21 Studies have shown that trainee physicians develop their own ideas of which clinical signs and symptoms indicate the presence of AOM and that these ideas persist throughout the first 2 years of training. Rotations that emphasize skill instruction lead to learning with a short-term effect but often not long-term acquisition.22
Pediatric residency program directors emphasize a need for additional AOM training and competency evaluations and use numerous methods to meet their goals.5 Other aspects of pediatric residency training have been formally evaluated and reported,23,24 but only 1 previous published study investigated diagnostic accuracy of otitis media in residents.25 In that study, family practice residents examined patients followed by a faculty examiner, with both examinations compared with tympanometry. The authors found concordance between resident and faculty for all levels of trainees to be 77% and provider and faculty examinations concurring with tympanometry 61% of the time. The authors developed an algorithm for their residency program whereby tympanometry was used to determine which patients who were seen by the residents should be reexamined by faculty.
Although tympanocentesis is the accepted standard of diagnosing AOM for research, this was impractical in our setting, where we evaluated providers’ clinical interpretation of both diseased and normal ears. We instead focused on comparing the practical examinations of pediatric trainees with accepted experts and used the clinical diagnostic examination of the pediatric otolaryngologist as the standard. Limitations to our study include the widely known technical difficulties of using pneumatic otoscopy and tympanometry in young children.26 We attempted to mitigate this problem by comparing clinical examinations immediately after the initial examination to best reproduce the original symptoms and otoscopic findings. A possible variable in examination quality between the 2 clinical examinations is cerumen obstruction of the TM. The pediatric otolaryngologists were much better equipped with otomicroscopy and suction than the residents in the ACC. An additional limitation is that the 2 pediatric otolaryngologists’ clinical examinations were independent because only 1 attended clinic at any point in time, although they each demonstrated similar κ statistics compared with the residents.
Another limitation is the power of the study afforded by the sample size; not every patient seen in the ACC was enrolled for various reasons. First, because the study sought to analyze the skills of pediatric residents, only the patients seen by those providers were eligible. Second, in addition to the increased time burden for families, those patients with a diagnosis might have been hesitant to undergo additional evaluation of a disease that had already been diagnosed and possibly prescribed treatment. After enrollment, not every patient was seen by all 3 providers, which further limited sample size. This small sample size creates difficulty to make conclusions about individual PGY levels. Another limitation is that with the present study design, the potential underdiagnosis of AOM was not evaluated. The possibility of a certain number of children with MEE at any given time is also considered, but the lack of symptoms should distinguish these patients from those with AOM.
Finally, to complete κ statistic analysis, tympanometry results were collapsed into 2 simpler categories: flat (type B) and mobile (types A and C). However, one can have OME with a type C tympanogram when there is some fluid in the middle ear but most of the middle ear space is filled with large air bubbles. In addition, type C tympanograms are not necessarily normal and can have severe retractions.
The future of medical education will not be a question of the number of procedures a resident completed during his or her training but an assessment of the core competencies. This is the first study to analyze pediatric residents’ otoscopic examination skills. Recent reviews in primary care journals have emphasized the otoscopic examination for general practitioners,27,28 and several continuing medical education courses on otitis media diagnosis and treatment are available to practicing general pediatricians after residency.29 However, a recent survey demonstrated few formalized training curricula for this common pediatric disease during pediatric residency training.5 Medical students and residents today also have much more medical information to learn than years past, so outpatient entities such as otitis media now often fill a smaller portion of the training curriculum. More mandatory time must be spent on this common disease, both didactically and clinically, so that medical practitioners will be competent and accurate in the diagnosis of this important disease.30
With increased antibiotic resistance inherently linked to otitis media and the frequent misdiagnosis of otitis media recognized by experts as a culprit, it seems clear that the focus of educational efforts are best spent on primary prevention with formalized education during residency when trainees are developing these skills. We also encourage established practitioners to reexamine their competencies through periodic self-assessment and, if possible, hands-on skills training to hone their skills related to the diagnosis of AOM and OME to optimize the use of antibiotics in pediatric patients.
This study was supported by an American Academy of Pediatrics Resident Research Grant to Dr Steinbach, who is presently supported as an Aventis pediatric infectious diseases fellow.
We recognize the enthusiasm of the 2000–2001 Stanford University pediatric residents and the zeal of Carrie Loutit, MD, for their education.
- ↵Klein JO, McCracken GH. Introduction. Pediatr Infect Dis J.1998;1998 :539
- ↵Steinbach WJ, Sectish TC. Pediatric resident training in the diagnosis and treatment of acute otitis media. Pediatrics.2002;109 :404– 408
- ↵Bondy J, Berman S, Glazner J, Lezotte D. Direct expenditures related to otitis media diagnoses: extrapolations from a pediatric Medicaid cohort. Pediatrics.2000;105(6) . Available at: http://www.pediatrics.org/cgi/content/full/105/6/e72
- ↵Fisher EW, Pfleiderer AG. Assessment of otoscopic skills of general practitioners and medical students: is there room for improvement? Br J Gen Pract.1992;42 :65– 67
- ↵American Academy of Pediatrics, Otitis Media Guideline Panel. Managing otitis media with effusion in young children. Pediatrics.1994;94 :766– 773
- ↵Nelson J. Management of chronic suppurative otitis media: a survey of practicing pediatricians. Ann Otol Rhinol Laryngol.1988;97(suppl 131) :26– 28
- ↵Mulvey HJ, Ogle-Jewett EAB, Cheng TL, Johnson RL. Pediatric residency education. Pediatrics.2000;106 :323– 329
- ↵Gaskin PRA, Owens SE, Talner NS, Sanders SP, Li JS. Clinical auscultation skills in pediatric residents. Pediatrics.2000;105 :1184– 1187
- ↵Kaleida PH. The COMPLETES exam for otitis. Contemp Pediatr.1997;14 :93– 101
- ↵Brown DH. Education of medical practitioners in otitis media management. J Otolaryngol.1998;27(suppl 2) :49– 52
- Copyright © 2002 by the American Academy of Pediatrics