PEDIATRICS Vol. 108 No. 2 August 2001, pp. 248-254

Evidence Assessment of Management of Acute Otitis Media: II. Research Gaps and Priorities for Future Research

Linda S. Chan, PhD^*, , Glenn S. Takata, MD^{, §}, Paul Shekelle, MD, PhD^{, #}, Sally C. Morton, PhD^{#, ¶}, Wilbert Mason, MD^{, **}, and S. Michael Marcy, MD

From the ^* Division of Biostatistics and Outcomes Assessment, Los Angeles County-University of Southern California Medical Center, Los Angeles, California;  Center for Pediatric Health Outcomes Research, Department of Pediatrics, University of Southern California, Los Angeles, California; ^§ Division of General Pediatrics, Childrens Hospital Los Angeles, Los Angeles, California;  Health Services Research and Development Service, Greater Los Angeles Veterans Affair Healthcare System, Los Angeles, California; ^¶ Statistics Group, RAND, Santa Monica, California; ^# Southern California Evidence-Based Practice Center, RAND, Santa Monica, California; ^** Division of Infectious Diseases, Childrens Hospital Los Angeles, Los Angeles, California; and  Department of Pediatrics, Southern California Kaiser-Permanente Health Care Program, Panorama City, California.

	ABSTRACT

Top Abstract Methods Results Discussion Conclusion References

Objectives.  To report research gaps and priorities of future research identified during an evidence assessment process on the management of acute otitis media (AOM).

Methods.  A conceptual framework for management of AOM was developed to guide the evidence assessment. An 11-member technical expert panel guided the selection of key questions, prioritization of influencing factors, development of scope, definition of AOM, and search strategy through polling processes and conference calls. Quality of clinical trials was evaluated using established scales. Outcome measures were abstracted from each study.

Results.  A total of 3461 titles and abstracts were screened, and 760 full-length articles were reviewed. Of the 760 articles, 80 studies addressed the key questions. In defining AOM, 42 (52.5%) of the 80 studies included the middle-ear effusion component, only 2 (2.5%) included the rapid onset component, and 26 (32.5%) included the signs/symptoms of inflammation component. None of the 80 studies used all 3 components. Of the 74 controlled trials, 39 (53%) were of acceptable quality (Jadad score of 3 or higher). The technical experts did not agree in the ranking of the importance of the 41 influencing factors (Kendall's coefficient of concordance was 0.0022). Another poll also indicated diverse opinions of the experts on the importance of 7 key questions derived from the conceptual framework (Kendall coefficient of concordance is 0.21). Furthermore, our review found that the type and definition of outcome measure varied.

Conclusions.  Despite the large body of literature on AOM, its quality is uneven and its findings are not generalizable. Future research should try to answer all key questions and investigate all risk factors in well-designed, scientific studies.  Key words:  otitis media, children, future research.

The treatment of acute otitis media (AOM) requires the understanding and completion of several clinical processes before a successful outcome can be achieved. Although readily recognized as a clinical condition, clinicians and researchers do not agree on a standard definition of AOM, even among often-cited sources.^1-5 In addition, 40% of articles reviewed by Hayden⁶ did not describe the criteria by which the diagnosis of AOM was established. In the 26 articles that did describe criteria for the diagnosis of AOM, 18 unique criteria sets were found. Froom et al⁷ found that a large proportion of physicians in their study were unsure of the accuracy of their diagnosis of AOM. The diagnostic accuracy of otoscopy was revisited by Pelton,⁸ who found that the position and mobility of the tympanic membrane are highly specific and sensitive for signs of AOM but the color is nonspecific. Factors influencing the possible outcome of treatment for AOM are numerous,^9-11 but disagreement exists on their relative importance. Explicitly defined AOM outcomes are needed to aid in comparing and synthesizing study results. Explicit description of populations is needed to aid in assessing the generalizability of study findings.

This is the second of the 2-part evidence assessment report on the management of AOM. In Part I we reported the findings of the role of antibiotics in treatment of uncomplicated AOM. In Part II we report the research gaps and priorities of future research identified during this evidence assessment process.

	METHODS

Top Abstract Methods Results Discussion Conclusion References

Evidence Assessment Process Overview

An outline of the assessment process is displayed in Fig 1. A conceptual framework for the management of AOM was constructed to guide the assessment process (available on request). It outlined 11 steps for the management of AOM, with steps 1 through 6 related to initial evaluation and treatment and steps 7 through 11 related to follow-up care. Factors influencing outcomes, outcome measures, and factors influencing decisions were noted as footnotes to the framework. The framework formed the basis on which key questions and the scope of the evidence report were developed. The technical experts were polled 3 times on the selection of key questions, definition of AOM, and defining the scope of the assessment. Four conference calls were held among the technical experts and the project staff. The selection process and the qualifications of the 11 technical experts have been described in the first part of this 2-part manuscript.

View larger version (19K): [in this window] [in a new window]   Fig. 1.   The evidence assessment process. The arrows indicate the sequence of the steps.

Procedures to Reduce Bias, Enhance Consistency, and Check Accuracy

Throughout the evidence assessment process, procedures and measures were implemented to minimize biases. Two physician reviewers screened and reviewed titles and abstracts during the initial screening stage and full articles during the subsequent screening stage. Retrieved articles were cross-checked with studies used or cited in other meta-analyses and review articles. Data abstracted by the 2 physician reviewers were further cross-checked by a biostatistician. Predesigned forms with explicit instructions were used, and discrepancies were resolved continuously and promptly. In analysis, funnel plots were examined to assess the extent of publication bias. Criteria for assessing study quality were established before the review of articles. The criteria developed by Jadad et al¹² were used to evaluate the quality of clinical controlled trials. The Jadad score had a range of 0 to 5. For a given study, the article was awarded 1 point if it was randomized, 1 point if it was double-blind, and 1 point if it described withdrawals and dropouts. An extra point was awarded if the method of randomization and/or double-blinding was appropriate; conversely, 1 point was subtracted if the method was inappropriate. The criteria used to evaluate the quality of cohort studies and case-control studies were based on the work by the McMaster University Group.^13-15 The quality of cohort studies was evaluated against 8 components, which included presence or absence of a clear definition of the study cohort, an early inception point, a clear pathway of patient entry, complete follow-up, description of dropouts, objective outcome criteria, blind outcome assessment, and adjustment for extraneous factors.

Information Synthesis

In preparation for supplemental analyses, a poll was taken among the technical experts to prioritize the 41 influencing factors identified during the process that might potentially affect the outcome of the course of AOM, treated or untreated. The technical experts ranked the top 10 influencing factors using the scale of 1 to 10, 10 being the most important influencing factor that should be used to further stratify supplemental analyses.

Meta-analyses were performed to determine the effectiveness of antibiotics versus placebo or observational treatment of uncomplicated AOM, and to determine the effectiveness of particular antibiotic regimens. Comparisons based on the type of antibiotic and the outcome variable under consideration were established a priori. A meta-analysis was performed for each comparison with 3 or more randomized controlled studies. Heterogeneity was measured for all meta-analyses, and a random effects model was used to estimate the absolute rate differences.

	RESULTS

Top Abstract Methods Results Discussion Conclusion References

A total of 3461 titles and abstracts were screened and 760 full-length articles were reviewed. Combining all sources, including the electronic databases, searches of reference lists, and peer reviewers' comments, a total of 80 studies in 85 articles were included in answering the 3 key questions. The evidence assessments for the 3 key questions are reported in Part I.

Selection of Key Questions

Based on the conceptual framework, 7 major components constituting the care of patients with AOM were identified. These 7 major components were addressed in terms of 7 key questions. Table 1 presents the original 7 questions derived from the conceptual framework, and Table 2 provides a summary of the rankings by the 11 technical experts in an attempt to select the key questions to be addressed in this evidence assessment. An analysis of the rankings showed that despite a marginal agreement among the experts (Kendall coefficient of concordance is 0.21, P = .03), opinions of the experts on the importance of the key questions were diverse. The experts strongly agreed on the importance of the questions on antibiotic treatment and natural history. They agreed that the question of nonantibiotic treatment was least important to them. Their opinions varied with respect to the other questions. The scoring of the question on definition of AOM and on diagnosis was particularly polarized with 4 technical experts ranking it 4 or 5, and 4 technical experts not ranking it at all. Because of the 1-year time constraint for the project, the evidence assessment focused on the top 3 ranking questions dealing with evaluating the effectiveness of antibiotic treatment.

Definition and Diagnostic Criteria for AOM

A short and a long version of the definition of AOM were developed based on the consensus of the technical experts. The short version states: "Acute otitis media is the presence of middle-ear effusion (MEE) in conjunction with the rapid onset of one or more signs or symptoms of inflammation of the middle ear." And the long version states: "1) Presence of middle-ear effusion as demonstrated by the actual presence of fluid in the middle ear as diagnosed by tympanocentesis or the physical presence of liquid in the external ear canal as a result of tympanic membrane perforation or indicated by limited or absent mobility of the tympanic membrane as diagnosed by pneumatic otoscopy, tympanogram, or acoustic reflectometry with or without the following: (a) opacification, not including erythema, (b) a full or bulging tympanic membrane, or (c) hearing loss, and 2) rapid onset (ie, up to 48 hours from the onset of acute signs or symptoms first noted by the parent or guardian to the time of contact with the health system) of one or more of the following signs or symptoms, with or without anorexia, nausea, or vomiting: (a) otalgia (or pulling of ear in an infant), (b) otorrhea, and (c) irritability in the infant or toddler, or (d) fever."

The definition contains 3 components: presence of MEE, rapid onset, and 1 or more signs or symptoms of inflammation. As specified by this definition, none of the 80 studies used all 3 components whereas 18 (22.5%) used 2, 34 (42.5%) used only 1, and 28 (35%) used none. Of the 80 studies, 42 (52.5%) included the MEE component, only 2 (2.5%) included the rapid-onset component, and 26 (32.5%) included the signs/symptoms of inflammation component. We had planned sensitivity analysis to evaluate the influence of the approximation of the study definition of AOM to that of our technical expert panel, but this was not possible because of the small number of studies eligible for inclusion in each comparison.

Influencing Factors

Table 3 presents the total rank score of the 11 technical experts on the importance of 41 factors on outcome of AOM. The Kendall's coefficient of concordance was 0.0022, P = .99, indicating a lack of consensus in the ranking of the importance of the risk factors among the 11 technical experts. For our evidence assessment, the 2 top ranking risk factors were planned for subgroup analysis: age of the child and otitis-prone status. In the description of the study population, 29 (36%) of the 80 studies mentioned these 2 factors, 47 (59%) mentioned only the age but not otitis-prone status, and 4 (5%) did not mention these 2 factors. Despite the planned effort, no subgroup or sensitivity analyses were possible because of an insufficient number of studies presenting AOM outcomes stratified by age or otitis-prone status. We had also planned sensitivity analysis to evaluate the influence of language, time, and place. None of these sensitivity analyses were possible because of the small number of studies eligible for inclusion in each comparison.

Outcome Measures

Several issues were brought up during the discussions related to identification and definition of common outcome measures. One such issue related to the definition of "natural history." Because of the lack of true longitudinal natural history cohort studies, patients in the placebo arms of randomized, controlled trials and those receiving observational treatment prescribed by the health care provider were included as natural history cohorts. Even with such relaxation of the definition for natural history studies, only 15 studies met our screening and inclusion criteria. Furthermore, the study populations and outcome measures varied. Only 5 studies reported the clinical failure rate between 1 to 7 days. Few studies reported on clinical failure, pain, fever, and MEE by age, and none of them reported these outcomes by otitis-prone status. Nine studies provided data on the reported occurrence of mastoiditis and other suppurative complications in children who were not initially treated with antibiotics for AOM.

Although the placebo or observational groups were not initially treated with antibiotics, all 9 of the randomized controlled trials and 2 of the 6 cohort studies mentioned clinical circumstances that would allow the investigators to administer antibiotics to children in the placebo or observational groups. Persistent symptoms or complications were common reasons for giving antibiotics to children in the placebo or observational groups.

Another issue of discussion related to the definition of an initial episode of AOM. An episode was considered initial if it was "separated by at least 4 weeks from the end of an antibiotic course of treatment for the last episode of AOM." Of the 74 randomized controlled trials assessed, 21 (28%) did not address this issue. All of the 6 (100%) cohort studies assessed did not address this issue. Of the 53 studies that addressed time of separation from the initial AOM episode, 17 (32%) met the criterion of 4 weeks or greater. None required that the last episode of AOM be documented as having resolved.

In general, we found that the type of outcome measure varied between studies, and the definitions of common outcomes, such as clinical failure, were not uniform. Studies concentrated primarily on short-term rather than long-term outcomes.

Study Quality

Of the 80 studies included in the assessment, 74 were randomized controlled trials. Of the 74 studies, 6 (8%) had the highest score, which is 5 on the Jadad scale, 12 (16%) scored 4, 21 (28%) scored 3, 26 (35%) scored 2, 8 (11%) scored 1, and 1 (1%) scored 0. Of the 74 randomized controlled studies, 30 (41%) mentioned double-blinding, 52 (70%) described the characteristics of the dropouts, 28 (38%) used appropriate randomization methods, and 17 (23%) used appropriate blinding strategies. Of the 6 observational cohort studies, 2 addressed 4 of the 8 quality components, 1 study addressed 3, 2 studies addressed 2, and 1 study addressed 1. We had planned sensitivity analysis to evaluate the influence of study-design quality but were unable to do so because of the small number of studies eligible for inclusion in each comparison.

	DISCUSSION

Top Abstract Methods Results Discussion Conclusion References

Despite the many attempts to identify the optimal use of antibiotics for AOM, including this large scale evidence assessment, there is still a woeful lack of substantial evidence on the question of antibiotic therapy in AOM. Despite the many efforts to address the key components for conducting research studies on AOM, such as defining and diagnosing AOM, identification of risk factors, and defining outcome measures, significant research gaps remain.

For the purpose of systematic review for this evidence-based analysis, we required a consensus among the technical experts on the definition of AOM. A well-structured definition would provide a framework on which we could develop diagnostic criteria to diagnose the presence of AOM in individual patients. For this systematic review, a 3-component definition of AOM was developed. However, none of the 80 studies included in our review used all 3 components of the definition as defined by our technical experts. Only half of the studies approximated the diagnostic criteria for MEE, almost none met the criteria for rapid onset, and one-third met the criteria for signs and symptoms of inflammation referable to the middle ear as defined by our technical experts in the definition of AOM. This systematic review identified the lack of consensus on the diagnostic criteria for AOM. Adherence to specific criteria for the diagnosis of AOM is crucial to accurately gauge the impact of antimicrobial agents on the outcome. Misclassification of disease (eg, otitis media with effusion as AOM) may bias the results of trials against a beneficial effect of therapy if misclassification is extensive. Such a lack of uniformity in these studies, whether in agreement or disagreement with our technical expert definition of AOM, suggests the need for a consensus definition of AOM and its diagnostic criteria to enhance the validity and generalizability of future research efforts.

From the literature, 41 factors that could affect the outcome of treatment for AOM were identified. However, the concordance in the ranking of the top 10 factors among the 11 technical experts was found to be close to 0 (0.0022). Such a diversity of opinion on this issue provides an explanation of the diversity of research studies reporting AOM treatment outcomes. Thus, even for the influencing factors age and otitis-prone status for which there seemed general agreement by the technical experts on their importance, our planned effort to conduct subgroup analysis had to be aborted because of an insufficient number of studies addressing these factors for our planned comparisons. We suggest not only more empirical work in the influence of factors other than treatment on AOM outcomes, but also a structured conceptual framework to provide a rational approach to future research

The definition of endpoints for outcomes has been a long standing issue in the study of AOM.^4,5,10,16,17 Our technical experts recommended leaving the definitions open to ones used by different studies. This evidence assessment found that the type of outcome measure varied between studies and the definition of common outcomes, such as clinical failure, were not uniform. These inconsistencies made it difficult not only to combine results across studies, but also to conduct analysis for subgroups of patients. We encourage the adoption of common outcome definitions to increase the comparability of future research outcomes.

Although all of the studies included in the meta-analyses were randomized, many studies did not adhere to protocols that would strengthen the internal validity of the findings. Because many of the studies did not report their procedures for randomization, it was difficult to determine if the randomization process was appropriate in those trials. A large number of studies were also not double-blind. About half of the studies were not of adequate quality based on the Jadad scale. Studies of poor quality, particularly those with inadequate randomization procedures or those not double-blind, may inflate estimates of treatment benefit.^18,19 The general quality of AOM studies must improve to insure the internal validity and generalizability of future randomized, controlled trials.

In summary, future research should 1) establish uniform definitions of AOM and relevant outcomes; 2) establish uniform diagnostic criteria for AOM; 3) work from testable conceptual frameworks; 4) address relevant influencing factors, such as age and otitis-prone state; and 5) strengthen internal and external validity. In addition, we encourage the measurement of long-term as well as short-term outcomes. Although not addressed in this evidence-based assessment, future research should more rigorously consider bacterial resistance as a factor influencing outcome and as an outcome itself. Investigators should prioritize AOM outcomes based, in large part, on their importance to patients, their families, and society as a whole since they bear the burden of AOM. Large-scale, randomized, controlled trials with sufficient patient variation for subgroup analysis are needed. Close monitoring of patients with a priori plans for appropriate intervention should allay concerns about suppurative complications in the case of placebo studies and should also be a focus of research.

	CONCLUSION

Top Abstract Methods Results Discussion Conclusion References

We have presented here not only a method, but also an outline, to guide future research goals. The rankings may not be important outside the context of this assessment. Although because of the time constraints of this particular evidence-based assessment, only the role of antibiotics in AOM was addressed; we feel that all 7 of the original candidate key questions should be investigated, never mind in what order, with future research. We should look at all the influencing factors, never mind in what order, with future research. Our plea is that these key questions and influencing factors be studied in a well-designed, scientific manner.

	ACKNOWLEDGMENTS

We thank our project staff, partners, technical experts, and peer reviewers for their tireless efforts and superb contributions and the Agency for Healthcare Research and Quality for their technical support and for funding this project.

	FOOTNOTES

This article is based on research conducted by the Southern California Evidence Based Practice Center under contract with the Agency for Healthcare Research and Quality (AHRQ, Contract No. 290-97-0001). The authors of this article are responsible for its contents, including any clinical or treatment recommendations. No statement in this article should be construed as an official position of the Agency for Healthcare Research and Quality or the US Department of Health and Human Services.

Received for publication Aug 22, 2000; accepted Nov 21, 2000.

Reprint requests to (L.S.C.) LAC+USC Medical Center, Room 12-900, 1200 North State St, Los Angeles, CA 90033. E-mail: lschan{at}hsc.usc.edu

	ABBREVIATIONS

AOM, acute otitis media; MEE, middle-ear effusion.

	REFERENCES

Top Abstract Methods Results Discussion Conclusion References

1. Paparella MM, Bluestone CD, Arnold W, 1A. Definition and classification. Ann Otol Rhinol Laryngol 1985; 94:8-9
2. Senturia BH, Paparella MM, Lowery HW, Definition and classification. Ann Otol Rhinol Laryngol 1980; 89:4-8
3. Chow AW, Hall CB, Klein JO, General guidelines for the evaluation of new anti-infective drugs for the treatment of respiratory tract infections. Clin Infect Dis 1992; 15:S62-S88
4. Stool SE, Berg AO, Berman S, et al. Otitis media with effusion in young children. Clinical Practice Guideline, Number 12. Rockville, MD: Agency for Health Care Policy and Research, Public Health Service, US Department of Health and Human Services. July 1994a. AHCPR Publication No. 94-0622
5. Bluestone CD, Klein JO. Otitis media, atelectasis, and eustachian tube dysfunction. In: Bluestone CD, Stool SE, Kenna M, eds. Pediatric Otolaryngology. Philadelphia, PA: WB Saunders Company; 1996:388-582
6. Hayden GF Acute suppurative otitis media in children: diversity of clinical diagnostic criteria. Clin Pediatr 1980; 20:99-104
7. Froom J, Culpepper L, Grob P, Diagnosis and antibiotic treatment of acute otitis media: report from International Primary Care Network. Br Med J 1990; 300:582-586
8. Pelton SI Otoscopy for the diagnosis of otitis media. Pediatr Infect Dis J 1998; 17:540-543 [CrossRef][Medline]
9. Uhari M, Mantysaari K, Niemela M A meta-analytic review of the risk factors for acute otitis media. Clin Infect Dis 1996; 22:1079-1083 [Medline]
10. Rosenfeld RM An evidence-based approach to treating otitis media. Pediatr Clin North Am 1996; 43:1165-1181 [CrossRef][Medline]
11. Froom J, Culpepper L, Bridges-Webb C, Effect of patient characteristics and disease manifestation on the outcome of acute otitis media at 2 months. Arch Fam Med 1993; 2:841-846 [Abstract]
12. Jadad AR, Moore AR, Carroll D, Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996; 17:1-12 [CrossRef][Medline]
13. Sackett DL How to read clinical journals. V: To distinguish useful from useless or even harmful therapy. Can Med Assoc J 1981; 124:1156-1162 [Medline]
14. Trout KS How to read clinical journals: IV. To determine etiology or causation. Can Med Assoc J 1981; 124:985-990 [Medline]
15. Tugwell P How to read clinical journals: III. To learn the clinical course and prognosis of disease. Can Med Assoc J 1981; 124:869-872 [Medline]
16. Dowell SF, Marcy SM, Phillips WR, Principles of Judicious Use of Antimicrobial Agents for Pediatric Upper Respiratory Tract Infections. Pediatrics 1998; 101:163-165 [Abstract/Free Full Text]
17. Culpepper L, Froom J Routine antimicrobial treatment of acute otitis media: is it necessary. JAMA 1997; 278:1643-1645 [CrossRef][Medline]
18. Moher D, Jones A, Cook DJ, Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352:609-613 [CrossRef][Medline]
19. Schulz KF, Chalmers I, Hayes RJ, Altman DG Empirical evidence of bias dimensions of methodological quality associated with estimates of treatment effects in controlled trials. JAMA 1995; 273:408-412 [Abstract]