OBJECTIVE. Anticipatory guidance is a cornerstone of modern pediatric practice. In recognition of its importance for child well being, injury prevention counseling is a standard element of that guidance. Over the last 20 years, there has been growing recognition that intentional injury or violence is one of the leading causes of morbidity and mortality among youth. The US Surgeon General identified youth violence as a major public health issue and a top priority. Yet, only recently has the scope of injury prevention counseling been expanded to include violence. Pediatric health care providers agree that youth violence–prevention counseling should be provided, yet the number of topics available, the already lengthy list of other anticipatory guidance topics to be covered, developmental considerations, and the evidence base make the selection of an agreed-on set a considerable challenge. The purpose of this study was to systematically identify and prioritize specific counseling topics in violence prevention that could be integrated into anticipatory guidance best practice.
DESIGN. A modified electronic Delphi process was used to gain consensus among 50 national multidisciplinary violence-prevention experts. Participants were unaware of other participants' identities.
METHODS. The process consisted of 4 serial rounds of inquiry beginning with a broad open-ended format for the generation of anticipatory guidance and screening topics across 5 age groups (infant, toddler, school age, adolescent, and all ages). Each subsequent round narrowed the list of topics toward the development of a manageable set of essential topics for screening and counseling about positive youth development and violence prevention.
RESULTS. Forty-seven unique topics were identified, spanning birth to age 21 years. Topics cover 4 broad categories (building blocks): physical safety, parent centered, child centered, and community connection. Participants placed topics into their developmentally appropriate visit-based schedule and made suggestions for an appropriate topic reinforcement schedule. The resulting schedule provides topics for introduction and reinforcement at each visit.
CONCLUSIONS. The Delphi technique proved a useful approach for accessing expert opinion, for analyzing and synthesizing results, for achieving consensus, and for setting priorities among the numerous anticipatory guidance and assessment topics relevant for raising resilient, violence-free youth.
The American Academy of Pediatrics (AAP) logo proclaims the commitment: “dedicated to the health of all children.” Increasingly, the greatest threats to child health are part of what has been called the “new morbidity.”1 More than a decade ago, the AAP's Committee on Psychosocial Aspects of Child and Family Health approved a policy statement advocating better preparation of pediatricians to address the behavioral and psychosocial factors that threatened child health with increasing prevalence.1 More recently, the AAP's commitment has been reaffirmed.2 In recognition of the newer morbidities, such as violence, suicide, substance abuse, and school problems, the committee concluded, “These are the morbidities that place our patients at risk. In other words, after infancy, children in the United States are more likely to die from injuries or violence and suicide than from infectious disease.”2
Increasingly, pediatricians are called on to address these new morbidities, effectively and efficiently, not only through diagnosis, treatment, and referral, but through prevention as well. Preventive care is a cornerstone of pediatric practice, but the new morbidities require new approaches from physicians: “To effectively address these new morbidities, pediatricians will need a model that encompasses expanded areas of competence in child behavior, development, and family function.”2
Violence ranks highly among the clear threats to child health. The World Health Organization operationally defined violence as “the intentional use of physical force or power, threatened or actual, against oneself, another person, or against a group or community, that either results in or has a high likelihood of resulting in injury, death, psychological harm, maldevelopment or deprivation.”3 Numerous statistics speak to the severity and prevalence of the problem, particularly in the United States.4 The US Surgeon General identified youth violence as a major public health issue and a top priority.5
In response to this threat to child health, the AAP's Task Force on Violence (TFOV) was charged with developing policy for the AAP regarding the pediatrician's role in the prevention of youth violence. The 10-member group developed a set of recommendations regarding clinical practice, advocacy, education, and research. Among the recommendations, clinical practice should include “violence-prevention counseling and screening as early as the pediatric prenatal visit and continuing into adulthood.”6
The TFOV commissioned an AAP periodic survey (No. 38) to solicit information on youth violence from the membership7; many questions from this survey were repeated in 2003.8 The results from these nationally representative samples underscored both pediatricians' awareness of the issue, as well as barriers to addressing it. More than 40% of pediatricians (44%) involved in direct patient care reported treating children with injuries resulting from community violence during the preceding 12 months. More than 70% of pediatricians believed that community-based programs could be effective, that pediatricians should be involved, and that they should screen for community violence (76%, 72%, and 71%, respectively).9
Yet, recognition of the problem and beliefs in efficacy were not enough. Most respondents did not feel confident in their abilities to identify children at risk for community violence (65%), did not feel they had adequate training in community violence (79%), and were not comfortable discussing these issues with parents (58%). Between 1998 and 2003, physician interest in incorporating violence prevention into routine care grew. The task force outlined ideas for the components to be included in a Violence Intervention and Prevention Program (VIPP), an anticipatory guidance program, if funding is secured for its development, implementation, and evaluation.6
Funded by the US Department of Justice Office of Juvenile Justice and Delinquency Prevention, VIPP was a 3-year project designed to develop systematically a program for pediatricians to incorporate the prevention of youth violence into routine health care. VIPP sought to develop a violence-prevention protocol for use in pediatric practices based on the knowledge, attitudes, and beliefs of 3 distinct groups of stakeholders: patients and their families, pediatricians, and public health experts. The central hypothesis underlying VIPP is that physicians, through assessment, counseling, and referral during routine health maintenance visits, can both promote positive youth development while they reduce injuries caused by interpersonal violence.
Although there is general acceptance that pediatricians must address injury prevention topics during health maintenance visits, it is also well established that there are numerous topics to cover, and not all of the topics are equally relevant at all stages of a child's development. Although the intent of the AAP directive is easy to accept, how best to translate that directive into practice is far more difficult. Priorities needed to be set. Similarly, developmentally appropriate guidance as to which topics are best discussed during which visits also needed to be established.
Previous work has demonstrated that written materials, reinforced by physician counseling, can be powerful tools for change in parent home literacy activities and childhood injury prevention behaviors.9,10 However, if the topics covered and the materials developed are not sufficiently compelling, then even demonstrated changes in practice do not lead to changes in family or child outcomes.11 In this particular field, earlier efforts at skills-based anticipatory guidance have demonstrated effects, particularly including the promotion of alternatives to corporal punishment.12 In a recent review, Glascoe et al13 demonstrated that the strategic use of physician counseling in the context of health care visits may be effective in leading to behavior change, particularly if the counseling addresses issues of concern to parents. Verbal counseling accompanied by personalized written information seemed to be most effective.
This article describes the results of a systematic approach used to obtain expert opinion to support the development of anticipatory guidance for the VIPP. The expert consensus described here was combined with other information14 to create the VIPP program titled “Connected Kids: Safe, Strong and Secure.”15 The complete package consists of parent and adolescent educational brochures, a clinical guide for pediatricians, a Web site with supporting literature, and supporting training materials available on the Internet.
The Delphi technique was first developed in the 1950s by scientists at the Rand Corporation for making informed decisions based on expert opinion. The technique has been modified over the years, but at its heart it remains an approach for gathering expert opinions, summarizing, and then synthesizing those opinions to reach consensus through a structured iterative process. One of the strengths of the technique is its ability to overcome some of the biases that may arise because of group dynamics, including personality, interpersonal dynamics, and dominance by key opinion leaders. A modified Delphi process, described in more detail below, was used to overcome some of these potential challenges to developing the guidelines with attention to priority topic areas, developmental timing, and evidence-based practice.
The Delphi technique has been used extensively in a variety of areas. With special relevance for the VIPP effort, a similar approach was used successfully by Cohen et al11 to assess consensus regarding the prioritization of prevention strategies for office-based injury prevention counseling for parents of children from birth to 2 years.
The Delphi method is a structured process for collecting and distilling knowledge from a group of experts through a series of questionnaires.16 As noted earlier, the process is particularly helpful in facilitating interactions among experts and for avoiding the barriers observed frequently in more traditional group discussions.17 The method allows discussion without the influence of social status, permits participants to change their minds without “losing face,” and provides an efficient and parsimonious approach to combine many views into a single set of criteria in a reasonable amount of time.
The VIPP development effort used a modified Delphi process. The first round consisted of a Web-based survey administered to a carefully selected group of experts, described below. The initial survey used an open-ended format to elicit a wide range of anticipatory guidance and assessment topics across 5 age groups (infant, toddler, school age, adolescent, and all ages). The results of each round were compiled, summarized, and distributed back to the original contributors. Participants were invited to refine the topics by adding clarification, as well as by identifying strengths and weaknesses of the offerings. Through a series of 3 subsequent rounds, the list of topics was pared down to a manageable set of core topics, by consensus, by combining some topics and eliminating others.
Participant selection is a critical element of successful Delphi processes. The VIPP project team developed an initial list of organizations and individuals based on expertise, the diversity of the target audiences, and the earlier work of the AAP's TFOV. Through this process, a list of 20 national groups was generated along with key contacts in each. Senior VIPP staff made direct contact and explained the purpose of the effort, as well as what involvement would entail. They asked each contact if they would be available to participate or to identify a suitable representative for their group or organization.
In addition to representatives of key stakeholder organizations and knowledgeable experts, each relevant AAP committee was asked to appoint a participant to represent their area of expertise and interest. In all, 50 experts agreed to participate.
During the first round, each expert was asked to suggest topics to be included in the new AAP violence-prevention program, topics that would become part of routine anticipatory guidance and health screening during well-child visits. No formal definition of “violence” was offered so as not to limit unnecessarily the topics generated for initial consideration. Participants were asked to list topics based on their appropriateness at different ages: infant (newborn to 1 year), toddler (1–4 years), school age (5–11 years), and adolescence (≥12 years). In addition, participants were asked to list topics that would be suitable at every visit, regardless of the child's age. Responses were compiled, and only clearly duplicate topics for each age group were eliminated. If there was any doubt as to meaning, all of the original wordings were maintained to avoid subjective judgments and possible misinterpretations.
During the second round, the VIPP team organized the results to provide more structure, consistent with the intent of the process. Within each age group, lists of anticipatory guidance and screening topics were further categorized as (1) topics pertaining to the child, (2) topics pertaining to the parent/caregiver, and (3) topics pertaining to the child's environment. Although the environment rubric was useful for the conceptual exercise, it was not formally incorporated into the final results, which were practically categorized by intended audience. Participants were asked to rate each topic on 4 dimensions using a subjective 4-point ordinal scale: (1) evidence: how strong is the evidence base demonstrating that screening and/or anticipatory guidance on this topic will have positive outcomes for children?; (2) effect size: what is the likely effect size associated with the outcome?; (3) feasibility: how feasible would it be for pediatricians to address this issue with their patients during health maintenance visits?; and (4) importance: how important is it to include this topic in this protocol?
For each topic, a rough but useful overall score was calculated by summing the ratings for evidence, effect size, and feasibility. The resulting score was then weighted by the level of importance that was assigned to the topic. The top scoring items were presented in the subsequent round.
For the third round, the top scoring items were broken down by age group and whether they constituted “anticipatory guidance” or “screening.” For this round, participants were asked to rank each item in order of importance for inclusion. Responses were again compiled, and the top scoring items were then presented in the subsequent and final round.
The top scoring items from round 3 were placed into a sample protocol modeled after the protocol used in the Injury Prevention Program, a well-known and widely used program for unintentional injury prevention. Participants were asked to comment on the flow of topics and to note where items needed to be rearranged. The experts were also asked to review items from round 3 that did not make the final cut and which they believed should be restored to the list, along with a justification for making the change. Finally, participants were asked to review and comment on the potential effectiveness of an “existing” adolescent protocol, the American Medical Association's Guidelines for Adolescent Preventive Services.18 In the section that follows, we report the results of this iterative process.
As will become apparent in the next section, the number of participants changed from round to round. The overall success of the consensus process relied in part on participants' willingness to remain engaged. Given logistic and time constraints, not all of the participants participated in all of the Delphi rounds. However, even if they missed a round, all of the participants were invited to review and contribute to subsequent rounds. In this way, the fundamental integrity of the process was preserved, because each participant had the opportunity to challenge or augment previous contributions, even if they themselves missed a round.
Table 1 presents the progression of contents through the 4 rounds of the Delphi process, summarizing the number of topics initially generated and then surviving through the subsequent rounds. The topics are presented within the same framework used with the expert respondents. Each topic is listed by age group and the type of physician activity involved (ie, anticipatory guidance or screening).
Even a cursory review of Table 1 shows the narrowing of focus achieved through each successive round of the Delphi survey. Although rare, there are even 2 instances where the number of topics increased: screening of school age children between rounds 1 and 2, and screening for children of all ages for the same 2 rounds. In large measure, the increase represents a recategorization of some topics as screens rather than as anticipatory guidance. The increase may also reflect a slight widening consideration of relevant content after the experts reviewed the results of the first open-ended survey.
During the earlier Delphi rounds, we erred on the side of inclusion, retaining some topics that might seem to be redundant, lest too early synthesis eliminate shades of meaning and distinctions that might prove important later on. Over time, however, once the preliminary results were shared with our expert panel, we were more confident in combining similar topics and in making more parsimonious categorizations. These recategorizations, coupled with expert ratings of the proposed topics, resulted in the final set of topics.
During the first round, 34 respondents generated a total of 233 anticipatory guidance and 211 screening topics. The results from round 1 were compiled, categorized, and distributed to the Delphi experts. During this second round, 28 participants rated each of the anticipatory guidance topics and the screening topics. As noted earlier, the topics were grouped by vector (child, parent/caregiver, or environment), and participants were asked to rate each topic on 4 dimensions: evidence base, effect size, feasibility, and overall importance for inclusion. For each topic, an overall score was calculated and then weighted by the level of importance it was assigned. Table 2 summarizes the topics that made it to Round 3.
In the third round, participants were asked to rank order each topic according to how important it would be to include in the final counseling schedule. The results of round 3 were compiled, and the topics judged to be most important were placed into a counseling schedule modeled after the Injury Prevention Program. Also in round 3, topics appropriate for adolescents themselves, as well as complementary guidance designed for parents of adolescents, were articulated. This schedule was presented to experts in the fourth Delphi survey. Participants were asked to comment on the flow of the topics and to comment on those topics that were included and excluded from the schedule.
The Delphi results were vetted with an expert advisory committee that also considered results from parent and practitioner focus groups15 and were used to develop a preliminary counseling schedule. Parent and adolescent education materials and clinic guides were developed to introduce and reinforce various topics developmentally.15
The Delphi technique proved a useful approach for accessing expert opinion, for analyzing and synthesizing results, for achieving consensus, and for setting priorities among the numerous anticipatory guidance and assessment topics relevant for raising resilient, violence-free youth. The amount of time required of our participating experts was considerable, especially during the more open-ended early rounds of the process. We are grateful for their expertise, their patience, and their effort.
Any structured communication method has limitations, and the Delphi method is no exception. Perhaps chief among these limitations is how dependent the process is on the composition and active engagement of the expert panel. Although we made considerable effort to be inclusive and diverse in the identification of key stakeholder groups and in the initial identification and recruitment of expert participants, in the end, the results of the iterative process depend on the initial pool of topics generated. To the extent that important perspectives may have been overlooked, either for systematic or less obvious reasons, then the end result of the process may be incomplete.
Another critique of the method is whether experts were willing to engage actively over controversial topics to reach consensus or whether the more controversial topics dropped out because agreement could not be reached. With respect to violence prevention, we observed that the controversy on topics such as suicide, firearms, bullying, and corporal punishment is more publicly apparent than it is among the experts. Even under the veil of anonymity, our panel had no difficulty grappling with these topics head on. Subsequent vetting of the clinical guidelines and materials with physician focus groups revealed no general disagreement with even the most publicly controversial content.14
As intended, the method enabled the VIPP team to work with professionals from across the United States in an efficient time-limited manner, reducing the risk of bias introduced by intrapersonal and interpersonal dynamics. Subsequent focus group results obtained from physicians, parents, and teens, as well as preliminary results from the field tests of the final educational materials, all support the efficacy of this approach toward development (unpublished data, 2005).14
In this article we have described the systematic development of one component required to develop effective anticipatory guidance: an expert consensus on the content most likely to effectively reduce violence. This information, combined with input from practitioners and families, was used to develop a new comprehensive approach to violence prevention. Well-meaning advice developed without these key inputs may be less likely to be effective. Although it has been unusual to expend the time and resources to gather these inputs before developing a new product, the sensitivity of the topic and the extraordinary potential implicit in the relationships between families and their children's health care providers suggest that this investment will be worthwhile. Future research may help determine the relationship between the predicted efficacy of these materials and the actual experience of providers and families who will use them.
This research-and-development project was supported by a grant from the US Department of Justice's Office of Juvenile Justice and Delinquency Prevention to the AAP.
We want to acknowledge the following individuals who agreed to participate in the Delphi process (the listing of participants' names does not imply their endorsement of the data or conclusions): Shari Barkin, MD, FAAP; Miriam Bar-on, MD, FAAP; Lisa Cohen Barrios, DPH; Carl Bell, MD; Wyndolyn Bell, MD, FAAP; Iris Wagman Borowsky, MD, FAAP; Diane Bridgeman, PhD; Stephanie Bryn, MPH; Katherine Kaufer Christoffel, MD, MPH, FAAP; George Cohen, MD, FAAP; Larry Cohen, MSW; Angela Diaz, MD, FAAP; Denise Dowd, MD, MPH, FAAP; Jacqueline Gentry, PhD; Edward Goldson, MD, FAAP; Peter Gorski, MDA, MPA, FAAP; Rodney Hammond, PhD; Gilbert Handal, MD, FAAP; Christian Hanna, MPH; Steven Kairys, MD, MPH, FAAP; Murray Katcher, MD, PhD, FAAP, Jane Knapp, MD, FAAP; Lyndee Knox, PhD; Victor LaCerva, MD, FAAP; Phillip Leaf, PhD; William Modzeleski, MPA; Luis Montes, MD, FAAP; Robert Pallay, MD; Deborah Prothrow-Stith, MD; Frederick Rivara, MD, MPH, FAAP; Susan Rosenthal, MD, FAAP; Peter Scheidt, MD, MPH, FAAP; Roland Smart; Peter Stringham, MD; Jack Swanson, MD, FAAP; Howard Taras, MD, FAAP; Patrick Tolan, PhD; Evelyn Tomaszevski, ACSW; Daniel Webster, ScD, MPH; Constance Wieman, PhD; Elaine Williams, RN, NP; Janet Williams, MD, FAAP; Linda Wolfe, RN, BSN, MEd, CSN; Joseph Wright, MD, MPH, FAAP; and Thomas Young, MD, FAAP.
- Accepted April 28, 2006.
- Address correspondence to Edward De Vos, EdD, Pediatric and Adolescent Health Research Center, Tufts-New England Medical Center, 750 Washington St, NEMC Box 531, Boston, MA 02111. E-mail:
The authors have indicated they have no financial relationships relevant to this article to disclose.
- ↵American Academy of Pediatrics, Committee on Psychosocial Aspects of Child and Family Health. The Pediatrician and the “new morbidity.” Pediatrics.1993;92 :731– 733
- ↵American Academy of Pediatrics, Committee on Psychosocial Aspects of Child and Family Health. The new morbidity revisited: a renewed commitment to the psychosocial aspects of pediatric care. Pediatrics.2001;108 :1227– 1230
- ↵World Health Organization Global Consultation on Violence and Health. Violence: A Public Health Priority. Geneva, Switzerland: World Health Organization; 1996. Document WHO/EHA/SPI.POA.2, as reported in: Krug EG, Dahlberg LL, Mercy JA, Zwi AB, Lozano R, eds. World Report on Violence and Health. Geneva, Switzerland: World Health Organization; 2002
- ↵Centers for Disease Control and Prevention, National Center for Injury Prevention and Control. Youth violence: fact sheet. Available at: www.cdc.gov/ncipc/factsheets/yvfacts.htm. Accessed August 4, 2006
- ↵US Public Health Service, Office of the Surgeon General. Youth Violence: A Report of the Surgeon General. Washington, DC: US Public Health Service, Office of the Surgeon General; 2001
- ↵American Academy of Pediatrics, Task Force on Violence. The role of the pediatrician in youth violence prevention in clinical practice and at the community level. Pediatrics.1999;103 :173– 181
- ↵American Academy of Pediatrics, Division of Child Health Research. Periodic survey #38: pediatricians' views on the treatment and prevention of violent injuries to children (executive summary). Available at: www.aap.org/research/periodicsurvey/ps38exs2.htm. Accessed August 4, 2006
- ↵Trowbridge MJ, Sege RD, Olson L, O'Connor K, Flaherty E, Spivak H. Intentional injury prevention and management in pediatric practice: results from 1998 and 2003 American Academy of Pediatrics Periodic Surveys. Pediatrics.2005;116 :996– 1000
- ↵Cohen LR, Runyan CW, Downs SM, Bowling JM. Pediatric injury prevention counseling priorities. Pediatrics.1997;99 :704– 710
- ↵Glascoe FP, Oberklaid F, Dworkin PH, Trimm F. Brief approaches to educating patients and parents in primary care. Pediatrics.1998;101(6) . Available at: www.pediatrics.org/cgi/content/full/101/6/e10
- ↵Sege RD, Hatmaker-Flanigan E, De Vos E, Levin-Goodman R, Spivak H. Anticipatory guidance and violence prevention: Results from family and pediatrician focus groups. Pediatrics.2006;117 :455– 463
- ↵Sege RD, Flanigan E, Levin-Goodman R, et al. Case study: The American Academy of Pediatrics' Connected Kids Program. Am J Prev Med.2005;29(5 suppl 2) :215– 219
- ↵Adler M, Ziglio E, eds. Gazing Into the Oracle: The Delphi Method and Its Application to Social Policy and Public Health. London, United Kingdom: Jessica Kingsley Publishers; 1996
- ↵Wissema JG. Trends in technology forecasting. R & D Manage.1982;12 :27– 36
- ↵American Medical Association, Department of Adolescent Health. Guidelines for Adolescent Preventive Services (GAPS): Recommendations Monograph. Chicago, IL: American Medical Association; 1997
- Copyright © 2006 by the American Academy of Pediatrics