PEDIATRICS Vol. 107 No. 2 February 2001, pp. 318-327

Improving Adolescent Preventive Care in Community Health Centers

Jonathan D. Klein, MD, MPH^*, Marjorie J. Allan^*, Arthur B. Elster, MD^§, David Stevens, MD, MA, Christopher Cox, PhD, Viking A. Hedberg, MD, MPH^*, and Rita A. Goodman

From the ^* Division of Adolescent Medicine, Strong Children's Research Center and Department of Pediatrics and the  Department of Biostatistics, University of Rochester School of Medicine, Rochester, New York; ^§ Department of Clinical and Public Health Practice and Outcomes, American Medical Association, Chicago, Illinois; and the  Bureau of Primary Health Care, Health Resources Services Administration, Department of Health and Human Services, Bethesda, Maryland.

	ABSTRACT

Top Abstract Methods Results Discussion References

Objective.  To evaluate implementation of the Guidelines for Adolescent Preventive Services (GAPS) in Community and Migrant Health Centers (CMHCs).

Design.  Before and after comparison of health center policy, clinician and adolescent self-report, and chart reviews in 5 CMHCs.

Participants.  Eighty-one preintervention and 80 one-year postintervention providers and 318 preintervention and 331 postintervention 14- to 19- year-old adolescent patients being seen for well visits at 5 CMHCs.

Intervention.  Health center staff were trained to implement GAPS and were provided resource materials, patient questionnaires, and clinician manuals.

Main Outcome Measures.  Delivery of and receipt of preventive services and perceived access to care.

Results.  CMHC systems changes were related to stronger leadership commitment to adolescent care. Providers reported high levels of preventive services delivery before and after guideline implementation. After guideline implementation, adolescents reported increases in having discussed prevention content with providers in 19 of 31 content areas, including increased discussion of physical or sexual abuse (10% before to 22% after), sexual orientation (13% to 27%), fighting (6% to 21%), peer relations (37% to 52%), suicide (7% to 22%), eating disorders (11% to 28%), weapons (5% to 22%), depression (16% to 34%), smokeless tobacco (10% to 29%), and immunizations (19% to 48%). Adolescents were also more likely to report knowing where to get reproductive or mental health services and were more likely to have received health education materials. Implementation also increased documentation of recommended screening and counseling in 51 of 79 specific content areas assessed in chart reviews.

Conclusion.  Implementing GAPS increased the receipt of preventive services at these health centers. Adolescents received more comprehensive screening and counseling, more health education materials, and had greater access to care after implementation. GAPS implementation may help improve the quality of care for adolescents.  Key words:  adolescents, preventive services, community health centers.

Many health problems of adolescents are caused by adverse consequences of behavioral choices. These choices are generally assumed to be preventable, in part, through clinical preventive counseling and screening services delivered in primary care settings. Various guidelines for adolescent preventive care recommend screening and counseling to promote healthy behaviors and reduce risks (Guidelines for Adolescent Preventive Services [GAPS],¹ American Academy of Pediatrics,² American Academy of Family Physicians,³ US Preventive Services Task Force,⁴ and Bright Futures⁵). However, despite evidence that adolescent preventive care may be cost-effective,^6-8 as many as 69% to 80% of adolescents' encounters with health care providers do not include counseling or screening interventions.^9,10

In adults, dedicated preventive service visits are more likely to result in the delivery of recommended screening and counseling.¹¹ Systematic implementation of guidelines have also improved delivery of preventive interventions.^12,13 However, clinicians need support to adopt and use office materials that are known to improve systematic preventive service delivery to adults.^14-16 There are few data on how to implement preventive care guidelines for adolescents effectively and materials to promote adoption of adolescent preventive services have not been widely tested.

In 1992, the American Medical Association (AMA) developed an evidence- and consensus-based guideline, GAPS.¹⁷ GAPS is a set of 24 recommendations for adolescents between the ages of 11 to 21 years and a system of care delivery for adolescent preventive care. GAPS recommendations address:

• The delivery of care: an annual preventive visit, developmentally and socioculturally sensitive care, and confidentiality;
• Annual health guidance (health promotion counseling) addressing: injury reduction, healthy diet, physical activity and exercise, responsible sexual behavior, and avoidance of tobacco, alcohol, and other substances;
• Targeted screening and/or counseling interventions (for those at risk for or with concerns about): hypertension, hyperlipidemia, obesity, eating disorders, substance abuse, sexual orientation, pregnancy, sexually transmitted diseases (STDs), human immunodeficiency virus (HIV), cervical cancer, school performance, depression, suicidality, abuse, and tuberculosis; and,
• Immunizations (as recommended by the Centers for Disease Control and Prevention Advisory Committee on Immunization Practices).

The AMA also developed materials to assist clinicians in implementing GAPS recommendations. These included: a GAPS users' manual, model patient and parent initial and periodic screening forms (trigger questionnaires designed for use in conjunction with a clinical evaluation), training materials, and programs for providers. These materials are designed to help providers organize preventive service visits, identify adolescents with biomedical problems, screen for health risk behaviors, develop plans for follow-up, and effectively counsel adolescents and their families. In private practice pilot tests, GAPS materials were found feasible to use and were acceptable to clinicians, adolescents, and their families.¹⁸ In school-based health clinics, GAPS training led to successful implementation of screening forms and increased the volume of preventive care visits.¹⁹ This article reports on our evaluation of the care provided to adolescents in Community and Migrant Health Centers (CMHCs), before and after GAPS implementation.

Nearly 750 CMHCs receive funding from the US Health Resources Services Administration.²⁰ CMHCs serve 8.3 million people from medically underserved, indigent populations in over 3000 clinical sites; 75% of patients are uninsured or covered by Medicaid.²⁰ Many adolescents cared for in CMHCs are at risk for preventable health problems, and CMHC goals are to provide high quality, comprehensive, accessible health care to all age groups. However, <10% of CMHCs have staff with special training in adolescent health. As part of continuous quality improvement activities, the Bureau of Primary Health Care of the US Health Resources Services Administration identified adolescent prevention as a quality improvement goal for 1992. This goal was identified at the same time that GAPS implementation materials were first being developed, and led to the current project.

In this study, we assess whether implementation of comprehensive adolescent preventive services using GAPS results in improved delivery of recommended preventive services to adolescents. We hypothesize that: 1) GAPS implementation in health centers will result in improved systems to ensure quality care delivery to adolescents; 2) that clinicians delivering GAPS services to adolescents will address more health guidance and screening topics and will deliver more effective brief counseling interventions during adolescent preventive visits; and 3) that adolescents using preventive services in health centers that implement GAPS will receive a greater proportion of recommended preventive services and will report greater access to care.

	METHODS

Top Abstract Methods Results Discussion References

Site Selection

In July 1994, the Bureau of Primary Health Care Regional Clinical Coordinators identified 36 potential health center participants, of which 33 were available for telephone screening interviews. Fifteen CMHCs were considered eligible, based on willingness to participate in the evaluation, data system capacity, clinical and administrative leadership stability, and motivation. Seven centers were selected for site visits, based on diversity in geography, population, and clinician types. Site visit interviews with health center leaders, clinicians, and staff in the fall of 1994 provided baseline data for qualitative assessment of the service system and the process of care delivery to adolescents at each center.

Five CMHCs were selected for implementation, based on the presence of an adequate number of adolescent patients, a management information system capable of tracking adolescent well visits, stable administrative and clinical leadership, and leadership commitment to improving adolescent preventive care. Selected health centers included: CAMCare Health Corp, Camden, New Jersey; Oak Orchard Community Health Center (CHC), Brockport, New York; Hudson Headwaters Health Network, Warrensburg, New York; Plan de Salud CHC, Ft Lupton, Colorado; and Highlandtown CHC, Baltimore Medical Systems, Baltimore, Maryland.

GAPS Training

Each health center identified a team of 3 staff members: a GAPS coordinator, and the CMHC's medical and executive directors, or their designees. Teams participated in a 3-day trainers seminar conducted by the AMA in January 1995. Training included adolescent health and behavior change theory; a review of the GAPS recommendations content and rationale; use of GAPS training, trigger questionnaires, and implementation materials; and development of implementation plans for individual clinical sites. Each team's primary focus was to implement GAPS at their CMHC's clinical sites.

Each team planned and implemented a local, on-site training for their CMHC clinicians and staff, reviewing the 24 GAPS recommendations and identifying technical assistance needs and resources for GAPS implementation at their health centers. A total of 11 clinical delivery sites were selected for GAPS implementation by the 5 CMHCs (Table 1). The University of Rochester provided technical assistance and support and facilitated periodic telephone conferences among health center teams. Each CMHC also received financial support for their staff to participate in implementation and to defray additional administrative costs related to the evaluation.

Evaluation

Evaluation of GAPS implementation consisted of before/after comparisons of: 1) policy change at the health centers; 2) clinicians' self-report of the services they provided; 3) adolescents' reports of the services they received at recent preventive care visits; and 4) chart reviews of the care provided to adolescents receiving care at the health centers.

Policy Changes Preimplementation, follow-up, and postimplementation site visits to the health centers were used to determine formal and informal policies and systems level effects of GAPS implementation. Semistructured face-to-face interviews were conducted with health center staff, including front office and nursing staff, providers, medical directors, and chief executive officers. Documents, chart forms, and written policies were also reviewed. Preimplementation site visits were conducted during the initial site selection round, a follow-up visit was made after each health center had conducted local training, and postimplementation visits were made after other follow-up data collection had been completed, at least 1 year after implementation. System level changes examined included: scheduling and confidentiality policies, trigger questionnaire and health education materials use, and need for and availability of referral resources.

Clinician Surveys All health center clinicians who care for adolescent patients were surveyed before and after GAPS implementation. We defined preventive visits as nonacute care visits, including checkups, health maintenance visits, sports, work and camp physicals, and routine gynecological visits. Providers described their usual screening and counseling practices during preventive visits. The questionnaire covered GAPS screening and counseling recommendations, as detailed in Table 2. In addition, detailed questions were asked about providers' discussions of reproductive health, smoking, and alcohol usage, because these areas have the strongest evidence base for clinical practice. During follow-up surveys, clinicians were also asked about their exposure to GAPS training sessions and materials and to other adolescent health continuing education.

Adolescent Surveys Two independent cross-sectional samples of 14- to 19-year-old adolescent patients were recruited during well visits at each of the health centers' GAPS implementation sites. The pre-GAPS sample was enrolled after the trainers training, but before local training occurred. The post-GAPS sample was enrolled between 9 and 15 months after implementation and enrollment ended before follow-up site visits or clinician surveys were conducted.

Chart Reviews Adolescents' medical records were reviewed for documentation of preventive care delivery. Each chart was reviewed by a trained research assistant and overread by an experienced clinician. Disparities were resolved by consensus and by a tiebreaker review. All adolescents who consented to be surveyed had their charts reviewed.

Measures

The effectiveness of counseling was measured for tobacco, alcohol, and reproductive care delivery by exploring the depth of screening and counseling provided (based on recommended guidelines for these services.) Access to care was measured by adolescents self-report of their perceived access to services for common adolescent problems.

Statistical Analyses

Comparisons between providers present during both before and after surveys were analyzed using McNemar's test for 2 related samples. The impact of GAPS implementation on the entire population of clinicians in the health centers both before and after GAPS (accounting for clinician turnover) was assessed by comparing proportions, using the entire before and after clinician samples. The significance of these changes was assessed using the combined variance for those present at both times and for those present in only the before or after sample, to account for the cohort of clinicians as well as those who entered and exited these samples.

² comparisons were used to compare adolescents' self-report of the services that they had received before and after GAPS implementation. Results are considered statistically significant at P  .05 for these analyses. All study protocols were approved by the University of Rochester Committee on the Protection of Human Subjects.

	RESULTS

Top Abstract Methods Results Discussion References

Site Visits

The 5 health centers varied with regard to their geography, size, number of sites, types of providers, and whether they already had specialized clinical services for adolescents (Table 1). The process of implementing GAPS also varied. Two CMHCs implemented GAPS at all sites, 2 selected only their main site, and 1 their main site and 2 satellite clinics for implementation.

Local Training Each CMHC's implementation team planned and implemented local training for their center's staff. Training occurred between 2 and 6 months after the trainers training sessions. Based on site visit data, it seemed that more effective and quicker planning for GAPS implementation was associated with stronger commitment to GAPS by medical and administrative leaders and other health center staff and by expansion of the team to additional decision makers. Conversely, a lack of dedicated clinician time and difficulty in dedicating resources delayed planning at some sites.

Implementation of GAPS Implementation occurred 4 to 11 months after trainers training. Incorporation of GAPS adolescent prevention goals into health center continuous quality improvement plans and empowerment of nonclinician staff to change the process of care delivery (ie, patient flow) were associated with timely and successful implementation. Delayed implementation was attributable to staff turnover in 2 CMHCs. Four of the 5 CMHCs' implementation led to institutionalization of GAPS at the initial delivery sites and to dissemination of GAPS to the CMHCs' other clinical sites.

System Changes Each CMHC adjusted the scheduling of adolescent well visits to allow for 30-minute visits. Additionally, policies were eventually adopted so that care could be provided confidentially for adolescents in need. Each site also implemented use of the GAPS trigger questionnaire for well visits, promoted private counseling time, and enhanced (to varying degrees) their health education materials and referral networks. However, most sites found it difficult to identify resources to meet mental health, specialty care, and nutrition needs.

Clinician Surveys

Seventy-nine of 81 clinicians (97%) were surveyed pre-GAPS and 74 of 80 (91%) were surveyed post-GAPS implementation about the services that they provide to adolescent patients. Clinicians' self-reported usual screening practices were high both before and after GAPS implementation. For example, clinicians reported asking >90% of their adolescent patients about tobacco, alcohol, drugs, and sex, both before and after GAPS implementation (Table 3). By clinicians' reports, only one screening behavior, asking about sexual orientation, was provided to <50% of adolescents seen for well visits. The post-GAPS surveys showed relatively few changes in the proportion of adolescents for whom clinicians reported providing counseling or screening. The only areas that significantly increased were screening for violence risk and for eating disorders.

The differences in pre- to post-GAPS self-reported clinician provision of screening and counseling did not vary by whether clinicians were part of the pre/post cohort (n = 60) or whether they were present only in the pre (n = 19) or post (n = 14) cross-sectional samples.

Only 65% of providers reported having received GAPS training. The number participating varied by health center, with 100% participation at one health center, versus 21% participation at another (² across all 5 health centers = 21.9; P < .001). Providers participated in 1 to 4 training sessions, with a mean of 1.13 sessions. There were no significant differences in delivery of screening or counseling services between providers that did or did not participate in training. Similarly, there were no differences in clinician performance by site or CMHC, either before or after GAPS.

Adolescent Surveys

Pre-GAPS implementation, 318 of 366 eligible adolescents (87%) seen at the health center sites were approached, recruited, and enrolled in the study. Of these, 260 (82% of enrollees, 71% of eligibles) completed postvisit telephone surveys. After GAPS training and implementation, we recruited and enrolled 283 of 360 eligible adolescents (79%) during the post-GAPS period. Of these, 274 (97% of enrollees, 77% of eligibles) completed postvisit telephone surveys. There was no overlap between the pre- and postadolescent samples. The mean age of surveyed adolescents was 16.2 years (standard deviation: 1.8 years) pre-GAPS and 16.3 years (standard deviation: 1.7 years) post-GAPS. The proportion of female adolescents was 69% and 72%, respectively, and 88% and 99% of visits were for preventive care. There were no statistically significant differences between pre- and post-GAPS adolescents by age, sex, or reason for visit, and each health center contributed a comparable proportion of patients to both before and after GAPS survey samples.

Screening Questionnaires, Health Education, and Physical Examinations

After GAPS implementation, there was a significant increase in the proportion of adolescents who completed health questionnaires and discussed their answers with their providers (Table 3). Additionally, post-GAPS adolescents were much more likely to have received and read health education materials than pre-GAPS teens (23% vs 11%; P < .001).

The physical examination services provided to adolescents did not change much after GAPS implementation. However, both breast examinations and urine tests were reported as being performed less frequently, and HIV tests increased slightly in frequency. These changes were in the directions that would be expected if clinicians were using GAPS guidelines.

Adolescents' Discussions With Providers During Preventive Visits

Adolescents reported having received substantially less counseling and screening services (both pre- and post-GAPS; Table 3) than were reported to have been delivered by their clinicians. For example, 58%, 63%, and 69% of adolescents interviewed recalled having discussed alcohol, cigarettes, or sexual behavior with their clinicians at their most recent well visits. In contrast, clinicians reported delivering these services to >90% of adolescents they see.

Discussion between adolescents and clinicians increased in 19 of 31 recommended preventive health content areas after implementation of GAPS. The largest pre/post differences in the proportion of adolescents who reported receiving services were for discussions about: weight, eating disorders, smokeless tobacco, immunizations, family functioning, school performance, and mental health. Interestingly, the proportion of adolescents who had discussed cigarettes, alcohol, or sexual behaviors, including STDs and HIV, had not significantly changed. However, the proportion of adolescents who reported receiving these services were relatively high compared with rates for having received other recommended preventive services.

Access to Care

At baseline, only 36% of adolescents seen for preventive care visits could identify a source of care for counseling, and nearly 1 in 5 did not know where they might get either birth control services or sports or school physical examinations. In contrast, after GAPS implementation, 64% of adolescents could identify a source for counseling, 91% knew where to go to get birth control, and 87% knew where to go for sports or school physicals (P values all <.01).

Adolescents' Medical Records

Medical records were reviewed for documentation of patient-provider discussions during preventive care visits. All adolescents who were recruited and consented had their chart reviewed; thus, there were 331 pre-GAPS implementation charts and 317 post-GAPS charts.

Before GAPS implementation, 4 of the 5 health centers had an adolescent or parent questionnaire; however, these forms were not regularly used at most sites. Overall, pre-GAPS, there were forms present in 29% of charts. After GAPS implementation, which included an emphasis on systematic screening for adolescents, a screening form/trigger questionnaire was present in 76% of charts (P < .001). Almost one half of the charts (47%) contained completed Initial Adolescent Visit forms and 33% had completed Interim Forms. Parent Forms were also present in 26% of adolescents' charts.

As with adolescents' self-reported services received, the chart review showed significant increases in the proportion of adolescents whose charts contained documentation of their having received recommended preventive services in 51 of the 79 content areas assessed (Table 4). In fact, the proportion of charts that had documentation of content increased well in excess of the magnitude of the increase in counseling receipt that was reported by the adolescents. There were also modest increases in documentation in some content areas that were not on the GAPS trigger questionnaire, but which were part of the guidelines (ie, counseling for bicycle helmet use). Additionally, the charts showed increased documentation of screening for family history of cardiovascular disease, hypertension, high cholesterol, substance abuse, mental illness, and physical or sexual abuse (Table 5). GAPS implementation also may have had a positive impact on delivery of hepatitis B immunizations; with rates of immunization rising from 17% to 36% (P < .001) among charts reviewed.

	DISCUSSION

Top Abstract Methods Results Discussion References

Implementation of comprehensive adolescent preventive services using the AMA GAPS in these 5 community health centers led to significant changes in the process of care delivery at these sites. In addition, GAPS implementation led to improvements in the amount and content of preventive care received by adolescents during well visits at these health centers.

The resources and processes needed to create changes in the 5 health centers varied greatly. In general, leadership by both clinical and administrative directors was needed to successfully incorporate GAPS into the culture of the health center. As each CMHC implemented GAPS, they not only had to manage significant change in the flow of adolescent patients, they also had to broaden their health care delivery to adolescents, from a pediatric or family-centered model to one that recognized adolescents' needs for privacy and confidentiality, and family involvement, within a developmental context.^21,22

Miller has described the steps needed to bring about culture change within a primary care practice organization as requiring joining, transforming, and learning.²³ Although relatively small organizations, CMHCs are complex systems. Thus, it was not surprising to see changes evolve slowly and with variation across the 5 centers' sites, despite selection of centers likely to succeed.²⁴ Most CMHCs incorporated continuous quality improvement methods into their activities during the early 1990s²⁵; thus, the centers we worked with were able to consider and plan for change within existing continuous quality improvement structures. Others have also found that quality assessment processes and involving office staff are most likely to lead to successful implementation of service delivery systems change.^12,16 In the case of these CMHCs, adding adolescent care required special consideration of confidentiality, and this led to some resistance from staff. There were also some new costs. Although we do not know whether financial pressure or nonfinancial barriers delayed implementation of adolescent preventive care at some of our sites, the center that did not have buy-in from their administration and quality improvement committee was not able to change practices or sites beyond the GAPS team's own clinical service delivery.

Interestingly, although there were other systems changes involved in implementation of GAPS at the health centers involved, many of the clinicians interpreted GAPS to mean the trigger questionnaire used during well visits, rather than the entire CMHC's commitment to comprehensive preventive care for youth. The trigger forms were seen as identifying many previously undetected issues among the adolescents seen in the health centers. Additionally, these CMHCs found themselves struggling to find resources to meet these needs. Special needs which were hardest to meet were those dependent on specialty care, nutritionists, and mental health providers.

The CMHCs that had the least previous experience with adolescent services were more easily able to implement systems-wide changes than were those with preexisting special services for adolescents. In these centers, individuals with clear clinical expertise were identified as adolescent leaders and helped deliver local training. However, these midlevel clinicians did not have, or were unable to assume, authoritative leadership roles within their health centers. Thus, they were unable to successfully leverage their clinical expertise into broader quality improvement activities in support of GAPS implementation. Previous studies have found that clinical nurse managers often have difficulty implementing changes in their practice settings.^26,27 In our study, it is possible that these health centers had already decided to separate (or marginalize) adolescent care, and, thus, the nurse practitioners involved had a harder task than in those centers that did not already explicitly limit their clinician's involvement in adolescent health care.

Physician Reports of Care

We examined the impact of GAPS implementation on service delivery at these CMHCs, both by examining the reported behaviors of the clinicians and by examining the care received by adolescents who were seen at well visits. Although there was some improvement in screening and counseling in a few areas, clinicians consistently reported very high levels of preventive service delivery both pre- and post-GAPS. Clinicians often overreport the services that they provide to patients.^28-30 Additionally, the validity of clinician report of preventive care delivery for adolescents is unknown. Previous studies have found that physician gender,³¹ patient socioeconomic status³² or gender,³³ and other practice factors, such as insurance case mix and patient volume,³⁴ affect preventive service delivery. Our design did not have sufficient power to explore these factors in detail. The lack of certainty regarding the validity of physician's reports makes our not having found a training effect less significant. Nonetheless, exposure to the interventions and educational materials/sessions may have resulted in clinician's becoming more aware of the GAPS content recommendations. Conversely, even those clinicians who did not report attending educational sessions may have been exposed to and affected by the GAPS forms and training materials.

Adolescent Receipt of Care

Adolescent self-report of the services that they had received showed substantial changes between the pre- and the post-GAPS periods. The cohort of adolescents having a preventive visit after GAPS implementation reported having received more counseling about a wide variety of health issues than those who had visits before GAPS implementation. Postimplementation adolescents were also more likely to have completed a screening questionnaire, reflecting the use of GAPS materials in these clinical sites. Additionally, they were more likely to have received and read health education material during their encounters, suggesting that multiple aspects of these CHC clinical care systems underwent changes in response to implementation of GAPS.

Mental health, nutrition, and violence counseling were among those services that both started lowest and increased most with implementation of GAPS. As has been seen in adults, the higher prevalence issues (at baseline) tended to have smaller magnitude of change.¹³ Some services were reported as having been provided less frequently; for example, neither teaching breast self-examination nor routine urinalyses are recommended by GAPS (or by most other guidelines for adolescent preventive care), and the rate at which these were reported as having been performed decreased in post-GAPS surveys. In contrast, the rate of reported discussion about bicycle helmets did not change significantly. Although counseling regarding their use is recommended, the GAPS trigger questionnaire does not contain an item for helmet use. We do not know from this study whether this recommendation is crowded out by those that are prompted or whether there were other reasons for a lack of improvement in counseling about this topic. However, these data lend credibility to the importance of the trigger questionnaires in promoting more comprehensive content delivery. Trigger questionnaires combined with clinical interviews have been found to increase the accuracy of risk identification in adolescents.³⁵ Thus, widespread adoption of patient trigger questionnaire use, combined with a strategy of comprehensive preventive counseling, may result in delivery of recommended content. The lack of difference between clinicians who received training in GAPS and those who did not (but who, nonetheless, used the materials in practice) may suggest the importance of trigger questionnaires and other materials in increasing provision of preventive services. Interestingly, this spillover effect also suggests that extensive training is not required to bring about significant changes in care.

Implementing GAPS at these CMHCs also increased documentation of the screening and counseling provided to adolescents and of significant family medical history. Documentation was most affected by inclusion of the trigger questionnaire in the charts. This has also been reported previously in implementation of trigger questionnaires in an adult clinic.¹³ In fact, the proportion with documentation of many areas was higher than the proportion of adolescents reporting having discussed these particular issues. However, clinicians were credited with discussing an item if the screening form was completed. Thus, screening questionnaires may overreport preventive care delivered to adolescents, compared with adolescent recall. Adolescents are more valid and reliable reporters of the content of the preventive care that they have received than their clinicians.^36,37 Thus, the adolescent surveys provide a more accurate view of the services that were actually delivered during these well visits. These adolescents' reports of increased rates of services that they have received reflect true improvements in the preventive care delivered at these CMHCs. Additionally, their reports of increased perceived access to mental health and reproductive health services suggests that the clinicians were discussing these services and issues, with increased attention to the adolescents' current or future needs.

Our study is limited in generalizability by our choice of health centers, because the CMHCs in this study were selected for their likelihood of success. However, our goal was to be able to demonstrate improvements in the care delivered to adolescents and random selection of clinical sites could not be performed. The clinician surveys are potentially biased by the validity of self-report, by our limited power to show clinician change, and by social desirability on the part of providers. The adolescent measures, while valid, may also have been subject to social desirability bias. Although adolescents are less likely to have been affected in their responses unless real changes in their care had occurred, nonetheless, we were not able to conduct a controlled trial to conclusively demonstrate this fact. Similarly, we were not able to fully control for the effects of individual clinician or site differences in training receipt or in service delivery. The adolescents' visits also were not equally distributed across all health center clinicians. Nonetheless, the adolescent surveys accurately reflect the care provided to teenagers at these clinical sites. Despite multiple comparisons, most of the differences we saw post-GAPS were significant at P < .01 or less. Additionally, the use of multiple data sources showing congruent changes over time adds to the validity of our findings. Our results provide useful lessons for promotion of improved preventive services for adolescents and for the effectiveness of clinical quality improvement and systems change strategies in improving the quality of primary care.

The evidence base for GAPS and other clinical guidelines has been criticized.³⁸ However, although various guidelines for adolescent care include recommendations with varying strength of evidence, the content of different adolescent care guidelines are nearly identical.³⁹ And, although many studies have tried to assess the effectiveness of brief counseling interventions, most have not had sufficient power or have not documented whether the clinical services ever were delivered.^7,40 The Agency for Health Care Research and Quality recently identified implementation as a key area for prevention effectiveness research.⁴⁰ Similarly, consumer and health insurance groups interested in the quality of care have begun to recognize the importance of effective preventive care delivery for children and youth.^41,42 This study provides efficacy evidence that GAPS clinical systems change strategies improve the quality of preventive care provided to adolescents. If these practice changes are generalizable and sustainable, implementation of better preventive care systems and service to provide consistent, comprehensive preventive care for adolescents and, hopefully, to decrease preventable morbidity and mortality, is within our reach.

	ACKNOWLEDGMENTS

This work was funded by the Bureau of Primary Health Care and the National Association of Community Health Centers and by a Generalist Faculty Scholars Award from the Robert Wood Johnson Foundation.

We thank Susan G. Millstein and Janet Gans-Epner for their assistance in measures development, and the clinicians and staff of the participating health centers for their participation and enthusiasm.

	FOOTNOTES

Received for publication Mar 2, 2000; accepted Jun 12, 2000.

Reprint requests to (J.D.K.) Department of Pediatrics, University of Rochester, 601 Elmwood Ave, 690, Rochester, NY 14642. E-mail: jonathan_klein{at}urmc.rochester.edu

	ABBREVIATIONS

GAPS, Guidelines for Adolescent Preventive Services; AMA, American Medical Association; STD, sexually transmitted disease; HIV, human immunodeficiency virus; CMHC, Community and Migrant Health Center.

	REFERENCES

Top Abstract Methods Results Discussion References

1. Elster AB, Kuznets MJ. Guidelines for Adolescent Preventive Services. Baltimore, MD: Williams & Wilkins; 1993
2. American Academy of Pediatrics, Committee on Practice and Ambulatory Medicine. Recommendations for Preventive Pediatric Health Care. Elk Grove Village, IL; 1998
3. Commission on Public Health and Scientific Affairs AAFP Age Charts for Periodic Health Examinations: 13 to 18 years. Am Fam Physician 1992; 45:808-810
4. US Preventive Services Task Force. Guide to Clinical Preventive Services. 2nd ed. Alexandria, VA: International Medical Publishing; 1996
5. Green M, ed. Bright Futures: Guidelines for Health Supervision of Infants, Children, and Adolescents. Arlington, VA: National Center for Education in Maternal and Child Health; 1994
6. Gans JE, Alexander B, Chu RC, Elster AB The cost of comprehensive preventive medical services for adolescents. Arch Pediatr Adolesc Med 1995; 149:1226-1234 [Abstract/Free Full Text]
7. Downs SA, Klein JD Clinical preventive services efficacy and adolescents' risky behaviors. Arch Pediatr Adolesc Med 1995; 149:374-379 [Abstract/Free Full Text]
8. Hedberg VA, Bracken AC, Stashwick CA. Long-term consequences of adolescent health behaviors: implications for adolescent health services. Adolesc Med. 1999;10:137-151, vii
9. Lewis CE, Clancy C, Leake B The counseling practices of internists. Ann Int Med 1991; 114:54-58
10. Igra V, Millstein SG Current status and approaches to improving preventive services for adolescents. JAMA 1993; 269:1408-1412 [Abstract/Free Full Text]
11. Kottke TE, Solberg LI, Brekke ML, Cabrera A, Marquez MA Delivery rates for preventive services in 44 Midwestern clinics. Mayo Clin Proc 1997; 72:515-523 [Abstract]
12. Dietrich AJ, Woodruff CB, Carney PA Changing office routines to enhance preventive care. Arch Fam Med 1994; 3:176-183 [Abstract/Free Full Text]
13. Madlon-Kay DJ, Harper PG, Reif CJ Use of a "health habits questionnaire" to improve health promotion. Arch Fam Med 1995; 4:459-462 [Abstract/Free Full Text]
14. Soumerai SS, Avorn J Principles of educational outreach "academic detailing" to improve clinical decision making. JAMA 1990; 263:549-556 [Abstract/Free Full Text]
15. Dickey LL, Kamerow DB Primary care physicians' use of office resources in the provision of preventive care. Arch Fam Med 1996; 5:399-404 [Abstract/Free Full Text]
16. Leininger LS, Finn L, Deickey L, An office system for organizing preventive services. Arch Fam Med 1996; 5:108-115 [Abstract/Free Full Text]
17. Elster AB, Kuznets NJ. AMA Guidelines for Adolescent Preventive Services (GAPS): Recommendations and Rationale. Baltimore, MD: Williams & Wilkins; 1994
18. Gans Epner JE, Levenberg PB, Schoeny ME Primary care providers' responsiveness to health-risk behaviors reported by adolescent patients. Arch Pediatr Adolesc Med 1998; 152:774-780 [Abstract/Free Full Text]
19. Brey LC, Gans Epner J Improving the delivery of clinical preventive services in school-based health centers (SBHCs). J Adolesc Health 1997; 20:159
20. Health Resources and Services Administration, Bureau of Primary Health Care Programs. Available at: http://www.bphc.hrsa.dhhs.gov/bphc/INDEX  1. HTM. Accessed September 15, 1999
21. US Congress, Office of Technology Assessment. Adolescent Health, I: Summary and Policy Options, OTA-H-468. Washington, DC: US Government Printing Office; 1991
22. Millstein SG, Petersen AC, Nightingale EO, eds. Promoting, the Health of Adolescents: New Directions for the Twenty First Century. New York, NY: Oxford University Press; 1993
23. Miller WL, Crabtree BF, McDaniel R, Stange KC Understanding change in primary care practice using complexity theory. J Fam Pract 1998; 46:369-376 [Medline]
24. Crabtree BF, Miller WL, Aita VA, Flocke SA, Stange KC Primary care practice organization and preventive services delivery: a qualitative analysis. J Fam Pract 1998; 46:403-409 [Medline]
25. US Department of Health and Human Services, Medicaid Bureau, Health Care Financing Administration. A Health Care Quality Improvement System for Medicaid Managed Care: A Guide For States. Washington, DC: US Department of Health and Human Services, Medicaid Bureau, Health Care Financing Administration; 1993
26. Caine C, Kenrick M The role of clinical directorate managers in facilitating evidence-based practice: a report of an exploratory study. J Nurs Manage 1997; 5:157-165
27. Stange KC, Zyzanski SJ, Jaén CR, Illuminating the "black box": a description of 4454 visits to 138 family physicians. J Fam Pract 1998; 46:377-389 [Medline]
28. Klein JD, Portilla M, Goldstein AE, Leininger L Training pediatric residents to prevent tobacco use. Pediatrics 1995; 96:326-330 [Abstract/Free Full Text]
29. Russell NK, Boekeloo BO, Rafi IZ, Leif I, Brekke ML, Cabrera A, Marquez MA Unannounced simulated patients' observations of physician STD/HIV prevention practices. Am J Prev Med 1992; 8:235-240 [Medline]
30. Boekeloo BO, Marx ES, Kral AH, Bowman M, Rabin DL Frequency and thoroughness of STD/HIV risk assessment by physicians in a high-risk metropolitan area. Am J Public Health 1991; 81:1645-1648 [Abstract/Free Full Text]
31. Lewis CE, Clancy C, Leake B, Schwartz JS The counseling practices of internists. Ann Intern Med 1991; 114:54-58
32. Taira DA, Safran DG, Seto TB, Rogers WH, Tarlov AR The relationship between patient income and physician discussion of health risk behaviors. JAMA 1997; 278:1412-1418 [Abstract/Free Full Text]
33. Joffee A, Radius S, Gall M Health Counseling for adolescents: what they want, what they get, and who gives it. Pediatrics 1988; 82:481-485 [Abstract/Free Full Text]
34. Zyzanski SJ, Williams RL, Flocke SA Selection of key community descriptors for community-oriented primary care. Fam Pract 1996; 13:280-288 [Abstract/Free Full Text]
35. Schubiner H, Tzelepis A, Wright K, Podany E The clinical utility of the Safe Times Questionnaire. J Adolesc Health 1994; 15:374-382 [CrossRef][Medline]
36. Klein JD, Graff CA, Santelli JS, Allan MJ, Elster AB. Improving Adolescent Health Care Surveillance: Health Survey Research Methods. Washington, DC: National Center for Health Statistics, Department of Health and Human Services, US Public Health Service. In press
37. Klein JD, Graff CA, Santelli JS, Hedberg VA, Allan MJ, Elster AB. Developing quality measures for adolescent care: validity of adolescents' self-reported receipt of preventive services. Health Serv Res. 34;1:391-404
38. Berg AO Clinical practice guideline panels: personal experience. J Am Board Fam Pract 1996; 9:366-370
39. Elster AB Comparison of recommendations for adolescent clinical preventive services developed by national organizations. Arch Pediatr Adolesc Med 1998; 152:193-198 [Abstract/Free Full Text]
40. Agency for Health Care Policy and Research. Translating Evidence into Practice: Conference Summary. Rockville, MD: Agency for Health Care Policy and Research; 1998. Available at: http://www.ahcpr.gov/clinic/trip1998/
41. Lansky D Measuring what matters to the public. Health Aff 1998; 17:40-41 [Abstract]
42. Foundation for Accountability. Accountability Action. Portland, OR: Foundation for Accountability; 1999;3:5