PEDIATRICS Vol. 122 No. 3 September 2008, pp. e564-e572 (doi:10.1542/peds.2006-3273)
ARTICLE |
Evaluation of Community-Based Health Projects: The Healthy Tomorrows Experience
a Independent Research and Evaluation Consultant, Evanston, Illinois
b Department of Clinical Pediatrics, University of Massachusetts Medical School, Worcester, Massachusetts
c Outcomes and Evaluation
d Bright Futures Pediatric Implementation Project, American Academy of Pediatrics, Elk Grove Village, Illinois
e Community Health Services, Denver Health, Department of Pediatrics and Preventive Medicine, University of Colorado School of Medicine, Denver, Colorado
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
OBJECTIVES. To address the "millennial morbidities," pediatricians must partner with community-based organizations to develop interventions. Little is known about the capacity of the resulting programs for program evaluation or the importance of evaluation in project success and sustainability. The objective of this study was to examine the capacity of community-based health programs to conduct project evaluations and determine the impact of project evaluation on project outcome.
METHODS. Project directors from 149 community-based programs funded from 1989 to 2003 through the Healthy Tomorrows Partnership for Children Program were surveyed regarding their project experience with evaluation and documentation of project outcomes and the current status of their project.
RESULTS. Program directors from 123 (83%) programs completed the survey. Despite barriers to the evaluation process, 83% of the respondents indicated that their evaluations produced useful information. Programs that were described by respondents as "well evaluated" were more likely to report that the evaluation was implemented as planned and that the evaluation included outcome measures. Projects were more likely to be sustained in their original form when at least 1 outcome was reported on the survey.
CONCLUSIONS. Evaluation of community-based programs, although challenging, is beneficial to project success and sustainability. Policy makers and funding agencies should consider ways to encourage community partnerships to incorporate evaluation into their planning process.
Key Words: program evaluation • outcomes • community health services • community pediatrics • community-based initiatives
Abbreviations: AAP—American Academy of Pediatrics • HTPCP—Healthy Tomorrows Partnership for Children Program
The health of America's children depends on social, community, and environmental factors that interfere with the pediatrician's ability to improve health outcomes.1 To address the "millennial morbidity" of mental health disorders, overweight and health disparities that affect children and families in the 21st century, pediatricians will need to participate in the development of community-based services to augment the care that they provide in their practices.2,3 During the past 20 years, federal and state governments, as well as private foundations, have funded collaborative activities to improve immunization rates, increase the quality of asthma management, enhance early brain development, and increase access to mental health services.4–8 The American Academy of Pediatrics (AAP) has actively promoted the integration of pediatric practice and community-based initiatives, through programs such as the Community Access To Child Health program initiative.9 Evaluation of the effectiveness of pediatric involvement in child health initiatives would help pediatricians and those who fund these projects to understand the "best practices" for a successful collaboration.
Although limited, the literature that exists supports process and outcome evaluation as beneficial to programs and their constituencies10; however, evaluation itself is a relatively young discipline, with an evolving set of methods and even terminology.11 Nevertheless, the funders of community-based health initiatives generally require some level of evaluation for continued funding. Despite recent refinements in the assessment of program effectiveness,12–15 funders seldom provide financial or technical support for required evaluations. Although many community-based health agencies find themselves collecting diverse types of evaluation information,16 most do not have the internal resources or expertise to gather and report outcomes data satisfactorily.17 The barriers to conducting outcome evaluation and the level of evaluation activity that is feasible for community-based organizations have not been extensively explored.18 It seems likely, however, that there would be benefits to projects from the evaluation effort, even if the evaluation is not as rigorous as those conducted by researchers in academic settings.10
To examine the capacity of community-based organizations to conduct project evaluations and the impact of the evaluation process on project outcome, we conducted a survey of program directors of projects that are funded through the Healthy Tomorrows Partnership for Children Program (HTPCP). HTPCP is a federally funded grant program that provides funding and technical assistance to community-based programs that partner with pediatricians to increase children's access to care in urban and rural areas throughout the United States. Because HTPCP grants are relatively small (approximately $50000 per year), most of the funding is committed to service delivery rather than to evaluation, making it likely that projects would encounter substantial barriers to evaluation in the course of their work. This study was conducted as part of a broader evaluation of the HTPCP program, which included project surveys, a retrospective review of project documents, and a series of case studies.19
|
METHODS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Healthy Tomorrows Partnership for Children Program
The HTPCP is a partnership between the AAP and the Maternal and Child Health Bureau of the United States Health Resources and Services Administration. Initiated in 1989, the program seeks to support innovative community-based efforts to improve children's health. The program provides funding; technical assistance; and access to other local, regional, and national resources. There are 4 major goals of the HTPCP:
- to implement innovative and cost-effective programs to promote preventive health care for vulnerable children and their families, especially for those with limited access to quality health services;
- to foster cooperation among community organizations, agencies, and families;
- to encourage pediatricians' and other pediatric health professionals' involvement in the process; and
- to build community and statewide partnerships among professionals in health, education, social services, government, and business to achieve self-sustaining programs to ensure healthy children and families.
Healthy Tomorrows projects have been funded in 44 states, the District of Columbia, Guam, and Puerto Rico and include sites in both urban and rural areas. Projects are diverse in focus, services delivered, and outcomes sought, as well as in institutional structure and resources available. After a competitive review process, projects are funded for up to $50000 per year for 5 years, with a matching requirement. The specific focus of each project is based on local needs; projects have addressed issues such as access to health care, mental health assessment, access to dental care, obesity prevention, asthma care, and a myriad of other topics. The Maternal and Child Health Bureau requires an annual report of use data and outcome measures.
Program directors for all projects initially funded through the HTPCP from 1989 through 2003 were eligible to participate in the evaluation. Of the 158 projects funded during this time frame, 3 programs were excluded because they had withdrawn from HTPCP, and 6 were excluded because no current contact information could be obtained, resulting in a final potential sample of 149 projects. This study was approved by the institutional review board of the AAP.
Survey Development
The survey was designed to answer specific research questions of interest to HTPCP staff and others involved in the provision of funding and technical assistance to Healthy Tomorrows projects. We began with structured discussions with HTPCP project directors and staff about their evaluations and the challenges that they faced. The survey questions were developed iteratively, in consultation with HTPCP staff, community-based pediatricians, site visitors, project directors, and AAP staff with relevant expertise. The survey was designed to explore 2 specific areas related to the experience of HTPCP projects. The first set of questions addressed factors other than financial support that make a difference for community-based programs. The second set of questions focused on the evaluation potential of community-based organizations and their capacity to document outcomes. The survey focused on the program's experience with evaluation (barriers, use of resources, and utility of information) and the documentation of project outcomes. (A copy of the survey is included in the Appendix)
|
Sampling Method
The survey was mailed in January 2004 to 149 project directors of HTPCP projects, both those that were currently funded and those that had completed funding. The latter projects received the evaluation questions in the context of a longer survey that included questions related to project experience with HTPCP and technical assistance. Three rounds of the survey were mailed.
Data Analysis
Survey data were held in an Access (Microsoft Corp, Redmond, WA) database, and SPSS (SPSS Inc, Chicago, IL) was used to calculate simple frequencies and cross-tabulations. Associations between categorical variables were assessed by using the 
2 test with an 
value of .05.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
After 3 rounds of mailed surveys, responses were received from 126 of the 149 projects, for a response rate of 85%. Three of the surveys, however, were blank, with a notation that no one from the period of the grant was available to respond to the survey, leaving the total number of usable surveys at 123. Seven of the oldest projects were unable to respond to some of the questions, particularly specifics about the evaluation, either because the respondent was not a part of the original project or simply because he or she could not recall. Missing values on individual variables are most often attributable to this issue for the older projects.
Almost half of the responding HTPCP projects were located in inner cities. Approximately 18% were in rural areas, and another 17% served an entire county or region. Nearly all projects served multiple ages of children, and a majority reported serving adults as well, most frequently pregnant women (69%). More than three quarters of projects served multiple ethnic groups, and most served uninsured (74%) and/or Medicaid-eligible (82%) families and children.
Given the diversity of HTPCP projects and project goals, services that are provided by the projects vary widely. Services can be broadly categorized as primarily referral services, case management, health education, home visitation, and medical services.
Program-Evaluation Results
Just half (50%) of respondents believed that their projects had been well evaluated; 35% reported that their evaluations produced useful information but were not satisfactory (Table 1). Only 5 respondents indicated that their projects were not well evaluated, and nearly all of these were older, completed projects. Projects in their first year of funding were most likely to respond that they did not know how well their projects were evaluated.
|
Despite the number who were not satisfied with or unsure about their evaluations, a substantial majority (102 [83%] projects) of respondents indicated that their evaluations had produced useful information. The most common uses of information were to improve services (69%), advocate for the service population (58%), or to obtain funding (51%). Among the 21 respondents who did not report that their evaluations produced useful information, the most frequent explanation was that it was still too early in the project (38%). More than one third of this group reported data issues, either that the project did not get the data it needed (38%) or that the data were not useful (14%).
Experience With Program Evaluation
More than half (54%) of respondents reported that their evaluation plans had changed and were not implemented as originally planned (Table 2). Among those who reported changing their evaluation plans, most (64%) reported multiple reasons for the change. The single most common reason for a change was that the project staff learned a way to improve the plan (51%). Other common reasons for change included discovery that the original plan was not feasible (38%) or that anticipated data were not available (35%). Program change led to adjustment of the evaluation plan in 32% of the projects.
|
Although 15 respondents did not report any barriers to evaluation, most (56%) respondents reported multiple barriers to evaluation (data not shown). The most commonly encountered barriers were related to resources: money (34%) or staff (33%). Other significant issues included an evaluation that was excessive for a modest program (28%), lack of evaluation expertise (28%), and staff turnover (24%).
Just under half (46%) of HTPCP projects reported using HTPCP grant resources in conducting their evaluations, but only 
14% used only the resources of their HTPCP grants. Ten percent obtained separate funding for evaluation activities. Almost two thirds of projects reported using the resources of their own organizations for evaluation. Of projects that did not use HTPCP resources for evaluation, only a slightly higher proportion, 70%, reported use of organizational resources (data not shown). Nearly half (48%; data not shown) reported using multiple resources for evaluation. A substantial proportion (44%; data not shown) also reported that they were able to obtain HTPCP evaluation technical assistance from the AAP, and virtually all of those who received such assistance indicated that it was helpful. The most common areas of assistance were refinement of the evaluation plan or logic model, measurement tools, and determining which outcome to track.
Table 3 displays the associations between the respondent's perception of the quality of the project evaluation and other evaluation results indicators. In this analysis, reporting that the evaluation was implemented as planned (P < .01), reporting documented outcomes (P < .001), and useful information (P < .001) all were associated with a report that the project was well evaluated. Among those who were unsure of how well the project was evaluated, 82% reported that their evaluations were not implemented as planned, none reported documented outcomes, and 54% reported that their evaluations produced useful information.
|
Reporting of Documented Outcomes
Seventy-two (62%) programs reported documenting both process (eg, number served, referrals made, training provided, materials produced) and outcome measures in their evaluations. Five (4%) documented only outcome measures, 31 (27%) documented process measures only, and 8 (7%) documented neither type of measure. The diversity of the programs is reflected in the broad range of outcomes reported, including indicators such as changes in knowledge or behavior, reductions in emergency department visits, and prevalence of a condition.
Reported documentation of outcomes was not significantly related to the respondent's confidence in the project's sustainability as reported on the project experience portion of the survey. For completed projects, we had determined whether they were actually sustained as we contacted them to participate in the evaluation project; 74 (79%) of the 94 completed projects still existed in some form. Among the 70 completed projects with returned surveys, whether the project was actually sustained was significantly related to reporting of at least 1 outcome on the evaluation survey (P < .05; Table 4). Projects that reported at least 1 outcome were much more likely than those with no outcomes reported still to exist in their original form (71% vs 36%); similarly, a higher proportion of those who reported no outcomes than of those with at least 1 outcome reported no longer existed (27% vs 8%).
|
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
The current call for pediatricians to address the "millennial morbidities" of today's children requires involvement with sustainable and productive community partnerships. The HTPCP has created sustainable programs, most of which could be contacted many years after the initial funding was complete. Our data suggest that the process of program evaluation may play an important role in the success and sustainability of community-based projects.
HTPCP programs are required to include an evaluation plan in their proposals for funding; however, HTPCP provides no evaluation-specific funding or monitoring. All HTPCP programs expend money and effort on evaluation activities, but they have varying levels of success in identifying and assessing outcome indicators. Although few programs that were funded through HTPCP have reported results of outcome evaluations in the scientific literature, most program directors reported having specific impacts for their service populations and their communities. Programs that reported at least 1 outcome were significantly more likely to report that the program had been sustained past the initial 5-year funding period. There are several possible explanations for this finding. Programs with outcome data may be better able to demonstrate a track record to potential funders and community supporters. The process of evaluation itself may be important in the sustainability of a program. Programs with demonstrated outcomes have gone through the sometimes difficult process of specifying measurable goals and objectives, which should have helped to guide implementation and refinement of the interventions. The ability to measure outcomes may also reflect better overall program design and planning. Successfully sustaining a program thus reflects multiple and interactive factors: better process and planning as well as the gathering of data that are useful for generating support of the program. Additional prospective study of the interrelationships among meaningful program process evaluation, adequate outcome evaluation, program quality, and program sustainability would help to clarify the aspects of program design that are most likely to lead to sustainable outcome.
This study did not collect data to examine further the underlying mechanisms for the association between reporting of outcomes and program sustainability, and this finding should be viewed cautiously. The cross-sectional data were collected by self-report from program directors; findings therefore rely on individual perception and memory. Moreover, the quality and the nature of the outcome measures were not assessed, specific findings were not reported, and other reports of outcomes were not generally available for corroboration. Lack of a good or an excellent program evaluation may reflect numerous other problems in the structures of organizations and coalitions. Moreover, other factors, such as matching requirements, long-term (5-year) funding support, and the leadership of a program champion, may play a role in sustainability.20,21 Despite these limitations, the survey results suggest an important link between successful evaluation of a community-based project and its ultimate success.
As a condition of funding, HTPCP proposals must include an evaluation plan. More than half of the HTPCP community-based health projects reported that their evaluation plans were changed in the course of implementing their project. The most frequently cited reasons for change were improving the evaluation, discovering that the plan was not feasible, or learning that anticipated data were not available. Some projects struggle for several years to identify appropriate outcomes and indicators, as well as strategies for data collection and analysis. Identified barriers to evaluation were organizational, methodologic, and financial. Although a majority of projects were able to identify and use resources for evaluation, some reported that limited staff resources or staff turnover inhibited careful assessment of their projects. More than 25% of programs reported a lack of evaluation expertise, and many encountered issues with the feasibility or the scale of their evaluation plans. Technical assistance helped to ameliorate this problem through site visits, contacts with HTPCP staff, or evaluation workshops and presentations at grantees' meetings. Those who reported receiving this technical assistance found it helpful and were able to specify the areas of evaluation in which they were helped. Financial factors generally reflected the limitations of the funding program. Project budgets are relatively small ($50000), making it a challenge to include evaluation as a line item.
Despite the reported barriers to evaluation, 50% of program directors believed that their projects were well evaluated, and 83% reported that their evaluations had produced useful information. Projects had used this information for important project and community functions, such as improving services, advocating for their service populations, obtaining funding, supporting replication, and promoting policy change.
|
CONCLUSIONS |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
Evaluation of community-based programs is challenging. Programs that are designed to address pressing community needs seldom have sufficient funding to conduct evaluations that will prove their benefit with scientific rigor. The "millennial morbidities" have multiple and complex causes that are part of the social fabric of the communities in which we function; community partnerships operate on a margin that seldom allows the detailed evaluation needed in the scientific literature. These data suggest that the process of evaluation is beneficial to program quality and help to ensure that the partnership meets the needs of the target population beyond the initial funding period. A combination of specific, realistic requirements and more extensive technical assistance may both reduce the burden and enhance the results of evaluation efforts by community-based organizations. Making evaluation an integral part of community-based programs requires an initial investment of time and money by both program staff and funders, but the potential for lasting benefits for programs and communities is substantial.
|
FOOTNOTES |
|---|
Accepted May 30, 2008.
Address correspondence to Holly Ruch-Ross, ScD, 9345 Avers Ave, Evanston IL 60203. E-mail: hruchross{at}aol.com
The authors have indicated they have no financial relationships relevant to this article to disclose.
| What's Known on This Subject Although program evaluation is frequently required by funders, little is known about the capacity of community-based programs for evaluation or about the value of evaluation in supporting program success and sustainability.
|
| What This Study Adds We used survey methods to examine community-based programs and experience of evaluation, both in terms of capacity and benefits to the program. The findings are encouraging and suggest a need for additional study of the value of evaluation for community-based programs.
|
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODSRESULTSDISCUSSIONCONCLUSIONSREFERENCES |
|---|
- Satcher D, Kaczorowski J, Topa D. The expanding role of the pediatrician in improving child health in the 21st century.
Pediatrics. 2005;115
(4 suppl):1124
–1128
[Abstract/Free Full Text] - Palfrey JS, Tonniges TF, Green M, Richmond J. Introduction: addressing the millennial morbidity—the context of community pediatrics.
Pediatrics. 2005;115
(4 suppl):1121
–1123
[Free Full Text] - Goldhagen J. Integrating pediatrics and public health.
Pediatrics. 2005;115
(4 suppl):1202
–1208
[Abstract/Free Full Text] - Maternal and Child Health Bureau, Health Resources and Services Administration. Available at: www.mchb.hrsa.gov. Accessed June 4, 2006
- Community-Campus Partnerships for Health. 2005 Community-Campus Partnerships for Health Award Recipient. Available at: http://depts.washington.edu/ccph/awards2005-CPT.html. Accessed June 4, 2006
- Commonwealth of Massachusetts, Department of Public Health, Bureau of Family and Community Health. Care coordination for children with special health care needs. Available at: http://mass.gov/dph/fch/care/index.htm. Accessed June 30, 2006
- Robert Wood Johnson Foundation. About RWJF. Available at: www.rwjf.org/about/index.jsp. Accessed June 4, 2006
- W.K. Kellogg Foundation. Available at: www.wkkf.org. Accessed May 30, 2006
- Grason H, Aliza B, Hutchins VL, Guyer B, Minkovitz C. Pediatrician-led community child health initiatives: case summaries from the evaluation of the CATCH program.
Pediatrics. 1999;103
(6 pt 3):1394
–1419
[Abstract/Free Full Text] - Goodman RM. Principles and tools for evaluating community-based prevention and health promotion programs. J Public Health Manag Pract. 1998;4 (2):37 –47[Medline]
- Schweigert F. The meaning of effectiveness in assessing community initiatives. Am J Eval. 2006;27 (4):416 –436[CrossRef]
- Quinn DC. Proving the effectiveness of community programs in jeopardy: the role of formative evaluation. Qual Manag Health Care. 2001;9 (4):33 –46[Medline]
- Aspen Institute. The Aspen Measures page. Available at: www.aspenmeasures.org/index.html. Accessed July 20, 2006
- Evaluation Unit, W.K. Kellogg Foundation. W.K. Kellogg Foundation Evaluation Handbook. Battle Creek, MI: W.K. Kellogg Foundation; 1998
- W.K. Kellogg Foundation. The evaluation toolkit page. Available at: www.wkkf.org/default.aspx?tabid=75&CID=281&NID=61&LanguageID=0. Accessed July 20, 2006
- Carman JG. Evaluation practice among community-based organizations: research into the reality. Am J Eval. 2007;28 (1):60 –75[CrossRef]
- Chaskin RJ. Lessons Learned From the Implementation of the Neighborhood and Family Initiative: A Summary of Findings. Chicago, IL: Chapin Hall Center for Children at the University of Chicago; 2000
- Napp D, Gibbs D, Jolly D, Westover B, Uhl G. Evaluation barriers and facilitators among community-based HIV prevention programs. AIDS Educ Prev. 2002;14 (3 suppl A):38 –48[CrossRef][Web of Science][Medline]
- Ruch-Ross H, Miller N. The Healthy Tomorrows Partnership for Children Program: Highlights and Lessons Learned From the National Evaluation. Elk Grove Village, IL: American Academy of Pediatrics; 2006
- Robertson LC. Sustainability of Healthy Tomorrows Partnership for Children Program [dissertation]. College Park, MD: Faculty of the Graduate School, University of Maryland; 2005
- Scheirer MA. Is sustainability possible? A review and commentary on empirical studies of program sustainability. Am J Eval. 2005;26 (3):320 –347[CrossRef]
PEDIATRICS (ISSN 1098-4275). ©2008 by the American Academy of Pediatrics
CiteULike 
Connotea 
Del.icio.us 
Digg 
Facebook 
Reddit 
Technorati 
Twitter What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
