Putting the Pediatrics Milestones Into Practice: A Consensus Roadmap and Resource Analysis
The Accreditation Council for Graduate Medical Education has partnered with member boards of the American Board of Medical Specialties to initiate the next steps in advancing competency-based assessment in residency programs. This initiative, known as the Milestone Project, is a paradigm shift from traditional assessment efforts and requires all pediatrics residency programs to report individual resident progression along a series of 4 to 5 developmental levels of performance, or milestones, for individual competencies every 6 months beginning in June 2014. The effort required to successfully make this shift is tremendous given the number of training programs, training institutions, and trainees. However, it holds great promise for achieving training outcomes that align with patient needs; developing a valid, reliable, and meaningful way to track residents’ development; and providing trainees with a roadmap for learning. Recognizing the resources needed to implement this new system, the authors, all residency program leaders, provide their consensus view of the components necessary for implementing and sustaining this effort, including resource estimates for completing this work. The authors have identified 4 domains: (1) Program Review and Development of Stakeholders and Participants, (2) Assessment Methods and Validation, (3) Data and Assessment System Development, and (4) Summative Assessment and Feedback. This work can serve as a starting point and framework for collaboration with program, department, and institutional leaders to identify and garner necessary resources and plan for local and national efforts that will ensure successful transition to milestones-based assessment.
- ACGME —
- Accreditation Council for Graduate Medical Education
- CCC —
- clinical competency committee
At its core, competency-based education seeks to link learner education and assessment to training outcomes needed to care for patients.1 The Accreditation Council for Graduate Medical Education (ACGME) first partnered with the American Board of Medical Specialties to move in this direction in the late 1990s, defining 6 competency domains.2 Taking the next step in advancing competency-based education and assessment, the ACGME again partnered with the American Board of Medical Specialties to create the Milestone Project, which further delineates development within each competency area and defines expected “milestones” of achievement.3 In pediatrics, 4 to 5 developmental levels of performance, or milestones, have been developed for each of the 48 competencies within the 6 ACGME competency domains.4–7 Through efforts to focus on a more practical number, 21 of these competencies have been selected for initial reporting purposes.6,8 Beginning in June 2014, all pediatrics residency programs will be required to report individual resident progress on these competencies to the ACGME semiannually.3
Implementing the Pediatrics Milestones: An Adaptive Challenge
Implementing milestones-based assessment focused on achieving the necessary outcomes of training is an adaptive challenge, requiring a paradigm shift in which simply tweaking the current system will not suffice.9,10 This is true of the broader landscape of 21st century medical education, which must evolve to meet the changing competencies required of physicians.11,12 As an adaptive challenge,9,10 several important truths emerge about implementing milestones:
ways to change will not always be easy to identify and will require innovative thinking and willingness to abandon what is tradition and feels most comfortable,
necessary changes will require fundamental shifts in approach to learner assessment so that training outcomes match the needs of patients and society,
commonly held values and beliefs about assessment must change to ensure methods actually help learners develop into physicians with necessary skills and abilities,
those on the front lines of assessment as well as those responsible for learner assessment at the program level must share the responsibility of determining how change will occur to facilitate successful implementation,
changes in several areas of the working and learning environment must occur to ensure experiences develop competencies required of 21st century physicians and not just those valued by outmoded faculty, and
new study and experimentation must be undertaken for successful change to be achieved.
For residency programs, most of which have limited resources and funding, there is no clear path to meet these challenges. National efforts in pediatrics seek to develop new data management and assessment systems, with a notable focus on assessment methods and validity study.13 Although these efforts are of tremendous benefit and hold unparalleled promise for the future, individual programs must determine how to both contribute to such multi-institutional efforts while simultaneously making changes in their own residency programs until such resources are available to address the needs that accompany milestones-based assessment implementation and the shift toward focusing on attaining the outcomes important to patient care. To address this issue, 13 pediatric residency leaders from 6 programs representing a range of program sizes (see Table 1) developed a consensus view of necessary requirements to fully implement and sustain assessment and reporting based on the Pediatrics Milestones. In this article, we report this view, with explanation of rationale and importance. The goal for this work is to serve as a roadmap to inform the efforts and resource allocations of program, department, and institutional leaders as they seek to implement and sustain milestones-based assessment.
Development of Consensus Roadmap and Resource Estimates
From November 2012 until July 2013, the authors developed a consensus roadmap for implementing and sustaining milestones-based assessment in pediatrics residency programs. We began by brainstorming the necessary components and then used an iterative review process. Through this work, we determined the components for implementing and sustaining this effort fall into 4 major domains with subcategories as shown in Fig 1: (1) Program Review and Development of Stakeholders and Participants; (2) Assessment Methods and Validation; (3) Data and Assessment System Development; and (4) Summative Assessment and Feedback. Some activities identified already have time dedicated to them in current residency structure; however, many components, or at least their requirements, are new.
To place additional context around the required activities, authors representing each program estimated the time resources that will be required to implement and sustain efforts in each domain. We used the Delphi method to repeat the estimates until no to little change was observed in the repeat estimates from previous ones (1–4 iterative cycles per domain). For each round of estimates, all programs provided time estimates for their own program for each topic in each domain (see Fig 1) for both implementing and sustaining efforts for each of the following categories of people: (1) program leaders, (2) rotation directors and faculty, and (3) staff. Some steps had only implementing or sustaining components, whereas most had both. Implementation estimates were provided as 1-time efforts to begin milestones-based assessment, and sustaining estimates were provided as yearly totals after successful implementation.
Table 2 shows the total time estimates for each domain for each program (Fig 2). Our 2 largest programs had the lowest estimate per resident, whereas our 2 smallest programs had the highest estimate. Domains 1 and 4 were estimated to require the most effort, with domain 3 requiring the least. Domain 3 may be the lowest because, unlike other domains, it is composed of activities that focus primarily on implementation, with little sustaining effort. We believe the range of time estimates represent appropriate variation based on differences in program size, institutional structure, financial support, educational philosophy, and approach to implementing milestones-based assessment.
Although this consensus effort was developed by residency program leaders, future vetting and collaboration with department chairs, designated institutional officials, hospital and medical school administrators, and trainees across the educational continuum will be necessary to make the broad-based, sweeping changes needed. However, we feel our current effort provides the foundation on which future collaboration can be built as well as a reasonable starting point and framework to begin to garner resources to help program leaders plan their own efforts as well as participate in national efforts that benefit all. To provide meaningful context to these activities, we devote the remainder of this article to discussing each of the 4 domains identified and provide an analysis of the component steps, importance, and resources needed to achieve them.
Domain 1: Program Review and Development of Stakeholders and Participants
Mapping Milestones to Educational Experiences
Implementing milestones-based assessment requires focus on 2 important areas: (1) optimizing utility of assessment, and (2) aligning areas of assessment with desired training outcomes. Focusing on the former, van der Vleuten proposed that the utility of an assessment method arises from the product of its acceptability, educational impact, validity, reliability, and cost, to which others have proposed adding feasibility as a sixth variable.14,15 Applying this framework, we suggest limiting the number of competencies assessed during each educational experience, such as a rotation or continuity clinic, to optimize the acceptability and feasibility of milestones-based assessment among busy assessors with limited time and variable motivation. This will also allow the alignment of competencies to the educational experiences where they are best developed and assessed while ensuring a more complete composite when individual educational experiences are combined to describe learners’ development every 6 months. Beyond faculty- and program-driven goals and objectives, this mapping must consider the necessary and desired training outcomes to meet patients’ needs and map competencies that correlate with those outcomes to educational experiences in which residents are most likely to develop them and assessors are most likely to observe them.1 For example, the continuity clinic is likely well-positioned to develop and assess the milestones in the “coordinate patient care within the health care system relevant to their clinical specialty” competency of the systems-based practice competency domain.
Faculty and Resident Development
One of the strengths in the current iteration of the Pediatrics Milestones is their ability to provide a shared mental model for what physician development looks like.6 This provides a learning roadmap to inform curriculum as well as supervisors’ efforts to help trainees take the next steps in their development.16–18
Milestones-based assessment is a paradigm shift from current assessment methods that rely heavily on faculty judgments of resident performance based on their level of training.19 With milestones, assessors will be required to match observed behaviors in trainees with descriptions of those behaviors along a developmental milestones trajectory independent of training level. Therefore, a developmentally “on target” resident may appropriately be at the third of 5 milestone levels for some competencies at the completion of his or her residency. Educating residents and faculty about this developmental approach to assessment will be critical and require significant time and effort. Fortunately, initial evidence suggests that both residents and faculty understand the milestones-based assessment framework when oriented to it.18,20,21
Staff and Other Assessor Development
Program and support staff will need to be trained to administer milestones-based assessment systems, provide support to faculty leaders and assessors, and help answer questions from residents and faculty regarding the program’s milestones-based assessments. In addition, members of the interprofessional health care team as well as patients and families should ideally assess competencies they are well positioned to observe, such as communication skills, professionalism, and humanism. To achieve this, these groups will need to be trained or oriented to use assessment tools, which underscores the importance of developing tools that are clear, focused, and easily understood.
Clinical Competency Committee Formation and Activity
Performance is context-dependent, so trainees will display different levels of milestone development for the same competency in different educational experiences. Culling the range of assessment data across experiences, deciding what milestone level will be reported to the ACGME every 6 months, and determining if residents are developmentally on track, precocious, or delayed will be the responsibility of clinical competency committees (CCCs).3 We recommend the ideal composition for the CCC would include program leaders, core faculty, noncore faculty, and potentially nonphysician members to represent the range of perspectives and baseline knowledge of residents. This composition would hope to balance intimate knowledge of residents to flesh out performance details as well as unbiased “outsider” observations that could otherwise be unintentionally missed or minimized.
The CCC will need to meet with sufficient regularity to review milestones-based assessment data, track residents, and ensure those who are in need of additional assistance are receiving the help necessary to advance their development.
Additional responsibilities that could be considered for the CCC include assisting in, or even being the primary drivers of, promotion, graduation, and remediation decisions, as well as recommending and monitoring professional development plans. Given their unique perspective to identify common deficiencies across a residency program, the CCC also could serve a role in program improvement efforts, working closely with and informing the new Program Evaluation Committee.22
Educate Departmental and Institutional Leaders/Garner Resources
As discussed previously, advancing competency-based assessment using milestones is an adaptive challenge, not a technical problem.9,10 Therefore, the scope and breadth of this effort will require personnel, time, and financial resources at the local and national levels to achieve success. On the local level, departmental and institutional leaders will need to support resident and faculty development, modifications of the working and learning environment to facilitate the execution of new efforts and focus on new and emerging competencies of physicians required to care for patients in the 21st century, and time for faculty to participate in “innovation incubator” work as well as contribute to national efforts to develop assessment tools and systems. Perhaps most importantly, there must be a plan for how this support will be sustained. On the national level, organizations with an investment in milestones-based assessment and training outcomes must provide mechanisms and funding for multi-institutional study and experimentation. The study efforts of the Association of Pediatric Program Directors and the National Board of Medical Examiners address the use of milestones to assess readiness to care for patients in an inpatient setting and are now broadening their scope to other core areas of physician practice, demonstrating the commitment and investment needed from national organizations.6,21
Domain 1: Bottom Line Practical First Step
This first domain has several components. An “easy win” and important first step could be mapping competencies to educational experiences. This aligns competencies to where they are best developed and assessed, which is important to achieving desired training outcomes. It is also relatively easy to complete. A focused meeting between rotation directors and program leaders can identify the competency areas for each educational experience. This partnership ensures rotation directors are assigned areas they feel best primed to assess and develop while also guaranteeing all areas are adequately covered across the entire residency curriculum.
Domain 2: Assessment Methods and Validation
Some milestones have rich evidence in the literature to build on, whereas others have a paucity of information informing how development should progress in that area.4,5 Thus, the milestones in their current form can stimulate discussions and reflections about physician development and competence; however, further study and adaptation will be needed before using them for high-stakes assessment and training year advancement decision-making.6 This study will need to include validating, and adapting where necessary, the behavioral descriptions that comprise the milestone levels and establishing normative ranges for the transitions from one milestone level to another.6 Studies will also need to generate evidence to inform the development of assessment tools that map back to the milestone levels for use in performing actual assessment. Perhaps most importantly, research will need to focus significant effort and resources on generating validity evidence to allow for the high-stakes interpretations of assessment data for which the Pediatrics Milestones are intended.6 As discussed previously, this work should focus not only on generating validity and reliability evidence, but also on acceptability, educational impact, cost, and feasibility.14,15 For example, an assessment tool that is too long and detailed will fail to be acceptable or feasible for the assessors; however, a tool that is overly simple so as to not collect meaningful information will fail to have optimal educational impact.
Study of the Pediatrics Milestones will be best carried out through multisite, national efforts, as only activities of this scope will garner the depth and breadth of evidence to produce transformative change. Fortunately, the Pediatrics Milestones are currently being studied in such a manner through a collaborative effort of the Association of Pediatric Program Directors Longitudinal Education Assessment Research Network and the National Board of Medical Examiners.21 Plans for future study include further collaboration with the American Board of Pediatrics.6,13
In the absence of nationally developed assessment tools with sufficient validity and reliability, individual programs must not only commit themselves to participate in the development of such tools through multi-institutional efforts, but also develop and/or identify currently available tools and mechanisms for assessing milestones in their programs until such tools are available. This dual effort represents an enormous challenge and time commitment.
Domain 2: Bottom Line Practical First Step
The focus in this domain needs to be on national, collaborative efforts. This is necessary to move the entire pediatrics community toward the common goal of truly meaningful assessment as quickly as possible. Although leading these efforts is not an “easy win,” participating in them is. For example, sharing milestone data and participating in multi-institutional study of assessment methods are examples of low-resource commitments on the local level that scale to large data sets on the national level if most programs participate.
Domain 3: Data and Assessment System Development
There is hope that future study will allow the competencies assessed to be combined or serve as proxies for one another.6 Absent this currently, assessment of milestones even for just the 21 reporting competencies has the potential to generate hundreds to thousands of data points for all residents across a program. These data will need to be collected efficiently, synthesized, and presented in a manner that is meaningful and facilitates review and interpretation by the CCC. With such a wealth of data to gather, there is a critical need for efficient data management. This effort will need to be supported by electronic assessment systems, ideally mobile applications that support real-time assessment on devices designed to facilitate data transfer to central servers for distribution to residents and faculty and for collation for CCC and program leader review.
Develop/Identify Data Management Systems to Collect and Report Milestones Assessment Data (Including Those With Novel Interfaces)
The systems that collect milestones-based assessment data may or may not be the same as those that assimilate the data. Commonly used current proprietary assessment systems focus on assimilating data from multiple assessment sources and creating summary reports. As program leaders, we hear frequent complaints from the end-users who complete assessment forms in these systems, such as log-in difficulties, suboptimal user interface and layout, and limited interface capabilities with smart phones and tablet computers. We believe optimal future systems will provide a clear layout with easy navigation on handheld electronic devices that feature saved passwords and wirelessly sync to central databases. Interfaces should be easy and intuitive to navigate, readily provide summary reports that synthesize and provide appropriate weight to performance assessments across settings and learning activities, and feature user-friendly layouts for use in both resident feedback and performance review by CCC members. Importantly, these systems also should have the ability to directly upload milestones reporting to external organizations, such as the ACGME. Although new systems provide promise, programs must still navigate with currently available resources.23
Develop Initial Standards for Milestone Achievement
Through study, normative ranges of milestones achievement as well as levels associated with critical transition points, such as becoming a supervising resident on an inpatient general pediatrics service, will be delineated.4,6 In the absence of this information currently, programs will need to establish and refine initial standards or thresholds in these areas to inform resident remediation, advancement, and graduation decision efforts. This is a significant task that should be approached with great care, as it will result in high-stakes decisions for resident advancement through training and into practice. We suggest that core faculty and CCC members work closely with program leaders to determine initial standards for individual educational experiences and the overall program.
Establish Policies and Professional Development Plans
For residents whose development falls behind established standards, professional development policies and mechanisms to ensure these residents are ready to be placed in specific educational experiences must be established. For example, a resident whose milestones-based development does not meet the standards for being a supervisor on the wards but has an impending ward supervisory rotation would need a schedule change and a professional development plan. This will require schedule flexibility and faculty members who are well versed in the development of professional development plans and in providing direct, specific feedback on current performance that is coupled with a discussion of future steps.
Develop Reporting Mechanism
The first reporting of milestones to the ACGME will occur in June 2014. This will require significant time for programs to enter information for 21 competency areas for all residents in their program at that time and every 6 months thereafter. Thus, a mechanism for local assessment systems to easily interface with national reporting systems to upload local data must be part of future assessment systems. Only systems fulfilling this need will support, rather than thwart, efforts to avoid administrative complexity that misappropriates time spent “checking the box” of milestones reporting with the more important effort of providing meaningful assessment information to residents. Although emerging systems provide promise in this area, programs must still navigate currently available resources.23
Domain 3: Bottom Line Practical First Step
The “easy win” in this domain is adapting professional development plans to incorporate the language of the milestones. After the first 1 or 2 reviews that CCC members and program leaders complete for all residents, each program will likely have these individuals as a core group of faculty that will be facile with the content and language of the milestones. This will allow them to use this knowledge to provide residents with the next steps to advance their developmental progression.
Domain 4: Summative Assessment and Feedback
Develop Processes to Provide Feedback and Make Advancement and Remediation Decisions
Feedback arising from milestones-based assessment data has the potential to facilitate the professional development of residents, underscoring the importance of presenting this information in a learner-centered manner. Although initial evidence provides insight into best practices,18,21 including emphasizing current and next steps in development, further study will be needed. Optimal milestones-based assessment will also require longitudinal relationships that allow adequate time and contact to identify current developmental level, define next steps in development, and effect behavior change from feedback.24–26
To address summative assessment, national studies are under way to identify normative ranges for milestone development that will inform common expectations.6 Until this information is available, each program must develop its own standards for advancement to new levels of responsibility and training, remediation for those whose performance falls below the lower limit of appropriate development, and criteria for graduation. We recommend training programs begin by adapting their current policies to incorporate milestones-based assessment data as an additional element to inform advancement and remediation decisions.
Use Milestones-based Assessment Data to Plan Individualized Curricula
The new ACGME requirements for pediatrics residency training include the equivalent of 6 months of individualized curriculum focused on career development for each resident.27 Although some competencies may be more or less important to certain careers, the milestones have great potential to inform the development of individualized curricula. A focus on the competency areas of greatest importance to residents’ future careers during individualized curricula will not only ensure that development in these areas is not lagging but also facilitate more rapid development to an advanced milestone level(s) in those areas.
Modify Educational Experiences Based on Deficiencies Uncovered
Milestones-based assessment will inherently uncover exemplar and deficient areas in residency programs based on common themes of where residents excel and struggle. This information should inform improvement efforts for residency programs with the goal of facilitating residents’ development. Assessment data collection and management systems should allow for these data to be easily displayed in aggregate format such as a dashboard. As noted previously, this information should be shared with the Program Evaluation Committee, necessitating discussions about potentially desirable overlapping work between the CCC and the Program Evaluation Committee.
Domain 4: Bottom Line Practical First Step
Similar to the previous domain, an early “easy win” for programs in domain 4 is using milestones to define and shape professional development plans for individual residents, providing residents with a tailored roadmap for their learning. Once CCC members and program leaders develop facility with the content and language of the milestones, this application to resident development will be readily made.
Conclusions and Next Steps
In this article, we provide a consensus opinion of residency program leadership for the domains of effort and time estimates that will be required to implement and sustain each domain. Although we recognize that the domains, and especially the time estimates, do not include the opinions of all important stakeholders, they do represent a good starting point for understanding what will be required to reach the full potential of milestones-based residency training. Implementing and sustaining milestones-based assessment represents an adaptive challenge of implementing and sustaining milestones-based assessment that will require significant effort and study. To achieve success, departmental and institutional leaders must support this work through time, money, and a commitment to achieving training outcomes that meet the needs of patients. Equally important, the pediatrics medical education community must engage in national sharing, study, and collaboration to develop valid, reliable, feasible, acceptable, and cost-effective methods of milestones-based assessment that have a meaningful educational impact by facilitating the professional development of residents.14,15 Stakeholder organizations should continue to take the lead in these efforts and help programs accomplish the necessary steps we have delineated. Without broad-based investment and resources, the next 10 years in competency-based assessment efforts risk mirroring the first decade in which slow progress was made. Furthermore, we may fail to reach our ultimate goal of determining learner training outcomes as the first step in linking educational and patient care outcomes.1
The authors thank and acknowledge Jennifer Vodzak, MD, of St Christopher’s Hospital for Children for her assistance with developing Figure 2.
- Accepted February 11, 2014.
- Address correspondence to Daniel J. Schumacher, MD, MEd, One Boston Medical Center Place, Boston, MA 02118. E-mail:
Dr Schumacher took the primary role in leading the effort, drafted the initial manuscript, and took the primary role in editing the manuscript; Drs Spector and Vinci helped lead the effort, and reviewed and revised the manuscript; Dr Calaman participated in all parts of the group effort, led the data analysis effort for the time estimates, and reviewed and revised the manuscript; Drs West, Cruz, Frohna, Gonzalez del Rey, Gustafson, Poynter, Rosenbluth, and Southgate participated in all parts of the group effort, and reviewed and revised the manuscript; Dr Sectish conceptualized this effort, helped lead the effort, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
COMPANION PAPER: A companion to this article can be found on page 917, and online at www.pediatrics.org/cgi/doi/10.1542/peds.2013-3827.
- ↵Pediatrics Milestone Project Working Group. The Pediatrics Milestone Project. 2012. Available at: https://www.abp.org/abpwebsite/publicat/milestones.pdf. Accessed February 12, 2012
- Heifetz RA,
- Linsky M
- ↵American Board of Pediatrics. Foundation Support of Milestones Study. 2013. Available at: https://www.appd.org/pdf/FoundationSupportofMilestonesStudy.pdf. Accessed February 25, 2012
- ↵Postgraduate Medical Education and Training Board. Developing and maintaining an assessment system—a PMETB guide to good practice. 2007. Available at: www.gmc-uk.org/Assessment_good_practice_v0207.pdf_31385949.pdf. Accessed February 27, 2014
- ↵Burke AE, Forbis S, Schumacher DJ, Lewis KO, Stolfi A. The Pediatrics Milestones: Obtaining Faculty Perspectives Through Cognitive Interviewing. Council on Medical Student Education in Pediatrics/Association of Pediatric Program Directors National Conference; 2013; Nashville, TN
- ↵Poynter S, Turner T, Chaffinch C, Hicks PJ, for the APPD LEARN-NBME Pediatrics Milestones Assessment Group. Learner Feedback on the Pediatrics Milestones Assessment Project. Orlando, FL: Accreditation Council on Graduate Medical Education; 2013
- ↵Data Commons, LLC. Data Commons home page. Available at: http://mydatacommons.org. Accessed December 3, 2013
- ↵Accreditation Council for Graduate Medical Education. ACGME program requirements for graduate medical education in pediatrics. 2013. Available at: www.acgme.org/acgmeweb/tabid/143/ProgramandInstitutionalGuidelines/MedicalAccreditation/Pediatrics.aspx. Accessed August 17, 2013
- Copyright © 2014 by the American Academy of Pediatrics