During my tenure as Chair of the Residents Forum Executive Committee and the CAP Residents Forum, we worked on a resolution to highlight the need to improve how graduated responsibilities are implemented in pathology training programs (see separate post). During the robust discussions at meetings, conferences and even a well attended webinar, several residents pointed out that before a trainee can request additional responsibilities, there must be a way for the resident to be deemed competent in certain areas. Current methods of evaluating trainees (e.g. ACGME milestones, program specific systems) were discussed and many people expressed serious concerns about the milestones. A recent twitter poll (Ok, it’s not publishable results but it’s composed of pathologists who I highly respect) showed that 79% of pathologists involved in resident education thought that the milestones needed major improvements or needed to be replaced all together! Only 16% considered the milestones adequate!
Considering all the concerns about the milestones, I thought it would be a good exercise for pathology residents to develop their own evaluation system. This new evaluation system wouldn’t be a requirement by any means but instead would serve as a guide post. The next time those at the upper echelons of pathology training revisit the topic of assessing resident competency during training, they can use this new system as a reference.
To start this process of designing an effective evaluation system, let’s start with the problem. I’ve found 6 major issues with the ACGME Pathology Milestones from talking with residents, faculty, and referring to my own experience as a pathology trainee:
1. Two scale system for every category: Every question has both a 5 scale and a 10 scale adding confusion and complexity. Solution: Use only one scale.
2. Single scale used by every category: Every category is forced to use the 10 scale system even when the category would be best evaluated with a different scale. Solution: Use the scale that best fits the question or category being evaluated. Examples: Professionalism (Yes or No, two scale). Diagnostic Concordance (Agree/Minor Discrepancy/Major Discrepancy, three scale).
3. Abundant subjectivity: Too many of the milestones categories are based on opinions. Those who may not get along well with certain faculty may suffer from a poor milestone evaluation even though their diagnostic skills are superb. Solution: Use objective evaluation tools that, by design, remove subjectivity and are inherently objective whenever possible.
4. Non-applicable categories: Giving a score to someone for something they don’t do wastes the time of the evaluator and confuses the recipient. It is inherently more subjective because the evaluator tends to base the made up score on how they feel about the recipient rather than on how they actually perform.
5. Too complex: Comparing/adding/taking averages of faculty evaluations is impossible when the results of the evaluation very widely due to difficulty interpreting the system. Complexity means the results carry little meaning to the recipient. Cannot be used to evaluate a program as a whole. Program interpretation of the milestones varies from program to program. Solution: Simplify, Clarify, Condense.
6. Evaluators have various (often very little) interaction with resident/fellow: Solution: Allow the evaluation system to be modified by any attending who interacts with the resident throughout the year. Also allow evaluators the ability to answer “N/A” to certain categories where they cannot properly evaluate the resident/fellow in certain categories.
I’m sure I’ve missed some things! What obstacles have you experienced in your program using the milestones?