BACKGROUND: Research on resident workloads has focused primarily on the quantity of hours worked, rather than the content of those hours or the variability among residents. We hypothesize that there are statistically significant variations in resident workloads and better understanding of workload intensity could improve resident education.
METHODS: The Stanford Children’s Health research database was queried for all electronic notes and orders written by pediatric residents from June 2012 to March 2014. The dataset was narrowed to ensure an accurate comparison among residents. A survey was used to determine residents’ self-perceived workload intensity. Variability of total notes written and orders entered was analyzed by χ2 test and a Monte Carlo simulation. Linear regression was used to analyze the correlation between note-writing and order-entry workload intensity.
RESULTS: A total of 20 280 notes and 112 214 orders were written by 26 pediatric interns during 6 core rotations between June 2012 and June 2013. Both order-entry and note-writing workload intensity showed highly significant (P < .001) variability among residents. “High workload” residents, defined as the top quartile of total workload intensity, wrote 91% more orders and 19% more notes than “low workload” residents in the bottom quartile. Statistically significant correlation was observed between note-writing and order-entry workload intensity (R2 = 0.22; P = .02). There was no significant correlation between residents’ self-perceived workload intensity and their objective workload.
CONCLUSIONS: Significant variations in workload exist among pediatric residents. This may contribute to heterogeneous educational opportunities, physician wellness, and quality of patient care.
- ACGME —
- Accreditation Council for Graduate Medical Education
What’s Known on This Subject:
Previous research on resident workloads has focused primarily on the quantity of hours worked, rather than the content of those hours. Workload variability is similarly understudied. Better understanding of resident workload intensity and variability could improve trainee education.
What This Study Adds:
Resident workloads can be quantified through large-scale analysis of electronic medical records. There exist significant workload variations among pediatric residents. Objective workload analyses could supplement Accreditation Council for Graduate Medical Education milestones and improve resident education.
Physician workload, especially among trainees, has been an active area of research since the Accreditation Council for Graduate Medical Education (ACGME) duty hour standards were implemented in 2003. Numerous studies have linked resident workload to wellness and education1–6; however, the vast majority of existing research on resident workloads has focused on the quantity of hours worked, rather than the content of those hours. Additionally, most research has studied self-reported hours data, which can be unreliable.
Objective data are sparse: a systematic literature review by the ACGME in 2010 identified 807 articles published on resident duty hours and related topics.7 Of these publications, only 2 analyzed the number of orders entered by clinicians (3101 orders were studied by Stahlfeld et al8 and 8195 orders were studied by Hendey et al9), 2 quantified average patient census,10,11 and none analyzed the number of notes written. Residents spend a significant amount of time on clinical documentation, as demonstrated by time motion studies dating back to 1961.12⇓–14 Recent publications repeat these findings: 3 studies conducted since 2012 show that interns spend between 20% and 40% of their time in front of computers, on tasks such as note-writing, order-entry, and other indirect patient care involving electronic health records.15–17 We propose that notes and orders make accurate markers for physician workload, as they are objective, numerous, and easily quantified.
Resident workload variability is also poorly understood. Previous studies on resident workload variability were conducted in the context of evaluating “clouds”: the concept that some physicians, who are said to have a “black cloud,” experience a more burdensome quantity or quality of work than their colleagues.18–20 Although this phenomenon is often discussed and debated among physician trainees, it remains unproven. An analysis by Walling18 examined internal medicine interns and reported significant variability in the number of patients admitted on call nights. However, studies by Meyr et al19 on podiatry residents and Tanz and Charrow20 on pediatric interns found no significant variation in workloads. The existence and extent of resident workload variability remains unclear.
We hypothesize that there are statistically significant variations in resident workloads that could have a substantial impact on resident education, physician wellness, and patient care. We aimed to quantify resident workload variability by analyzing and comparing the workload of Stanford Children’s Health residents, as measured by the number of notes and orders written.
An objective calculation of resident workload was quantified by leveraging the electronic medical records system at Stanford Children’s Health. The Stanford Translational Research Integrated Database Environment database was queried for all electronic notes and orders written by pediatric residents at Stanford Children’s Health from June 2012 to March 2014. This yielded a de-identified database of 299 133 notes and 3 253 990 orders written by 83 residents.
To ensure an accurate comparison among residents, the dataset was narrowed to the 20 280 notes and 112 214 orders written by 26 residents during 6 core inpatient rotations of their intern year: general pediatrics, gastroenterology, night float, pediatric intermediate care nursery and well-infant nursery, pulmonology and endocrinology, and nephrology and rheumatology. The data selection process is described in Fig 1. These notes and orders represented, respectively, 85% and 94% of the total notes and orders written by these residents throughout the 8 months of intern year spent at Lucile Packard Children’s Hospital. The dataset spanned from June 2012 to June 2013.
Outcome Measures: Workload Intensity
Resident workload was quantified by calculating the percentage of total notes written by each resident on each rotation and in aggregate, as shown in Equation 1: Workload Quantification. The calculation was repeated with order-entry data.
(number of notes written by resident in given rotation)/(total number of notes written by all residents in that rotation)
Each resident’s workload was compared with average to quantify workload intensity, as shown in Equation 2: Note-Writing Workload Intensity.
(total number of notes written by a given resident)/(average number of notes written by residents)
To calculate each resident’s overall workload intensity, the note-writing and order-entry workload intensities were averaged, as shown in Equation 3: Overall Workload Intensity.
Avg [(total number of notes written by resident)/(average number of notes written by residents) + (total number of orders entered by resident)/(average number of orders entered by residents)]
The inter-resident variability of total notes written and orders entered was analyzed by χ2 test and a modified Monte Carlo simulation. Linear regression was used to analyze the correlation between note-writing and order-entry workload intensity.
Survey: Self-Perceived Workload Intensity
To obtain subjective data about residents’ self-perceived workload, interns were surveyed regarding their own workload intensity, as described by cloud status. Twenty-two of 26 residents responded with a rating between 1 and 10, with 1 corresponding to a “very white cloud” and 10 a “very black cloud.” The survey was approved by the Stanford University institutional review board under protocol ID 29539.
Tables 1 and 2 shows the number of notes written and orders entered by rotation and in aggregate. The total number of notes written per intern in core rotations ranged from 595 to 945 (mean = 780, SD = 80). The total number of orders entered per intern ranged from 1776 to 6004 (mean = 4316, SD = 1057). On average, the high workload residents, defined as the top quartile of combined note-writing and order-entering workload intensity, wrote 91% more orders and 19% more notes than the low workload residents in the bottom quartile.
The intensity of each resident’s workload was plotted by rotation. Figure 2A shows the percentage of total orders entered by each resident, by rotation and in aggregate. Figure 2B shows data for note-writing. There was statistically significant order-entry and note-writing workload variability when analyzed by rotation and in aggregate, as calculated by χ2 test (P < .001).
Each resident’s note-writing workload intensity was also compared with her or his order-entry workload intensity, as shown in Fig 3. Interns in the right upper quadrant wrote an above-average number of notes and entered an above-average number of orders during their intern year. Residents in the left lower quadrant had a below-average workload for both note-writing and order-entry. There was a statistically significant correlation between note-writing and order-entry workload intensity as calculated by linear regression analysis (R2 = 0.22, P = .02).
An alternative description of workload intensity is shown in Fig 4, which depicts a workload intensity index that combines the note-writing data and order-entering data, as described in Equation 3. This index can also be described as the distance from origin for each data point in Fig 3. This methodology accounts for the magnitude of work completed by each resident and describes incremental differences between residents. We defined high, medium, and low workload intensity resident groupings using this methodology.
Survey: Self-Perceived Workload Intensity
Figure 5 shows the correlation between each resident’s objective workload intensity and his or her subjective, self-assigned workload assessment. Linear regression analysis did not show statistical significance (R2 = 0.14, P = .09). Interestingly, residents avoided the extremes when reporting their workload intensity. The range of reported values was 4 to 8, with a mean of 5.5 and a median of 5, suggesting that most residents believe their workload was close to average.
We present a novel, quantitative, large-scale analysis of physician trainee workload. Our investigation indicates that there is significant intra- and inter-rotation variability in resident workload, as measured by note-writing and order-entry data. On average, the high workload residents entered 91% more orders and wrote 19% more notes than the low workload residents.
Although most previous studies have not found a significant difference in workload variability among residents, these studies may have been limited in scope. In our analysis, we studied more resident shifts than the combined total of all previous studies. Furthermore, our use of notes and orders as markers for resident workload enabled the analysis of many more data points. Additionally, although available research on resident workloads has focused on self-reported data, such as duty hours or pages received, which may be unreliable, our study characterizes strictly objective data obtained from electronic medical records.
Although a root-cause analysis of why resident workload differences exist is outside the scope of this study, one traditional explanation could be external factors, such as the seasonality of disease. Izurieta et al21 found that infants and young children are at increased risk for hospitalization during influenza seasons, with subsequent reasoning that high workload residents may have been scheduled on core rotations during busier winter months. However, our data do not support this explanation. When we compared total resident workload accrued during the months of March to August versus September to February, the average number of orders entered and notes written differed by only 3.4% and 1.8%, respectively. Another traditional thought is that variable staffing ratios could affect the workload distribution. However, the resident workforce schedule was uniform during the study period.
Relevant to the discussion, we also offer the possibility that workload variability may be a function of trainees themselves. Although high workload residents could be receiving a disproportionate share of educational opportunities by random chance, they might be hypervigilant or inefficient workers who “create work” for themselves. This explanation is consistent with findings from Tanz and Charrow,20 who found that the reputation and self-perceived workload of “black cloud” residents had an inverse correlation with hours of sleep, but not actual workload. Alternatively, high workload residents may be effective workers who are covering for inefficient or inexperienced colleagues on team-based rotations. Similarly, although low workload residents may simply be missing out on educational opportunities, they also could be inefficient workers who are “shielded” from normal appropriate workloads by their coresidents and supervisors due to poor performance.
Ultimately, our data show that workload distribution among physician trainees is unequal. In an ideal training environment, the workload intensity distribution in core rotations would be uniform, to allow equal learning opportunities and clinical service burden. Our findings yield concerns for both high and low workload residents. In either case, there exist opportunities for residency training programs to optimize their workload distribution and for residents to improve the way they approach and process their work.
Of note, linear regression analysis did not find a statistically significant correlation between objective and self-perceived workloads. These findings are consistent with studies by Myer et al19 and Tanz and Charrow,20 and suggest that the “black cloud” phenomenon is not a function of actual workload. Given the inverse association between cloud status and sleep described by Tanz and Charrow,20 the cloud phenomenon may be driven by other factors, such as a trainee’s emotional interpretation of his or her workload.
Implications for Program Directors
Our results can inform the curriculum development and trainee assessment tools on which residency program directors rely. The historical approach to resident assessment has been largely subjective, relying heavily on descriptive feedback. The ACGME Pediatrics Milestone Project, implemented in 2014, was a significant step toward the creation of valid, standardized educational goals and the achievement thereof.22
The availability of an objective workload report, as described in our data, could supplement the Milestone Project by offering a reproducible, quantifiable, and impartial means of assessing resident activities. For example, one of the ACGME Pediatrics Milestones within the Patient Care section is, “Organize and prioritize responsibilities to provide patient care that is safe, effective, and efficient.”23 If there is concern about a resident’s ability to complete her or his patient care responsibilities, it would be helpful to have an objective description of that resident’s workload intensity. A resident burdened by an unusually large workload requires different support and interventions than a resident who is struggling with a normal or below-average workload. Objective data regarding residents’ workload could help inform resident development and the achievement of ACGME Pediatrics Milestones in all ACGME competency areas.
Our workload intensity quantification tool has numerous advantages compared with traditional resident workload measurement techniques. However, there are limitations to our analysis and its application. Although physician trainees spend a significant and ever-increasing amount of time on computer-oriented tasks, such as note-writing and order-entry, they have many other responsibilities that are not captured in our research. Furthermore, our analysis describes only the quantity of notes and orders written, not the complexity or time requirements thereof. Our data also do not capture the extent or influence of task-sharing, which can occur between residents on inpatient rotations at Stanford Children’s Health.
Additionally, as all physicians know well, one particularly sick or challenging patient can often require the same amount of time and attention as 10 other patients. We considered including variables that would capture this aspect of patient care, such as code blues, rapid responses, or patient acuity as measured by Pediatric Early Warning Score, International Classification of Diseases, Ninth Revision codes, and problem lists, but ultimately deferred due to limited data availability. We also considered including other quantifiable markers for resident workload, such as phone calls and text pages, but declined due to poor data quality. Future analyses of resident workload would be improved by the inclusion of these data. Last, although our data set is large, comprising tens of thousands of data points from >2 dozen residents, our analysis would have benefitted from additional resident cohorts, including those at other institutions.
Nevertheless, the availability of our workload quantification tool suggests numerous interventions. Workload distribution could be directly affected by adjusting resident schedules based on historical workload: a low workload resident in need of more educational opportunities could be given extra clinical exposure. Alternatively, a resident who has been receiving a disproportionately high workload could be placed on less clinically demanding rotations in following years. Having an objective measure of workload can also provide additional context to help program directors better understand residents’ abilities, particularly in the areas of efficiency, ability to ask for help, and professionalism.
A reliable measurement of resident workload could also inform physician wellness. Numerous studies across multiple specialties have described a correlation between resident workload and burnout or emotional exhaustion.24–26 A better objective understanding of resident workload could help identify trainees at risk for burnout. Similarly, it would be instructive and useful to analyze correlations between resident workload and patient outcomes to guide workforce deployment.
We describe significant variations in workload among pediatric residents. This disproportionate resident workload may contribute to heterogeneous educational opportunities, physician wellness, and quality of patient care.
The authors thank Dr Gomathi Krishnan and Dr Todd Ferris for their assistance with data collection and Dr Lu Tian for his assistance with statistical analysis.
- Accepted April 20, 2016.
- Address correspondence to Adam Was, MD, Department of Pediatrics, Stanford Children’s Health, 725 Welch Rd, Palo Alto, CA 94304. E-mail:
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
FUNDING: No external funding.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
- Wagner MJ,
- Wolf S,
- Promes S, et al.
- Fletcher KRD,
- Aroroa V
- Meyr AJ,
- Gonzalez O,
- Mayer A
- Schumacher DJ,
- Spector ND,
- Calaman S, et al
- Carraccio C
- Landrigan CP,
- Fahrenkopf AM,
- Lewin D, et al
- Hutter MM,
- Kellogg KC,
- Ferguson CM,
- Abbott WM,
- Warshaw AL
- Copyright © 2016 by the American Academy of Pediatrics