pediatrics
April 2003, VOLUME111 /ISSUE Supplement E1

Using Organizational Assessment Surveys for Improvement in Neonatal Intensive Care

  1. G. Ross Baker, PhD*,
  2. Hannah King, MPH,
  3. Jeanne L. MacDonald, MD§,
  4. Jeffrey D. Horbar, MD§
  1. *Department of Health Policy, Management and Evaluation, University of Toronto, Toronto, Ontario, Canada
  2. Oakland, California
  3. §Vermont Oxford Network, Burlington, Vermont
  4. Department of Pediatrics, University of Vermont College of Medicine, Burlington, Vermont

Abstract

Background. Problems with organizational culture, lack of or poor team communications, and conflict are often seen as barriers to improvement efforts.

Methods. A survey measuring aspects of organizational culture was administered twice to staff in neonatal intensive care units participating in the Neonatal Intensive Care Unit Quality Improvement Collaborative Year 2000 collaborative. The surveys provided comparative data on coordination, teamwork and leadership, conflict management, unit leadership and unit culture. These data were summarized and fed back to NICU teams with guidance on their use. Interviews on the use of the survey were held with 12 medical directors and patient care leaders in 9 different NICUs.

Results. The findings indicated that all the units contacted saw themselves as committed to undertaking the organizational survey and using the results. Some units shared the data widely and initiated changes. Other units limited the distribution of data to the unit leadership. There was no apparent relationship between scores on the survey and activities undertaken. Several respondents credited the survey with helping to promote discussions about organizational and team issues.

Conclusions. Future use of the survey should include additional materials to assist in disseminating the results to staff.

  • organizational culture
  • organizational survey
  • collaborative quality improvement
  • teamwork
  • NIC/Q 2000

KEY POINTS OF ARTICLE

  • Organizational culture surveys can provide useful information on team and organizational issues that impede improvement.

  • Culture surveys can promote discussion of team issues and assist in identifying changes in policies, communications, and interactions that promote more effective team behaviors.

APPLYING LESSONS LEARNED TO PRACTICE

  • Unit leadership can use the survey results to compare performance with similar units and to identify specific projects aimed at improving their results on measures of unit culture.

  • Interpretation of the survey results is aided by the use of graphics that summarize complex measures and by materials that suggest how to present the survey to staff.

New approaches to quality improvement offer important tools to health care organizations, but the culture of organizations may influence their ability to use these tools effectively. In 1998, an organizational assessment survey was developed and administered to all 34 neonatal intensive care units (NICUs) that were participating in the Vermont Oxford Network Neonatal Intensive Care Unit Quality Improvement Collaborative Year 2000 (NIC/Q 2000). The survey and data provided NICU staff with feedback on organizational culture and behavioral issues, such as teamwork, leadership, and communication. Each center received a report on its scores and a comparison with the rest of the collaborative. This assessment was repeated in 2000. This article explores the development and rationale for the survey and assesses the experiences of those who use it to make changes in their units. The analysis of the survey results will be reported in a separate article.

BACKGROUND

Teams and organizations that attempt to implement quality improvement often identify organizational culture as an important barrier to or facilitator of success. Making improvements in health care requires learning how to identify problems, collect relevant data, select and test likely improvements, and analyze the impact of these improvements.1,2 Numerous courses have been developed to help individuals learn these skills, but not all organizations value such activities and support those who want to use them for improvement. This resistance to change may stem from individual skepticism or reflect more widely held values and beliefs about appropriate behaviors and activities in the organization. Such shared values and beliefs are often referred to as organizational culture.35 Researchers who have studied organizational culture suggest that these values and behaviors are products of organizational experience and influence many areas of organizational life. Because such cultures are deeply rooted and integral parts of organizational behavior, they are often difficult to change. Culture remains important, as several studies, including those by Shortell et al6 and Parker et al,7 have demonstrated the relationship of leadership and organizational culture to aspects of unit performance in health care organizations.

One method for helping organizations address organizational culture issues is survey feedback. Organizational development psychologists developed survey feedback methods as a means of stimulating change. These methods typically involve the development and use of an organizational survey, which is distributed among an organization’s staff and managers. Data from the survey are fed back to the organization and used to identify issues and stimulate identification of potential solutions.810 Burke et al11 noted that although the information provided by the survey is usually informative, the process of implementing, administering, and reporting back the data from an organizational survey is as important as the actual results. The survey provides important information about the current status of the organization and its environment and the central issues for organizational change. Just as important, the organizational survey offers a means of “unfreezing” the organization,12 preparing it to undertake new efforts to change.

Survey feedback has been used in a number of change initiatives in health care organizations. For example, Jones et al13 reported the use of an organizational culture assessment in conjunction with the implementation of patient focused care in a 500-bed nonprofit hospital. Results of surveys of hospital staff were used to help identify barriers to implementing a major restructuring of hospital care and patient services roles. In health care and elsewhere, such assessments are typically part of large-scale change initiatives and are directed by senior leadership. Front-line clinicians determined the change initiatives undertaken by the NIC/Q 2000 teams. There are no empirical studies of the use of organizational and team assessment tools to direct such changes developed at the unit level.

MEASURING ORGANIZATIONAL CULTURE IN NIC/Q 2000

Members of the Vermont Oxford Network, a group of several hundred NICUs, launched a benchmarking collaborative in 1995 to identify best practices in neonatal intensive care.14 The 10 centers that participated in that effort were successful in locating many practices to improve care in NICUs but also experienced some difficulties in transferring these practices to their own hospitals. On the basis of these observations, a second collaborative was initiated in 1998 to integrate rapid-cycle improvement efforts with the benchmarking methods, providing the NICU teams with the skills and knowledge to identify and test changes in their hospitals that would improve care. At the same time, collaborative leaders decided to address the barriers created by organizational cultures to making and sustaining improvement. To this end, a survey was designed to assess organizational culture with the 34 NICUs participating in the NIC/Q 2000 quality improvement collaborative. The experience of the first Vermont Oxford Network collaborative made it clear that the development of functioning multidisciplinary improvement teams and the integration of the core teams into the wider NICU and institutional environment requires an understanding of the specific organizational and cultural features unique to each NICU.

THE NIC/Q 2000 ORGANIZATIONAL ASSESSMENT

The Organizational Assessment Instrument was designed to serve 2 purposes. From a research perspective, the measures of organizational and managerial variables provided a set of independent variables to explore the extent to which differences in these measures contribute to variation in clinical outcomes and in the ability of organizations to make improvements. The reports of the results of the organizational survey were designed to be shared with the members of the collaborative. Each center could identify strengths and weaknesses in its performance and compare its scores with those of other participants in the collaborative. On the basis of this assessment, each NICU could specify activities or interventions to improve organizational scores. For example, an organization that scored low relative to other organizations in terms of communication scores might implement improvement projects aimed at improving communication between disciplines. At the same time, as part of the collaborative activities, working groups developed a number of potentially better practices that could be tested by the centers to improve performance. One group (labeled the CARE Group) developed practices and provided references around organizational issues including many identified in the organizational assessment survey.

ORGANIZATIONAL ASSESSMENT INSTRUMENT

A search for an appropriate research instrument that measured relevant elements of team performance, organizational culture, and managerial variables identified a number of relevant studies, but no 1 instrument covered the issues needed to assess the members of the Vermont Oxford Network NICUs participating in this new change effort. A literature review was used to help identify relevant measures. The final questionnaire included scales from several research instruments used in several studies of hospital organizations and units (see Table 1).

TABLE 1.

Organizational Dimensions and Sources for Scales

Data were collected from NICUs participating in the organizational survey in July to August 1998 and June to July 2000. A report of each center’s scores was sent to key contacts in each NICU and given to participants in meetings held of center teams. Results from the survey were reviewed in detail with members of the NICUs who attended meetings of the collaborative. The results from the first survey were sent to centers in August 1998 and discussed at the meeting in September 1998. Results from the second survey were mailed to centers in August 2000 and discussed at the September 2000 meeting.

The survey feedback method relies heavily on organizational participants’ understanding the nature of the data provided and being able to link these data to local issues and potential changes. A series of exercises were designed and used in a meeting of the collaborative to help the NICU teams understand the data and communicate them to center staff. Each team was asked to identify 1 area of strength and 1 area of potential weakness and then to specify behaviors, activities, and norms that make this area a strength or a weakness. Working in larger groups, composed of personnel from several centers, the NICU teams then identified potential change ideas that might help to address specific issues. Teams were encouraged to identify specific activities and behaviors that might be useful in improving performance in specific areas. These change ideas were then collected and shared among all groups. In addition, a number of handouts were provided to the members of the center teams who attended these collaborative meetings to assist in communicating the results of the survey and possible changes that could be tried in the local environments.

METHODS

In the 1998 survey, 2768 questionnaires were distributed and 1753 were completed for a response rate of 63.3%. In the 2000 survey, 2774 questionnaires were distributed to NICU staff and 1687 were returned, yielding a response rate of 60.8%. Surveys questionnaires were completed by a cross-section of unit leaders, physicians, nurses, therapists, and other staff. All NICUs were provided with detailed assessment results on their unit’s predominant culture and 5 other domains (see Table 1 for a listing).

Interviews on the distribution of the survey and its use were held in the spring and fall of 2001 with key contacts of NICU centers participating in the NIC/Q 2000 collaborative. An interview guide developed by the authors was used to gather information about the use of survey data by NICU center staff and other hospital personnel. Key contacts were asked how broadly results were shared and whether the questions and ideas suggested in the collaborative meeting were used in discussing the results. The key contacts were also asked whether the results of the survey were used in making changes in the units and, if so, what specific changes were identified. On the basis of experiences in using the survey to assess the units and stimulate change ideas, these key contacts were also asked about the usefulness of this approach.

Interviews were held with 12 contacts in 9 NICUs. These individuals were patient care leaders and medical directors of the units. All had participated in the core teams of the NIC/Q project. Centers selected included both centers that had scored highly on a number of dimensions of organizational performance as well as several that scored less well in comparison with other centers. Interviews were conducted by telephone by 1 member of the team. Detailed notes were taken in each interview, transcribed, and shared between 2 authors. A content analysis of the themes in these notes was done by 1 investigator along with the development of a list of actions and initiatives undertaken by teams in response to the survey.

RESULTS

Responses from the key informants suggested that all of the NIC/Q centers saw themselves as committed to undertaking the organizational assessment survey and using the results. Each center supplied a list of people to survey, encouraged survey completion, and reviewed the results. However, the extent to which the results were shared with staff who completed the survey and how the survey results were used to enhance quality improvement varied markedly between centers. Informants at all centers interviewed reported discussing the survey results in the NICU core leadership team meeting. Some teams even spent more than 1 meeting trying to determine what the results meant to them and how they should use the results to make improvements. However, for many centers, results were not shared with the full range of staff who participated in the survey.

Some units developed specific quality improvement projects to address an organizational or cultural issue that was identified in the survey. These issues included areas such as teamwork and coordination of care between disciplines. Other centers decided not to embark on specific quality improvement projects that focused directly on these topics. They chose to remain consciously aware of current culture and to incorporate what they had learned from the study into how they did their work. For example, 1 center reported making a conscious decision to increase communication around all of their improvement work. Other centers decided to keep discussing culture issues only within the core management team, and still others decided that the information was interesting but did not require specific responses. Finally, some units decided to address specific issues that came up in the survey but not through improvement teams. To some extent, the nature of the issues that were identified influenced the choice of strategy. For example in several units where the survey suggested issues around the authority of a nursing manager or medical director, the core team believed that the sensitive nature of this issue precluded additional attention.

Legitimizing the Study of Culture and Providing New Vocabulary

For all interviewees, participating in the survey and receiving the results provided an important opportunity to discuss aspects of the unit’s operation openly. Many informants noted that the survey results legitimized the discussion of culture. They noted that although people had casually talked about unit beliefs, norms, attitudes, and values before the survey, there was not a framework or vocabulary to describe overall unit culture. In 1 unit, they may have said, “This is a nice place to work and we try to help each other out. We are very committed to teamwork and consensus building.” In another unit, they may have said, “We have a very formal structure where we carefully follow the unit’s policies and procedures. When there are new decisions, they are communicated to us through our immediate supervisors.” The survey gave them a broader language to describe the cultures. They could now say that the first unit described above was predominantly a group culture and the second work environment was more hierarchical. The vocabulary has helped people take what they knew about the unit’s organizational behavior and make it explicit (see Fig 1 on descriptions of the 4 types of cultures). The development of this new vocabulary shifted the conversation about culture and heightened awareness of these issues in many centers. At 1 center, for example, an interviewee recalled when someone made a remark in a meeting, indicating that a behavior was “a very hierarchical thing to do.”

Fig 1.

Types of organizational cultures.19

No respondents reported being surprised by the survey results regarding their type of culture. However, several respondents noted that receiving the results, along with an explanation of what the results meant, created a means for team members to talk about culture and, therefore, to address it systematically. In addition to giving them a vocabulary to discuss different types of culture, the survey results gave them data on their own unit and comparative data on several specific organizational issues, including teamwork and leadership, coordination and communication, managing disagreements, and the ability to do and enjoy their work. They were then able to focus on areas of specific need, such as poor communication between disciplines, low morale, or a leadership problem.

Along with the data, each unit received a survey results document that provided 2 visuals. The first showed the predominant culture of the unit on a radar graph (see Fig 2). The radar graph of organizational culture provided an accessible display summarizing each unit’s profile on the organizational culture scales. The graph illustrates the dominant aspects of organizational culture and the extent to which other cultural elements are present for each center. The second visual (Fig 3) was a set of box plots of indicators showing where each unit stood in relationship to the other 30+ centers in the collaborative. This visual helped each center to identify how its results compared with those of the other centers, and it reduced the likelihood that a center could dismiss its results as similar to those of the other centers.

Fig 2.

Box plots of sample NICU results on views of leadership.

Fig 3.

Radar plot of organizational culture.

Several respondents also reported that having data based on validated research measures of culture helped to make the idea of “culture” more credible. Beyond the common sense notion of culture, the use of validated instruments to document the characteristics of the culture lent scientific credibility to the results and made it easier for many clinical leaders to report the results to staff. Informants noted that the use of a psychometrically valid survey instrument backed up by earlier research on culture along with published evidence on culture change allowed them to address organizational issues in the same way as other unit practices. Also, by providing comparative data, the survey results reinforced the notion that there are distinct differences in unit cultures and raised the prospect for units to address specific aspects of behavior in their units that they want to change. A manager at 1 center reported that although they were conscious of the nature of their culture, staff never talked about it openly because it was perceived, especially by the physicians, as soft and intangible. This manager reported that receiving the survey results made it possible to address culture openly and directly because of specific results on the culture of the unit along with literature resources and comparative information from >30 other NICUs.

Other clinicians, particularly unit neonatologists, were not convinced that culture is critical to unit effectiveness and in particular had no clear relationships with clinical outcomes. Despite an interest in quality improvement and medical outcomes, they did not believe that there is a link between organizational culture and quality improvement efforts.

Sharing the Survey Results

When the first set of survey results were available, all of the centers interviewed shared the entire report with the core management team of 15 to 25 people. Some centers shared results from the survey beyond the core team with key groups such as nursing staff, neonatologists, and hospital administration. A small group of centers made concerted efforts to ensure that all staff heard about or saw the major results. Most often, these results were shared in meetings. A few centers put a summary of this information into unit newsletters. Overall, however, there was a wide variation in the percentage of staff who saw the results. The interviewees estimated that they shared the survey results with anywhere from a very few (3%) to almost all (90%) staff on their unit.

A number of respondents noted that they had been very interested in receiving the results of the second survey to assess whether the improvements that they had made in areas such as communication, conflict resolution, and culture had measurable impact. When the results of the second survey were available, most centers shared the results with the same groups that received the reports of the first survey. Some centers shared the results with other units and staff, including the quality improvement and regulatory departments within the hospital, nursing researching, respiratory therapy, and fellows. When people shared this information outside the unit, many respondents reported considerable interest in the instrument, the results, the graphs, and the language. Two organizations reported plans to conduct organizational assessments in other units. A few centers shared the results with almost no one except the people who attended the Vermont Oxford Network NIC/Q meeting where results were distributed.

Evidence from the interviews does not support any clear relationship between unit scores on the survey scales and the intensity of their dissemination of survey results and follow-up. Instead, respondents attributed the intensity of dissemination with the attitudes of the leaders in the unit toward the survey. In 1 unit, where survey results had gone up on a number of indicators, the leader thought that staff would not be interested in the results, and in another unit, where results had stayed about the same, the leader was busy with other projects and never shared the results. Several centers reported plans to use this information to plan additional efforts around cultural development.

Using Survey Results to Make Improvements

A number of respondents noted that the survey results provided useful information to guide quality improvement efforts. By understanding the positive aspects of each culture, as well as the negative, center staff could use this information to identify how different aspects of their culture could help or hinder an improvement effort. This understanding helped to identify concrete projects and specific approaches to improvement. It made them believe that they could actually change their culture if they put effort into it. For example, staff in 1 unit that had been identified as possessing “hierarchical culture” restructured some reporting relationships to make the unit less hierarchical. They brought more staff into the management team committees and created 5 shared governance committees around issues such as education, quality, and family-centered care. They empowered each of these councils to make their own decisions. They held retreats and team-building sessions for members. They tried to include more people in unit improvement teams. Furthermore, in the redesign process, 1 nurse manager who had been perceived as very hierarchical left the unit.

Respondents noted interest in other aspects of the survey results besides the information provided on unit culture, including results on leadership and teamwork, coordination and communication, managing disagreements, and job satisfaction (see Table 1 for a complete list on 6 domains of the survey). For example, some unit leaders reported using the information in the leadership section to gain insights into their leadership styles. They used the feedback to change specific behaviors. For example, 1 unit medical director noted that the results stimulated his interest in broadening his leadership role beyond physicians. He now regularly attends and participates in nursing update meetings and writes a column on evidence in the unit newsletter. Another physician leader noted that the organizational survey gave him confidence that he is doing a good job working with the people in his unit. Before the survey, he noted, there was no information available to assess his role, except for anecdotal comments.

The access to other aspects of organizational performance allowed some units to identify strengths and make decisions not to change. For example, 1 manager operating in a very hierarchical culture noted that the results on job satisfaction deterred her from attempting to change her unit culture. In this unit, the nursing staff was very satisfied with their work, especially in comparison with other NICUs in the collaborative. As a result, the nursing manager decided not to attempt to change the culture that was satisfying them. Nursing satisfaction in this unit increased from the first to the second survey. This manager also noted, in keeping with a very hierarchical culture, that even when there was positive information to share, the results of the second survey were shared with almost no one.

A variety of projects and activities were initiated as a result of the survey. Units began projects on team building, managing disagreements, and improved communication. One unit developed social activities such as celebrating special holidays, playing games, and having contests to improve team building and add some fun to the nursery. Another unit conducted a 6-hour conflict resolution training program for 34 leadership people and a 2-hour training for approximately 50% of the entire unit staff. Other units worked on improving communication. In 1, they developed a new system for communicating unit information that involves giving presentations on all 3 shifts and on weekends. In another, they wanted to address communication between disciplines. Although they already had multidisciplinary rounds, representatives from certain groups were not attending and 1 group, respiratory therapy, was not even represented. They changed this by adding respiratory therapy and reminding people when rounds started and how important their participation was.

In still another unit, they developed a service team that solicits opinions before they formalize new policies in the unit. One facility received feedback from the survey that the unit was not very receptive to new nurses. After they discussed this, they met with new nurses to find out how they could welcome them into the unit. They received some advice and began to make changes. They continue to meet with new nurses to work on issues that get in the way of making them feel a part of the unit. In this last case and several others, the survey provided an indication of where an issue might exist and allowed staff to collect additional information to guide improvements.

In some units, projects that related directly to culture were given lower priority to those that addressed clinical issues. However, several respondents in these units reported that they attempted to incorporate an awareness of culture issues when working on a clinical project. For example, they noted the need to be aware of communication and conflict between staff as they identified ways to change clinical policies.

Besides identifying the type of culture and performance in other organizational domains, the survey reports included the specific comments made by staff on the surveys. Respondents in several centers noted that the verbatim comments were the most helpful element of the feedback. For them, the specific comments provided information about the functioning of their unit. For many of these units, this was the first time that they had asked employees and physicians to provide feedback on how things were going in the unit. At some of these centers, the leaders knew that they had problems in a certain area, such as many people dissatisfied with their work schedules or a problem with a particular manager’s communication style, but they were unable to talk about them because no one wanted to bring them up or there was no explicit evidence. The comments, sometimes repeated on many surveys, gave the core management teams information that they could not ignore.

Making the Survey Information More Useful

Respondents identified a number of ways in which the survey feedback could have been improved. For several centers, respondents noted that the pressures of daily work make it difficult to disseminate results. Only a handful of centers took the detailed results that they received and summarized them or developed overheads for presentations and/or publications. Several respondents noted that prepared summaries and sets of overheads on the survey would have made it easier to disseminate results to staff in meetings or newsletters.

Respondents in several centers also commented that it would have been helpful to have a facilitator or an expert on organizational behavior present when they reviewed the survey results. Some of the information was very hard for certain people on the core leadership team to interpret without the support of a facilitator. In addition, several respondents noted that an expert in organizational development might have been helpful to them to interpret the results, highlight key areas on which to focus, and help them select on which projects to work.

CONCLUSIONS

Organizational culture assessments have been used in a variety of change initiatives in both health care and other industries. The feedback from the experience of NICU centers that were given surveys in the NIC/Q 2000 project suggests that these data provided some centers with concrete information that guided their improvement efforts. In other cases, the major impact of the survey was the opportunity to discuss issues of organizational culture and leadership, using the labels and data provided by the survey. Survey feedback in rapid-cycle improvement efforts by clinical teams is thus very similar to its use in more traditional change initiatives directed by senior leadership. Recent quality improvement efforts in many organizations have focused on unit-based rapid-cycle change, led by front-line clinical leaders. Such changes rarely include organizational assessment tools and the survey feedback techniques used in organizational development. The Vermont Oxford Network’s quality improvement project for NICUs demonstrates the value of incorporating an organizational assessment as part of the package of change ideas and change strategies provided to the participating NICUs.

Acknowledgments

This research was supported by a grant from the David and Lucile Packard Foundation to the Vermont Oxford Network.

NICU, neonatal intensive care unitNIC/Q 2000, Neonatal Intensive Care Unit Quality Improvement Collaborative Year 2000

REFERENCES