The global need for patient safety initiatives in health care has been a topic of recent public and professional concern.¹ To develop a stronger understanding of local needs, a fundamental shift in the healthcare climate must occur—moving from one of blaming individuals for errors to one of improving healthcare systems to maximize patient safety. Before implementing patient safety initiatives in healthcare organizations, one important step is to understand the perceived patient safety climate.

Different methods may be employed to achieve this goal. In-depth individual interviews and focus groups provide detailed insights about individual and collective perceptions, although these methods are time consuming and resource intensive. Self-administered surveys can also help to understand institutional perceptions of safety culture. Surveys are an efficient way to ask standardized questions of all respondents concurrently and anonymously.

Patient safety was identified as one of the key strategic focus areas of Hamilton Health Sciences (HHS) in 2003. The intensive care unit (ICU) was designated as one of several “innovation and learning centres” selected to advance quality improvement through patient safety initiatives. Our first project was to perform a baseline measurement which would inform us about the need for future interventions and would allow comparisons over time.

It was decided a priori that a peer reviewed published instrument needed to be identified with three primary characteristics: (1) disclosure of the items of the instrument; (2) documented reliability characteristics (internal consistency, test-retest reliability); and (3) specific to patient safety in the ICU. Using PubMed (MEDLINE), a literature search was conducted in Fall 2003. No date limitations were imposed on retrieved articles. The search terms used included “patient safety questionnaire”, “safety climate questionnaire”, and “patient safety reliability”. If a relevant article was found, the “Related Articles” feature was used to identify other potentially relevant articles. Reference lists from relevant articles were also searched.

No studies were found which met the a priori criteria. Sexton et al² published the first peer reviewed results from the Intensive Care Unit Management Questionnaire (ICUMAQ), identifying four items related to stress and teamwork. Thomas and colleagues³ reported results of the seven item teamwork climate scale of the ICUMAQ. The paper by Pronovost et al⁴ was the first peer reviewed article to disclose the Safety Climate Scale (SCSc), 10 patient safety related items from Sexton’s work. As identified by Pronovost et al,⁴ one of the limitations of the SCSc was that no formal evaluation of the performance properties of the instrument had been conducted.

The Safety Climate Survey (SCSu), a 21-item instrument developed by Sexton and colleagues, was endorsed by the Institute for Healthcare Improvement (IHI).⁵ Of the 21 items in the SCSu, nine were similar to the SCSc. Because of the widespread accessibility of the SCSu and the use of identical items to the SCSc, we chose to use this instrument. Since the completion of our study, the full SCSu instrument has been published in the peer reviewed literature although its reliability characteristics were not reported.⁶

In this study we describe the use of the SCSu and two of its derivatives—the SCSc and the Safety Climate Mean (SCM)—in our ICUs. The study had two objectives: (1) to establish the internal consistency and test-retest reliability of the SCSu (21 items), SCSc (10 items, nine of which are from the SCSu), and SCM (seven items from the SCSu) based on administration of the SCSu in our hospital; and (2) to report the results of administering the SCSu in our hospital.

METHODS

Setting

This study was conducted in a tertiary care medical center affiliated with McMaster University in Hamilton, Ontario, Canada, serving more than 2.2 million citizens in Central South and Central West Ontario. The survey was conducted in four ICUs in HHS (one cardiovascular-surgical ICU (CVICU) and three medical-surgical ICUs (MSICUs)), representing 62 ICU beds.

Survey format

The survey format consisted of a brief demographic section outlining the participant’s professional group (intensivist physician, nurse, etc), position (full time, part time, other), hours of work (days, nights, rotating shifts, other), total experience in critical care, duration of practice in current profession, duration of employment in current job, duration of employment by HHS, and base ICU. We used all of the items from the SCSu (21 items), plus one question from the SCSc which was not included in the SCSu (”I am aware that patient safety has become a major area of improvement in this institution”). To reflect our staffing structure and institutional leadership, we added one question to reflect the availability of registered respiratory care practitioner leadership (”I am satisfied with the availability of registered respiratory care practitioner leadership.”). The questionnaire therefore had a total of 23 items. Respondents were asked to rate each item on a 5-point Likert-like scale (disagree strongly, disagree slightly, neutral, agree slightly, and agree strongly). Table 1 outlines the SCSu, SCSc, and SCM items.

Individual item responses were transposed into numerical values ranging from 1 to 5 (1 = disagree strongly, 2 = disagree slightly, 3 = neutral, 4 = agree slightly, 5 = agree strongly) or not applicable. Higher values reflected a more positive safety climate. Thus, questions worded to reflect negative safety climates—for example, “Personnel frequently disregard rules or guidelines that are established for this clinical area”—were scaled in reverse (disagree strongly = 5, agree strongly = 1).

Respondents

All regular full time and part time employees were invited to complete the survey, including nurses, allied health professionals (chaplains, occupational therapists, pharmacists, pharmacy technicians, physiotherapists, registered respiratory care practitioners, speech language pathologists), non-clinical staff (ward clerks, housekeepers), attending intensivists, and management (clinical managers, directors, educators). Staffing rosters were used to identify potential participants and active staff members were confirmed with scheduling coordinators. Employees on leave of absence or maternity leave were excluded from the sample, as were resident physicians.

Thirty one volunteers from all sites were asked to complete the questionnaire within 3–14 days of first administration to calculate test-retest reliability of the instruments.⁷

Recruitment

A multimodal recruitment strategy was used to solicit employee participation. Each employee was allocated an individualized questionnaire package which consisted of an index card with the employee’s name paper clipped to the questionnaire. The questionnaire packages were filed alphabetically by surname and placed in a box which was positioned in a prominent location in each ICU. Respondents were asked to complete the questionnaire and detach the index card with their name on it to maintain anonymity. Respondents were asked to deposit questionnaires in a central location in the ICU, while index cards were entered into a lottery draw for a prize. All employees were informed of the survey by a management endorsed electronic mail message. Posters and signs were placed in all the ICUs to inform staff of the survey. We held several individual and group in-services on day and night shifts to introduce staff to the survey and answer questions. Local opinion leaders and managers verbally encouraged staff to complete the survey. Site-specific participation rates were reported to staff and management on a biweekly basis.

A separate recruitment strategy was used for intensivists who work for 1 week on call in a row. A cover letter and questionnaire were sent by interoffice mail to the administrative assistant of each intensivist. The Critical Care Division director endorsed completion of the questionnaire by electronic mail and encouraged participation at staff meetings.

Research ethics

The protocol was approved by the HHS Research Ethics Board, Hamilton, Ontario, which waived the need for written informed consent.

Data management

We transcribed the instrument onto an optical scanning card specifically designed for our use. Participants were asked to complete the optical scanning card with an HB pencil. The data were read by an optical scanner and entered into an electronic database.

Statistical methods

The internal consistency for all three instruments was calculated using Cronbach’s alpha. We decided a priori not to pursue further analysis if internal consistency was less than 0.8.⁷ Individual scale means were calculated by summing the item scores and dividing by the total number of items.

For individuals completing test-retest reliability, we used the first response for group analyses. We calculated a priori that a sample size of 22 raters was needed to test whether an obtained reliability of 0.80 exceeded a reliability of 0.50, given two raters, a one tailed α = 0.05, and power of 80% using the methods of Walter et al.⁸ Assuming a modest participation rate of 70%, a convenience sample of 31 people were invited to complete the test-retest reliability study. The sample was stratified by caregiver type. The intraclass correlation coefficient (ICC) and 95% confidence intervals (95% CI) were used to express test-retest reliability.

To examine differences in scale scores by ICU and between professional groups, univariate analysis of variance (ANOVA) was used. The Student-Newman-Keuls test was used post hoc for multiple comparisons.⁹ The statistical tests were considered significant at the p⩽0.05 level (two tailed). All statistical analyses were conducted with SPSS (Version 12.0 for Windows).

RESULTS

Data collection took place between 19 January 2004 and 31 March 2004. The overall response rate was 74% (313/426). By job category, the response rates were as follows: nurses 66% (178/269), allied health professionals 77% (59/77), non-clinical staff 89% (40/45), physicians 80% (16/20), management 67% (10/15). Ten respondents identified their job category as “other”. The characteristics of the respondents are shown in table 2. As expected, the majority of the respondents were nurses (56.9%, 178/313), followed by allied health professionals (18.8%, 59/313), non-clinical staff (12.8%, 40/313), physicians (5.1%, 16/313), and management (3.2%, 10/313). Overall, 69.3% (217/313) of the respondents were employed full time by HHS and 57.5% (180/313) worked rotating shifts.

Test-retest reliability analysis was evaluated on data from 31 respondents who completed the instrument in duplicate. The job categories of the test-retest reliability study participants were: nurses (45.2%, 14/31), allied health professionals (9.7%, 3/31), non-clinical staff (9.7% 3/31), physicians (6.5%, 2/31), management (12.9%, 4/31), and others (16%, 5/31).

Reliability testing

Internal consistency

Using Cronbach’s alpha, the internal consistency of the SCSu was 0.86, of the SCSc was 0.80, and 0.51 for the SCM. Thus, following our a priori decisions, we continued analysis of only the SCSu and SCSc instruments.

Test-retest reliability

Using the ICC, the test re-test reliability of the SCSu was 0.92 (95% CI 0.82 to 0.97) and of the SCSc was 0.92 (95% CI 0.82 to 0.96).

Survey analysis

The mean (SD) scores for each scale are shown in table 3. Of the 313 respondents, 68.6% (n = 215) answered all the items of the SCSu while 78.3% (n = 245) answered all the items of the SCSc. Of 5 maximum points, the overall mean (SD) score of the SCSu was 3.4 (0.6) while the SCSc was 3.4 (0.7). Using univariate ANOVA, there were no significant differences in the mean score by ICU (SCSu: F-statistic = 0.022, df = 1,3, p = 0.996; SCSc: F-statistic = 0.127, df = 1,3, p = 0.944).

In table 4 the mean (SD) scores are presented by discipline. Clinical disciplines were defined as follows: nurses, allied health professionals (registered respiratory care practitioners, pharmacists, physiotherapists, and chaplaincy), non-clinical staff (administrative staff, housekeeping), physicians, and management (managers, directors, clinical nurse specialists, and educators). There were significant differences between disciplines in both the SCSu and SCSc surveys (SCSu: F-statistic = 7.8, df = 1, 4, p<0.001; SCSc: F-statistic = 8.0, df = 1,4, p<0.001). Post hoc, the Student-Neuman Keuls test identified significantly higher scores from respondents in health management than all other disciplines using both the SCSu and SCSc surveys (p<0.05). No other differences were identified between disciplines.

DISCUSSION

The safety climate was viewed positively in these four ICUs, although room for improvement exists. Participants in the study scored an average of 3.4 out of 5 possible points on both the SCSu and the SCSc. No differences were identified in respondents from the MSICUs and the CVICU. Among frontline staff, no differences in perception of safety climate were noted between disciplines. However, management participants perceived a more positive safety climate than frontline staff, regardless of discipline or activity, which may reflect year long discussions of patient safety initiatives within management circles before administration of the SCSu and SCSc tools. Leaders may believe they are creating a positive environment and reflect this bias in their responses. Also, due to the nature of their management positions, information about patient safety may be more available to managers than to frontline staff, and the former may also be more attuned to the identification and resolution of patient safety issues. However, these discordant perceptions identify an opportunity for communication between management and frontline staff to discuss patient safety issues and concerns. Management participants had mean (SD) scores of 4.2 (0.3) and 4.3 (0.6) on the SCSu and SCSc tools, respectively, corresponding to scores which fell between the “agree slightly” and “agree strongly” anchors.

Our results differ from those of Pronovost and colleagues⁴ who administered the SCSc in 2001 to a convenience sample of staff (clinical leaders, clinical staff, and non-clinical staff) at Johns Hopkins Hospital. Investigators dichotomized data from the SCSc into “agree” (respondents answered “somewhat agree” or “agree”) or “disagree” (respondents answered “neutral”, “somewhat disagree”, or “disagree”). Physicians gave lower scores than nurses in adverse event reporting and awareness of safety initiatives. In contrast, we did not find differences in the perceived safety climate between physicians, nurses, allied health professionals, or non-clinical staff. Compared with Pronovost et al, we conducted our study exclusively among ICU staff, analysed data by scale rather than by item, and maintained data in continuous form rather than dichotomizing responses, which may account for the differences in results between our studies. Compared with other areas of the hospital, frontline ICU staff members may perceive a similar safety climate across disciplines due to closer teamwork in the ICU—for example, explicit multidisciplinary roles during patient resuscitation, high patient turnover requiring cross-disciplinary coordination for discharge planning. Psychometrically, it is possible that the instrument is not sufficiently sensitive to identify small but important differences among disciplines in this context. In addition, our study was conducted in a country with a universal healthcare system rather than the US model of health care.

Of three possible safety climate scales, two (the SCSu and SCSc) met our a priori reliability criteria. Cronbach’s alpha (internal consistency) examines the pattern of answers between the items and the total score. If items are tapping into the same construct (in this case, safety climate), then respondents should answer items in a similar manner. “Acceptable” ranges of internal consistency range from 0.7 to 0.9.⁷ We chose a cut off score of 0.8. The internal consistency of the SCM was 0.51, well below published criteria. Poor performance in internal consistency indicates that the group of items is tapping into different constructs, so respondents do not answer this particular group of items in a similar way. We examined this group of items to see if removing one of the items would improve internal consistency, but we were still unable to meet a minimum of 0.7. Thus, based on this analysis, the 7-item SCM is not as psychometrically sound as an indicator of safety climate as the other instruments tested in this study.

A fundamental property of a good instrument is test-retest reliability. Landis and Koch¹⁰ proposed five benchmark classifications to assist with the interpretation of the reliability coefficient: “slight”, 0.0–0.20, “fair”, 0.21–0.40; “moderate”, 0.41–0.60; “substantial”, 0.61–0.80, and “almost perfect”, 0.81–1.00. The ICC of the SCSu and SCSc was 0.92. We therefore have confidence that these are valid instruments to measure patient safety in the absence of change.

The strengths of this study include a high survey response rate achieved by using multiple methods to maximize participation, suggesting that the results reflect staff perceptions of the ICU safety climate. All members of the ICU team were surveyed, including clinical and non-clinical staff as well as management, reflecting the complex interaction of multidisciplinary individuals contributing to the care of critically ill patients. We used rigorous statistical methods to establish the psychometric properties of the scales including Cronbach’s alpha and test-retest reliability measures to confirm the appropriateness of our measurement tools. The instruments we used in this survey therefore appear to be measuring similar constructs. Our results suggest that respondents working in our healthcare organization perceive the setting to have a somewhat positive safety climate, although there is room for improvement.

One of the limitations of this study is the small number of management participants; only 10 persons with management responsibilities participated in the survey but, because of missing data, there were only seven (SCSu) and eight (SCSc) complete datasets for analysis. However, the absolute number of management positions will always be fewer than that of frontline staff. Assuming a similar participation rate, administration of this survey among other ICUs or in other hospital departments would increase the sample size of the management group. Secondly, due to limited resources, we excluded non-regularly scheduled staff and residents. It is possible that residents’ perspectives on patient safety may differ as a result of their comparative experiences at other institutions. As important members of the critical care team, residents’ perceptions would potentially offer fresh insights from the perspective of newcomers rather than stable staff members, which may be different from other disciplines. Thirdly, we added one survey item to inquire about the patient safety leadership of registered respiratory care practitioners; however, we believe that the inclusion of these clinicians had minimal impact on the internal consistency and reliability properties of these instruments. Because of the cross sectional nature of this study, our results are only applicable to the perception of safety climate at the time of the survey.

CONCLUSIONS

Based on the results of this study, we conclude that both the SCSu and SCSc are reliable measurements in our Canadian university affiliated ICU setting. The SCM items appeared to measure different constructs and we suggest that, alone, it is not the best instrument to monitor safety climate, as suggested by the IHI. Future research is needed to examine the properties of these tools in other ICUs, other hospital settings, and other countries, as well as examination of the responsiveness of these tools to change after patient safety initiatives are initiated in the ICU setting.