An updated analysis of safety climate and downstream outcomes in two convenience samples of U.S. fire departments (FOCUS 1.0 and 2.0 survey waves)

Abstract

Background

The Fire service Organizational Culture of Safety (FOCUS) survey is an assessment tool comprised of psychometrically validated metrics of safety climate, safety behavior, and downstream outcomes (organizational and injury) that are specific to the U.S. fire and rescue service.

Methods

This analysis consists of a descriptive summary of two independent survey waves (FOCUS 1.0 and 2.0). The fire departments included in these survey waves were from convenience sampling (n_1.0 = 275; n_2.0 = 170). In addition to department level characteristics, we examined individual level characteristics for firefighters and EMS providers in participating departments (n_1.0 = 22,719; n_2.0 = 16,882). We conducted regression analyses to examine the associations between safety climate and safety behaviors, organizational outcomes, and safety outcomes. All analyses were stratified by organization type (career, volunteer).

Results

Our analysis indicated that a majority of respondents were males (90.7%_{FOCUS 1.0}; 90.4%_{FOCUS 2.0}), non-officers (68.4%_{FOCUS 1.0}; 66.4%_{FOCUS 2.0}), and non-Hispanic Whites (70.8%_{FOCUS 1.0}; 69.5%_{FOCUS 2.0}). For both samples there was a higher prevalence of injuries among individuals in career departments (n_{FOCUS 1.0} = 3778 [17.5%]; n_{FOCUS 2.0} = 3072 [18.7%]) than volunteer departments (n_{FOCUS 1.0} = 103 [8.8%]; n_{FOCUS 2.0} = 34 [7.4%]). We observed an approximate 10-point difference between the mean scores of Management Commitment to Safety for career and volunteer departments in both samples. We observed associations for two organizational outcomes, Safety Behavior and Job Satisfaction, with Management Commitment to Safety and Supervisor Support for Safety overall and when stratified by organization type. We observed a decrease in the odds of injuries associated with a one-unit increase in Management Commitment to Safety (OR_{1.0 overall}: 0.98, 95% CI 0.97–0.99; OR_{2.0 volunteer}: 0.90, 95% CI 0.85–0.95) and Supervisor Support for Safety (OR_{1.0 overall}: 0.95, 95% CI 0.93–0.97; OR_{1.0 career}: 0.95, 95% CI 0.92–0.98).

Conclusions

From our current study, and a prior analysis of a geographically stratified random sample of U.S. fire departments, we identified that from all the organizational outcomes, job satisfaction was most consistently associated with FOCUS safety climate. Further, firefighters in our samples consistently rated Supervisor Support for Safety higher than Management Commitment to Safety. Future interventions should support fire departments in improving their departmental Management Commitment to Safety and maintaining their Supervisor for Safety.

Background

In 2020, it was estimated that 62.3 per 1000 firefighters were injured on the job (Campbell and Evarts 2020), underscoring the burden of injuries among this occupational group. Colloquial understanding of firefighter injuries may assume that most of their occupational injuries are related to fighting fires (i.e., on the fireground (Marsh, et al. 2018)), but prior research has demonstrated many injuries occur during training/physical activity exercises (Hollerbach et al. 2020; Poplin et al. 2012) or on non-fire calls (Poplin et al. 2012; Phelps et al. 2018). Emergency medical services (EMS) response constitutes 65% of all 9-1-1 calls to fire departments (Available from 2019) and there is some evidence to support that paramedics stationed in fire departments have higher all-injury rates than firefighters within the same department (Widman et al. 2018). Additionally, EMS providers have higher odds of experiencing injury from violence and assaults (Taylor et al. 2016; Maguire et al. 2018; Reichard et al. 2017).

Traditional firefighter and EMS provider injury research has primarily focused on the outcome of injuries (Marsh et al. 2018; Hollerbach et al. 2020; Poplin et al. 2012; Phelps et al. 2018; Available from 2019; Widman et al. 2018; Taylor et al. 2016; Maguire et al. 2018; Reichard et al. 2017), thus more needs to be known about its upstream predictors of safety climate, which is a well-established predictor of occupational injury (Christian et al. 2009; Huang et al. 2016). Safety climate reflects the perceptions of individuals regarding their organization’s regard for their occupational safety as expressed through their policies, procedures, and practices (Zohar 1980). An industry specific safety climate scale, the Fire service Organizational Culture of Safety (FOCUS) survey, was developed and its psychometric validation described previously (Taylor et al. 2019). The FOCUS survey is an assessment tool that is now in its fourth wave of assessment. Its research to practice impact has been described previously (Davis et al. 2020).

A previous analysis of the FOCUS beta-test survey wave identified two fire-service specific safety climate dimensions: Management Commitment to Safety and Supervisor Support for Safety (Taylor et al. 2019). FOCUS Management Commitment to Safety refers to members' perception that management (leadership) values their safety and engages in communication and actions that support safety procedures and practices (Taylor et al. 2019). FOCUS Supervisor Support for Safety refers to members' belief that their supervisors (company officers) value their safety based on communication, encouragement, and consequences (Taylor et al. 2019). A recent analysis of the FOCUS beta-test survey wave, a geographically stratified random sample, identified that the mean Management Commitment to Safety scores for career fire departments were lower compared to combination and volunteer departments, however a similar variation for mean Supervisor Support for Safety scores were not observed (Geczik et al. 2022). Safety behavior refers to the types of behaviors of workers that contribute to safety in the fire department, such as using the correct safety equipment (Neal and Griffin 2002). On the FOCUS survey we measured safety behavior using two metrics: Safety Behavior and Safety Compliance.

In the current study, we conducted a secondary analysis of cross-sectional survey data for two independent survey waves. The purpose of this study was to independently investigate the descriptive statistics of the individual and department level data from U.S. fire departments that completed the FOCUS 1.0 and 2.0 survey. Further, we were interested in examining the relationships between FOCUS safety climate and outcomes of interest (i.e. safety behaviors, organizational outcomes, and safety outcomes). This study replicates the published work of Geczik AM et al. (2022), using updated survey wave data (FOCUS 1.0 and 2.0) to further understand safety climate in a subset of U.S. career and volunteer fire departments. We wanted to replicate the previous analysis since there was a change in sampling methods from the FOCUS beta-test (random) to that used in FOCUS 1.0 and 2.0 survey waves (convenience).

Methods

Study population

The departments surveyed in the FOCUS 1.0 (2017–2018) and 2.0 (2019–2020) survey waves constituted convenience samples. Enrollment into these survey waves was open to any fire department in the U.S. that was interested in assessing their departmental safety climate to better understand their organizational safety culture on a first come, first serve basis until resources were exhausted. Survey administration was conducted over a 30-day period, on average, and departments had a goal to include at least 60% of members at each station. The FOCUS 1.0 survey wave included 31,508 individual respondents from 304 departments. The FOCUS 2.0 survey wave included 20,141 individual respondents from 178 departments. Both survey waves included U.S. fire departments that were career and volunteer and from urban, suburban, and rural areas.

Demographic variables

Both FOCUS surveys obtained demographic information on the individual level. These variables have been previously described for the descriptive analysis of the FOCUS beta-test survey wave (Geczik et al. 2022). Respondents of the survey were asked to “select all that apply” when answering questions regarding their race/ethnicity, and professional role. The survey's response options for race and ethnicity were as follows: American Indian or Alaska Native, Asian, Black or African American, Hispanic, Native Hawaiian or Other Pacific Islander, Other, White. Individuals that self-identified as more than one race and ethnicity were categorized as “more than one.” Due to small numbers in our analytic sample and to preserve anonymity, our reported “Other” group included individuals that self-identified as: American Indian or Alaska Native, Asian, Native Hawaiian or Other Pacific Islander, or Other. For reporting race and ethnicity, we followed the 2021 guidelines from the Journal of the American Medical Association (JAMA) (Flanagin et al. 2021).

On the survey, the response options for “what is your role in this fire department” were as follows: firefighter, paramedic, emergency medical technician (EMT), Lieutenant, Captain, Battalion Chief (Division/District), or Chief (Fire, Deputy, Associate, Assistant), Commissioner. Since individuals could select more than one professional role, we created a three-level categorical variable (non-officer, officer, leadership) to identify the officer status of individual respondents based on the highest rank they reported. Individuals may hold officer or leadership roles in addition to being cross-trained firefighters and EMS providers (EMTs or Paramedics), single role-firefighters, or single-role EMS providers. In many municipalities, firefighters are cross-trained as they also respond to medical calls. The categorical variable was created based on the self-identification of rank: firefighter, EMT, or paramedic (non-officer); lieutenant or captain (officer); battalion chief, chief, or commissioner (leadership) (Geczik et al. 2022).

Department characteristics

Department characteristics (i.e. call volume, population served, organization type, roster size) were collected during enrollment. Continuous variables were recategorized based on quantiles to create four categories (minimum to Q1, Q1 to median, median to Q3, Q3 to maximum). The State reported by each department was matched to its corresponding Federal Emergency Management Agency (FEMA) Region (n = 10). During enrollment, departments reported their Insurance Services Office (ISO) rating and Center for Public Safety Excellence (CPSE) accreditation, when applicable. ISO ratings are measures of a fire department’s ability to protect its community based on its preparedness to fight fires based upon relevant NFPA Standards (National Firefighter Protection Association et al. 2014). CPSE accreditation is a voluntary process that U.S. fire departments can undergo to ensure they provide the best support for their communities (Center for Public Safety Excellence 2024).

Exclusion criteria

We restricted our analytic sample to departments that provided both fire and EMS response because we wanted to study the differences between those call types. Additional file 2: Supplemental Fig. 1 shows a flowchart of the exclusion criteria. Our exclusionary criteria for the analysis at the individual level was as follows: individuals missing fire department name on survey (n_{FOCUS 1.0} = 115; n_{FOCUS 2.0} = 82), individuals missing scale items of interest (n_{FOCUS 1.0} = 6,542; n_{FOCUS 2.0} = 2,869), individuals missing fire department identifier (n_{FOCUS 1.0} = 203), individuals under the age of 18 (n_{FOCUS 1.0} = 44; n_{FOCUS 2.0} = 49), individuals missing all demographics of interest (n_{FOCUS 1.0} = 435; n_{FOCUS 2.0} = 73), individuals from departments that did not fully complete enrollment (n_1.0 = 133), individuals from departments with data entry errors (n_{FOCUS 1.0} = 411; n_{FOCUS 2.0} = 146), individuals from departments that did not provide EMS response (n_{FOCUS 1.0} = 853; n_{FOCUS 2.0} = 13), individuals over the age of 90 (n_{FOCUS 1.0} = 23; n_{FOCUS 2.0} = 22), and individuals with implausible years of experience (n_{FOCUS 1.0} = 30; n_{FOCUS 2.0} = 5).

At the department level, the following exclusions were made: departments that did not receive a fire department identifier due to incomplete survey (n_{FOCUS 1.0} = 2), departments that did not provide completed demographic data (n_{FOCUS 1.0} = 3), departments with data entry errors (n_{FOCUS 1.0} = 10; n_{FOCUS 2.0} = 3), departments that did not provide EMS response (n_{FOCUS 1.0} = 11; n_{FOCUS 2.0} = 5), and departments missing outcomes of interest (n_{FOCUS 1.0} = 3).

These exclusions resulted in the following analytic sample for each FOCUS survey wave: FOCUS 1.0 had 22,719 individuals from 275 departments and FOCUS 2.0 had 16,882 individuals from 170 departments. The analytic sample had an average response rate of 80.9% and 74.1%, for FOCUS 1.0 and 2.0 survey waves respectively.

Safety climate metrics

For this analysis, the primary predictors of interest were the FOCUS safety climate scales Management Commitment to Safety and Supervisor Support for Safety (Taylor et al. 2019). Each of these constructs have 7-items with a 5-point Likert scale response. The psychometric properties of the measures, in terms of construct and criterion-related validities, have been previously reported (Taylor et al. 2019). Individual survey responses were aggregated to the group level (fire department) to calculate a departmental mean score (5-point scale) for Management Commitment to Safety and Supervisor Support for Safety, separately. The departmental means were converted to a 100-point scale for interpretability by the fire service and are presented as such in this paper.

Outcome variables

We analyzed the following outcomes: safety behaviors (Safety Behavior, Safety Compliance), organizational outcomes (Engagement, Job Satisfaction), and injury outcomes (injury status in the past 12 months). Safety Behavior measures the safety awareness and behaviors among firefighters. Safety Compliance measures the degree to which firefighters act in accordance with established safety protocols, processes, and standards regarding fire runs (Taylor et al. 2019). Engagement measures the work-related state characterized by absorption and dedication to the job (Taylor et al. 2019). Engagement was assessed separately on EMS and fire service runs. Job Satisfaction measures the degree of positivity about work (Taylor et al. 2019). These four metrics were asked on a 5-point Likert scale, anchored with Strongly Disagree and Strongly Agree. The number of items per metric are as follows: 3-items (Safety Behavior); 4-items (Safety Compliance and Job Satisfaction); 6 items (Engagement). Individual survey responses were aggregated to the group level (fire department) to calculate a departmental mean score for each outcome. The departmental means were converted to a 100-point scale for interpretability by the fire service. The analyses presented in this paper use the converted scores on the 100-point scale. Respondents were asked about their injury status, “During the past 12 months were you injured while performing your job?”, with the following response options: yes, no. Respondents were asked, “Did that injury require medical treatment or consultation?”, with response options of yes or no. We cleaned this variable to report back responses for only those that indicated they were injured in the past 12-months.

Statistical analysis

We reported descriptive statistics of all continuous (mean ± standard deviation (SD), range) and categorical (counts, percentages) variables from both analytic samples. In addition to overall summaries, we stratified by organization type (career, volunteer) to evaluate any potential differences related to organizational structure of the participating fire departments. Pearson correlation matrices were computed to investigate correlations between continuous measures at the department and individual levels.

Organizational Outcomes: Multivariable linear regression models were used to estimate the relationship between organizational outcomes and FOCUS safety climate, adjusting for departmental characteristics. Individual models were run for each organizational outcome and FOCUS safety climate predictor (Management Commitment to Safety and Supervisor Support for Safety). The linear regression models were adjusted for roster size, annual call volume, and population served. These models were run for the entire sample of departments and stratified by organization type. For models stratified by organization type, the adjustment variables were recategorized based on quartiles for each corresponding organization type due to inherent differences in size for career versus volunteer departments.

Individual Outcomes: Given the clustered nature of the data, multilevel logistic regression models were used to estimate the odds of self-reported injury from the past 12 months with FOCUS safety climate mean scores (Management Commitment to Safety, Supervisor Support for Safety). For our analysis, we analyzed both survey waves independently from one another. Further, in each wave, individuals were nested within departments for a multilevel approach. These models were run for each predictor for the total sample and then stratified by organization type. We calculated estimated associations using generalized estimated equation (GEE) methods to account for potential correlation between individuals within a department. Individual identifiers were not collected for respondents, thus linkage across survey respondents could not be taken into account. Effects were summarized using odds ratios (OR) and 95% confidence intervals (CI). These models were adjusted for age, years of experience, sex (male, female), and officer status (non-officer, officer, leadership). To conduct a complete case analysis for the logistic regression model we made the following additional exclusions: individuals missing injury status (n_{FOCUS 1.0} = 353; n_{FOCUS 2.0} = 127), age (n_{FOCUS 1.0} = 1028; n_{FOCUS 2.0} = 816), years of experience (n_{FOCUS 1.0} = 1327; n_{FOCUS 2.0} = 400), sex (n_{FOCUS 1.0} = 828; n_{FOCUS 2.0} = 528), and officer status (n_{FOCUS 1.0} = 218; n_{FOCUS 2.0} = 199). The final analytic sample for the logistic regression models were 17,627 individuals and 275 departments for FOCUS 1.0 wave and 14,806 individuals and 170 departments for FOCUS 2.0 wave. The two FOCUS waves were analyzed separately for this analysis.

Sensitivity analyses included: (1) Organization type as a covariate in the linear and logistic regression models as an alternative to stratification to increase power, due to small sample size of volunteer departments, and assess if the regression results are similar; (2) Exclude departments that have assessed their safety climate using the FOCUS survey more than once as their department may be more attune of their overall safety; (3) Exclude departments that did not achieve 60% response rate from their survey response (n_{FOCUS 1.0} = 60; n_{FOCUS 2.0} = 33); and (4) Compare the FOCUS 2.0 descriptive statistics among departments that finished assessing between September 1, 2019 and March 1, 2020 (n = 82) and those that finished assessing between April 1, 2020 and July 1, 2020 (n = 88), due to the impact of COVID-19 response among a subset of U.S. fire departments (Raposa et al. 2023).

Statistical significance was set to < 0.05 for all analyses. All statistical analyses were conducted using SAS 9.4 (Cary, North Carolina). The protocol received Institutional Review Board approval from Drexel University.

Results

Descriptive statistics

The individual-level demographics and characteristics were similar between the FOCUS 1.0 and 2.0 survey waves (Table 1). In the FOCUS 1.0 analytic sample there were 22,719 respondents (n_{1.0 career} = 21,551 [94.9%]; n_{1.0 volunteer} = 1168 [5.1%]) and in the FOCUS 2.0 sample there were 16,882 respondents (n_{2.0 career} = 16,421 (97.2%); n_{2.0 volunteer} = 461(2.8%)). Overall, the majority of respondents were males (n_{FOCUS 1.0} = 20,611 [90.7%]; n_{FOCUS 2.0} = 15,258 [90.4%]), non-officers (n_{FOCUS 1.0} = 15,548 [68.4%]; n_{FOCUS 2.0} = 11,206 [66.4%]), and non-Hispanic Whites (n_{FOCUS 1.0} = 16,086 [70.8%]; n_{FOCUS 2.0} = 11,733 [69.5%]). For both samples there was a higher prevalence of injuries in career departments (n_{FOCUS 1.0} = 3778 [17.5%]; n_{FOCUS 2.0} = 3072 [18.7%]) than volunteer departments (n_{FOCUS 1.0} = 103 [8.8%]; n_{FOCUS 2.0} = 34 [7.4%]). Among those that were injured 68.7% (FOCUS 1.0) and 65.9% (FOCUS 2.0) required medical treatment following the injury.

Table 1 Descriptive characteristics of the FOCUS 1.0 and 2.0 analytic samples stratified by fire department organization type

For the FOCUS 1.0 analytic sample there were 275 departments (n_{1.0 career} = 206 (74.9%); n_{1.0 volunteer} = 69 (25.1%)) and for the FOCUS 2.0 analytic sample there were 170 departments (n_{2.0 career} = 145 (85.3%); n_{2.0 volunteer} = 25 (14.7%)). Most departments were career departments (74.9% in FOCUS 1.0 and 85.3% in FOCUS 2.0). The mean scores, standard deviations (SD), and ranges for safety climate at the department level are reported (Table 1). For the total population, we observed approximately a 10-point difference in the mean scores for Engagement on EMS calls (mean_{FOCUS 1.0} = 73.8; mean_{FOCUS 2.0} = 73.9) versus Engagement on fire calls (mean_{FOCUS 1.0} = 82.2; mean_{FOCUS 2.0} = 83.8). We observed a similar difference of approximately 10-points among career departments for Engagement on EMS calls (mean_{FOCUS 1.0} = 72.7; mean_{FOCUS 2.0} = 73.0) versus Engagement on fire calls (mean_{FOCUS 1.0} = 81.7; mean_{FOCUS 2.0} = 83.5). Whereas the mean scores for volunteer departments were closer to 6-point difference for Engagement on EMS calls (mean_{FOCUS 1.0} = 77.1; mean_{FOCUS 2.0} = 78.8) versus Engagement on fire calls (mean_{FOCUS 1.0} = 83.6; mean_{FOCUS 2.0} = 85.2). The mean scores of Job Satisfaction (mean_{FOCUS 1.0} = 78.2; mean_{FOCUS 2.0} = 78.1), Safety Behavior (mean_{FOCUS 1.0} = 86.0; mean_{FOCUS 2.0} = 86.6), and Safety Compliance (mean_{FOCUS 1.0} = 86.4; mean_{FOCUS 2.0} = 88.3) were fairly similar between samples, which held when stratified by organization type. We observed an increase in mean scores for Management Commitment to Safety when stratified by organization type, with volunteer departments having a higher mean score than career departments in both sample waves (Table 1; Fig. 1). For FOCUS 1.0, we observed an 11-point difference in the mean Management Commitment to Safety scores between career and volunteer departments, with volunteer departments having the highest mean score. For FOCUS 2.0, we observed a 15-point difference between career and volunteer departments, with volunteer departments having the highest mean score. A similar increase in mean scores by organization type was not observed for Supervisor Support for Safety. Box plots of the FOCUS safety climate metrics by department size variables (roster size, annual call volume, and population served) appear in Additional file 2: Fig. 2A and B. For both FOCUS waves, we observed more variation in the mean Management Commitment to Safety scores with the different size variables compared to the mean Supervisor Support for Safety scores. We observed that larger departments had lower mean Management Commitment to Safety scores compared to smaller departments.

Fig. 1

Box and whisker plots comparing mean FOCUS safety climate scores by organization type. A Comparison between mean safety climate scores (Management Commitment to Safety and Supervisor Support for Safety) by organization type for the FOCUS 1.0 survey wave. B Comparison between mean safety climate scores (Management Commitment to Safety and Supervisor Support for Safety) by organization type for the FOCUS 2.0 survey wave

There was variation in fire departments participation across the FEMA regions (Table 1). FEMA region 5 (Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin) had the most participation (25.5%_{FOCUS 1.0}; 22.4%_{FOCUS 2.0}). The FEMA region that had the least participation (5.5%_{FOCUS 1.0}; 4.7%_{FOCUS 2.0}) was FEMA region 2 (New Jersey and New York).

The Pearson correlation matrices are visualized at the department (Table 2) and individual level (Table 3). At the department level, we observed moderate positive correlations between annual call volume with roster size [r_{FOCUS 1.0} = 0.58; r_{FOCUS 2.0} = 0.88]. and with population served [r_{FOCUS 1.0} = 0.69; r_{FOCUS 2.0} = 0.58]. At the department level for both samples, we observed moderate positive correlations between Job Satisfaction and FOCUS safety climate (Management Commitment to Safety [r_{FOCUS 1.0} = 0.67; r_{FOCUS 2.0} = 0.69] and Supervisor Support for Safety [r_FOCUS1.0 = 0.62; r_FOCUS2.0 = 0.54]). For both survey waves, we observed moderate positive correlations between Safety Behavior and Safety Compliance [r_FOCUS1.0 = 0.59; r_FOCUS2.0 = 0.61].

Table 2 Department level Pearson correlation coefficient matrices for FOCUS 1.0 and FOCUS 2.0

Table 3 Individual level Pearson correlation coefficient matrices for FOCUS 1.0 and FOCUS 2.0

For individual-level characteristics, the notable correlations were not consistent between two samples (Table 3). For both samples, we observed a moderate positive correlation between Safety Behavior and Management Commitment for Safety [r_FOCUS1.0 = 0.68; r_FOCUS2.0 = 0.61]. For the FOCUS 2.0 survey wave, we observed a moderate positive correlation between Job Satisfaction and FOCUS safety climate (Management Commitment to Safety [r_{FOCUS 2.0} = 0.68] and Supervisor Support for Safety [r_{FOCUS 2.0} = 0.67]).

Department-level organizational outcomes

In FOCUS 1.0, we observed that all outcomes were positively associated with both FOCUS Management Commitment to Safety and Supervisor Support for Safety when we did not stratify by organization type (Table 4). For FOCUS 2.0, we observed that all outcomes, except for Safety Compliance, were positively associated with Management Commitment to Safety. Further, we observed that all outcomes, except for Engagement on EMS runs, were positively associated with Supervisor Support for Safety.

Table 4 Linear regression models examining the relationship between safety climate scores and safety behaviors/organizational outcomes for FOCUS 1.0 and FOCUS 2.0 overall and stratified by organization type

Overall, and stratified by organization type, we observed positive associations for Safety Behavior and Job Satisfaction with both FOCUS safety climate metrics. The estimated change in Safety Behavior was associated with a one-unit increase in mean Management Commitment to Safety (B_{FOCUS 1.0 overall}: 0.39, 95% CI 0.31–0.47; B_{FOCUS 2.0 overall}: 0.35, 95% CI 0.24–0.46). The estimated change in Safety Behavior was associated with a one-unit increase in Supervisor Support for Safety (B_{FOCUS 1.0 overall}: 0.67, 95% CI 0.51–0.82; B_{FOCUS 2.0 overall}: 0.58, 95% CI 0.36–0.80). The estimated change in Job Satisfaction was associated with a one-unit increase in mean Management Commitment to Safety (B_{FOCUS 1.0 overall}: 0.69, 95% CI 0.61–0.77; B_{FOCUS 2.0 overall}: 0.62, 95% CI 0.52–0.71). The estimated change in Job Satisfaction was associated with a one-unit increase in Supervisor Support for Safety (B_{FOCUS 1.0 overall}: 1.07, 95% CI 0.89–1.25; B_{FOCUS 2.0 overall}: 0.88, 95% CI 0.64–1.11). Consistently the magnitude of estimated change was strongest for Job Satisfaction, as indicated by a higher beta estimate. The results stratified by organization type are presented in Table 4. For the FOCUS 2.0 survey wave, results for volunteer departments should be interpreted cautiously due to smaller sample sizes.

Individual-level safety outcomes

We reported the results of the unadjusted and adjusted multilevel logistic regression analyses in Table 5. In the adjusted models for FOCUS 1.0, we observed a 2% decrease in the odds of injury reported in the last 12 months associated with a one-unit increase in mean Management Commitment to Safety scores (OR_{FOCUS1.0 overall}: 0.98, 95% CI 0.97–0.99). Additionally, we observed a 5% decrease in the odds of injury reported in the last 12 months associated with a one-unit increase in mean Supervisor Support for Safety scores (OR_{FOCUS 1.0 overall}: 0.95, 95% CI 0.93–0.97; OR_{FOCUS 1.0 career}: 0.95, 95% CI 0.92–0.98).

In the adjusted models for FOCUS 2.0, we observed an association between in the odds of injury reported the last 12 months and mean Management Commitment to Safety scores (OR_{FOCUS 2.0 volunteer}: 0.90, 95% CI 0.85–0.95).

Table 5 Multilevel logistic regression between safety climate and injury status among individuals stratified by organization type for FOCUS 1.0 and 2.0

Sensitivity analyses

We conducted sensitivity analyses to assess the robustness of our findings across various scenarios (Additional file 1: Tables 1–9). Notably, when organization type was considered as a factor, we no longer observed an association between the odds of injuries and Management Commitment to Safety in the FOCUS 1.0 survey wave sample (Additional file 1: Table 2). When we excluded the departments that assessed with FOCUS more than once, most of our associations held (Additional file 1: Tables 3–4). We did observe that the association between Safety Behavior and FOCUS safety climate as well as Job Satisfaction and FOCUS safety climate was null for volunteer departments in the FOCUS 2.0, which is likely due to the reduced sample size (Additional file 1: Table 3).

When we excluded departments that did not achieve a 60% response rate based on their reported roster size and number of departmental respondents (Additional file 1: Tables 5–6) we no longer observed an association between Engagement (on fire or EMS runs) and Management Commitment to Safety for volunteer departments (Additional file 1: Table 5), however the magnitudes of the associations were similar for Engagement on EMS runs and there was overlap in the 95% CI for Engagement on fire runs. The associations between Safety Behavior and FOCUS safety climate as well as with Job Satisfaction and FOCUS safety climate were null for volunteer departments. From the logistic regression analysis, compared to the main analysis, we no longer observed associations between the odds of injury and Supervisor Support for Safety among individuals in career departments for the FOCUS 1.0 survey wave (Additional file 1: Table 6). However, there was overlap in the 95% CI. The results for the volunteers should be interpreted cautiously due to reduced sample size for both survey waves.

In the FOCUS 2.0 data, we examined descriptive characteristics and regression analyses for departments that finished assessing before (September 1, 2019–March 1, 2020) and during (April 1, 2020–July 1, 2020) the U.S. response to the COVID-19 pandemic (Additional file 1: Tables 7–9). Descriptive characteristics were largely similar between both groups. Notably, those that finished assessing before March 1, 2020, reported a 16.5% injury rate in the last 12 months, compared to 19.4% for those that finished assessing after March 1, 2020. Linear regression results indicated associations between Engagement on EMS runs and Management Commitment to Safety (B: 0.17, 95% CI 0.04–0.29; p-value: 0.01), as well as between Engagement on fire runs and Management Commitment to Safety (B: 0.15, 95% 0.04–0.26; p-value: 0.01) for career departments who finished assessing before March 1, 2020, which were not observed in the main analysis or for those who finished assessing after March 1, 2020 (Additional file 1: Table 8). Further, in comparison to the main analyses the associations between Management Commitment to Safety and Supervisor Support for Safety with Job Satisfaction no longer held for volunteer departments for either assessment period. Logistic regression results showed a 2% decrease in the odds of injuries with a one-unit increase in Management Commitment to Safety among individuals who finished assessing before March 1, 2020, which was not observed in the overall FOCUS 2.0 sample (held (Additional file 1 Table 9). Further, we no longer observed an association with the decrease in odds of injuries for a one-unit increase in Management Commitment to Safety among volunteers that finished assessing before March 1, 2020.

Discussion

The purpose of this study was to investigate the descriptive statistics of respondents to the FOCUS 1.0 and the FOCUS 2.0 survey waves, to better understand if two independent convenience samples were reflective of the geographically stratified random sample (FOCUS beta-test). A prior analysis of FOCUS beta-test data observed a dose response relationship when examining mean FOCUS Management Commitment to Safety by organization type (career, combination, and volunteer) (Geczik et al. 2022). Mean scores were lower among career departments compared to combination departments and to volunteer departments, however there was no difference in the mean scores of Supervisor Support for Safety (Geczik et al. 2022). While this current study only captured organization type on two levels, we still observed that career departments had a lower mean score of Management Commitment to Safety compared to volunteer departments. This further supports our previous claim that department size does matter, especially regarding member perceptions of their leadership’s response to their safety. Furthermore, Christian et al., found that management commitment to safety was a stronger driver of safety climate compared to supervisor support (Christian et al. 2009), so departments should focus on improving that score. Given discussions with our fire service partners about the structures of their organizations, the limited face time that leadership has with their members may be driving this phenomenon, especially in larger departments. We consistently observed lower mean Management Commitment to Safety scores compared to mean Supervisor Support for Safety scores using the data in this analysis and in the previously published analysis of the FOCUS beta-test data (Geczik et al. 2022). The lower mean Management Commitment to Safety scores are indicative of fire department members not necessarily hearing from their upper management leadership regarding their safety. If fire departments implement more face time between department leadership and members at the individual stations, the perceptions to Management Commitment to Safety may increase thus further reducing the odds of injuries. There are inherent organizational differences between career and volunteer fire departments, such as number of calls, size of population served, and work hours, that may also be affecting the differences in these scores.

A novel component of the initial analysis of the FOCUS beta-test data was to stratify by organization type, which was also assessed for the FOCUS 1.0 and 2.0 survey waves, however there were differences in the stratified groups sample sizes. In our current study we did observe that FOCUS 1.0 had a larger overall sample size compared to FOCUS 2.0 and FOCUS beta-test, which made it a more robust sample of career and volunteer departments. The FOCUS 2.0 sample had more career departments (n = 170) and a similar number of volunteer departments (n = 25) compared to the FOCUS beta-test (n_career = 125; n_volunteer = 24). We observed similar demographic characteristics between the individual respondents from the FOCUS 1.0 and FOCUS 2.0 survey waves, which is reassuring for comparing across the two convenience samples. Of the organizational outcomes analyzed in our study, Job Satisfaction was the most robust in terms of its relationship to FOCUS safety climate, which was consistent with the beta-test data. For this analysis we were able to examine the relationship between Safety Behavior and FOCUS safety climate because it was asked on both survey waves. We observed an association overall and by organization type for this downstream outcome as well. For our current analysis, this outcome was also associated with FOCUS safety climate for both samples. The reduction in the odds of injuries we observed for FOCUS safety climate among career departments in the FOCUS beta-test sample (Geczik et al. 2022) and FOCUS 1.0 sample was not observed in the FOCUS 2.0 sample. Further, unique to the FOCUS 2.0 sample, we observed a reduction in the odds of injuries associated with Management Commitment to Safety among volunteer departments. This association was not observed for volunteer departments in FOCUS 1.0 or the FOCUS beta-test (Geczik et al. 2022).

Based on the observed differences in the data, it would appear that the FOCUS 1.0 and 2.0 samples are not truly reflective of the FOCUS beta-test sample. This is likely due to the fact that the samples are from different departments overall, which would be expected given our understanding that safety climate is unique to the organization that is being examined. It is possible that the different sampling methods, convenience sampling versus random sampling, are contributing to these differences. Due to the higher number of departments that assessed in FOCUS 1.0 compared to FOCUS 2.0 and our previously studied FOCUS beta-test sample, we are unable to definitively conclude that the convenience sample of departments are reflective of the geographically stratified random sample.

Study strengths and limitations

A strength of our current analysis was the large sample size we had of individual respondents across both survey waves, which provided us with more data to analyze the perceptions to the different metrics assessed in the FOCUS survey. Our study had some important limitations. Considerations should be made regarding the FOCUS 2.0 sample given the impact of COVID-19 on the U.S. fire service. As evidenced by our sensitivity analysis, there are some associations that are only present among the sample that completed assessing prior to March 1, 2020. Previous research has identified the impact of the COVID-19 pandemic on a sample of U.S. fire departments (Raposa et al. 2023). The researchers identified that there were concerns regarding the “low and decreasing scores” of Management Commitment to Safety in the sample (Raposa et al. 2023), which could explain why we observed associations in the regression analyses for Management Commitment to Safety among the departments that completed assessment prior to March 1, 2020. Additionally, the researchers identified that Engagement on EMS runs and Engagement on fire runs had an average score decrease over their 6-month study period (Raposa et al. 2023). Again, this may explain why we did not observe an association between these metrics in the overall sample, but did for the departments that assessed before March 1, 2020.

Further, given the voluntary participation for members within participating departments, there may be non-response bias present in our study samples. At the department level, there may be the presence of selection bias, as departments that are more in tune with their safety may be inclined to enroll to assess. Additionally, the data used for this study is self-reported by individual respondents, which may result in social desirability bias. Based on our previous work, we are not overly concerned with the presence of non-response bias, selection bias, or social desirability bias due to the heterogeneity of responses for each of the metrics (ranges in terms of mean scores). There is potential that nondifferential misclassification by outcome may arise, due to the self-reporting of injury among respondents due to recall bias. The outcome will likely be underreported in instances where minor injuries in the last year were forgotten. If present, we anticipate that this would bias the results towards the null. Additionally, the wording on the survey for the injury status question is open to interpretation by the respondent. The survey question does not account for varying levels of severity and does not ask about the number of injuries an individual experienced in a year, so we are not able to ascertain the full burden of injuries among our survey population.

There is potential that aggregation fallacy is present for our logistic regression results as we are attributing the departmental mean scores of Management Commitment to Safety and Supervisor Support for Safety to the individuals within departments. This was done because FOCUS safety climate constructs are meant to be measured at the group level. If present, this may bias our results toward the null. Finally, while we accounted for the department and individual level clustering of this data, we were unable to account for station level hierarchical clustering due to data quality issues for the station level identifiers that were inconsistently self-reported by individuals and their departmental contact. Previously, Supervisor Support for Safety has been identified as a station level construct (Taylor et al. 2019), however in our current analysis we could only assess it on the departmental level. However, our findings of positive relationships with Supervisor Support for Safety and odds of injuries at the department level indicate these relationships might be stronger when assessed at the station level. Future survey iterations should create numeric only options for station identifiers to prevent data management limitations in classifying station level identifiers.

Additionally, the stratification by organization type (career, combination, volunteer) turned out to be an important variable, so it should be added back into future waves of FOCUS. While we consider our large sample a strength of this study, we must mention that the small number of volunteer departments in the FOCUS 2.0 survey wave presented some concerns regarding power. Future iterations of FOCUS should aim to recruit a larger sample of volunteer and combination departments to further support the stratification of these constructs by organization type.

Conclusion

This evaluation of the descriptive statistics of two independent convenience samples helps inform our understanding of safety climate within the sample of U.S. fire department that have taken the FOCUS survey. The findings from our current analysis suggest that the FOCUS 1.0 and 2.0 survey waves are not reflective of the geographically stratified random sample from the FOCUS beta-test survey wave. However, both of these samples are larger, in terms of individual respondents, than the initial FOCUS wave which highlights the reach of the FOCUS assessment tool.

Our current research confirms previous findings that the size of a fire department does have an impact on the organizational safety climate, however additional analyses of convenience samples with the three levels of organization types represented is warranted. Further, our regression analyses show that both Management Commitment to Safety and Supervisor Support for Safety are important upstream factors in the maintenance of positive organizational outcomes and reduction of firefighter injury. Future research should investigate potential interventions to help our fire service partners increase these scores, with a particular focus on Management Commitment to Safety. Interventions can aim to support departments in maintaining their Supervisor Support for Safety scores.