- Original Contribution
- Open Access
End-user experiences with two incident and injury reporting systems designed for led outdoor activities - challenges for implementation of future data systems
Injury Epidemiology volume 6, Article number: 39 (2019)
Injury and incident (near miss) prevention is heavily dependent upon robust and high-quality data systems. Evaluations of surveillance systems designed to report factors associated with incidents and injuries are essential to understand their value, as well as to improve their performance and efficiency. Despite, this there have been few such evaluations published in the peer-review literature.
The attitudes and experiences of industry representatives who used one of two variants of an incident and injury surveillance system to collect injury and incident data for the led outdoor activity setting were obtained through an online self-report survey following a 12-month trial. Survey respondents were 18 representatives of 33 organisations who were users of a comprehensive incident reporting and surveillance system – the Understanding and Preventing Led Outdoor Accidents Data System Software Tool (UPLOADS-ST) - and six out of 11 users of a modified system (UPLOADS-Lite). The survey collected information on user experiences in relation to system training, accessibility, ease of use, security, feedback and perceived value to the sector of collating and reporting data across organisations.
Only four UPLOADS-ST responding users found the system easy to use and just three considered entering incident reports to be easy. However, many considered the training on reporting incidents to be sufficient and that the incident reports contained relevant details. Fewer than half of respondents (seven for UPLOADS-ST, three for UPLOADS-Lite) believed entering data was a good use of staff time and resources. Nonetheless, a majority of respondents (seven for UPLOADS-ST, five for UPLOADS-Lite) found the reporting format easy to read and felt the information provided was useful for their organisation.
Usability barriers to incident reporting were identified, particularly for UPLOADS-ST, including time constraints and user friendliness. The majority of users believed aggregating and reporting incident and injury data across organisations would be of value in making the led outdoor activity sector safer. Improving the utility of the surveillance systems will assist in ensuring their sustainability in the led outdoor activity sector.
Accurate and timely information about injuries and near-miss incidents from data collection and reporting systems is needed to inform the identification and implementation of preventive solutions that can reduce the risk of harm. The perceived relevance and usefulness of such data systems can influence user engagement with them and hence can impact on reporting rates and data quality (Ekegren et al., 2014). The World Health Organization and Centers for Disease Control injury surveillance guidelines (Holder et al., 2001) highlight the quality attributes of good surveillance systems, which include simplicity, flexibility, acceptability, reliability, utility, sustainability and timeliness. The design of any surveillance system and its user interface is critical, as is evaluation of its effectiveness (Mitchell et al., 2009; Bromley et al., 2018). The effectiveness of training and resource materials provided to guide use of the system, and the level of infrastructure and resourcing devoted to data collection and reporting by organisations, are important considerations for data system implementation and sustainability (Kerr et al., 2014; Finch & Staines, 2018). Injury surveillance approaches have long been used in the sports injury settings (Finch, 1997) but have not been widely applied in broader activity settings, especially those not undertaken in traditional sports participation environments.
Led outdoor activities, such as bushwalking, kayaking and abseiling, are defined as facilitated or instructed activities within outdoor education and recreation settings with an associated learning goal. The sector that delivers these activities is both diverse and complex. The number of organisations that deliver such programs is unknown but it is expected to be in Australia’s top 20 sectors of employment (Service Skills Australia, 2015). The organisations themselves range from government bodies to commercial enterprises to not-for-profits (including volunteer organisations). The number of people who participate in such activities under the leadership of a sector employee, either through school programs or outdoor recreation programs, is also unknown but is predicted to be on the increase (Service Skills Australia, 2015). These sector-size limitations also mean that there is currently a paucity of high quality information about the epidemiology and causation of led outdoor activities injuries (Dickson, 2012; Salmon et al., 2010). The Understanding and Preventing Led Outdoor Accidents Data System (UPLOADS) was developed in partnership with key sector representatives to support the collection and reporting of injury and incident data in the led outdoor activity sector in Australia (Salmon et al., 2017; Goode et al., 2018). The system features a reporting form and classification scheme for coding the contributing factors involved in injury, illness, and near miss incidents based around a systems thinking framework (Rasmussen, 1997), so as to provide important information to inform the design and targeting of preventive programs across the Australian led outdoor activity setting (Goode et al., 2018).
Usability of the initial-design of the UPLOADS system was first evaluated after an initial 6-month trial in 2013. In response to some of the trial feedback, which included concerns about system complexity and data entry burden, a modified “Lite” version of the system was developed to reduced data reporting requirements (Goode et al., 2015). Over the following 12-months, led outdoor activities providers and industry representatives were given the option of using either the UPLOADS Software Tool (UPLOADS-ST) or UPLOADS-Lite to record and report on injuries and incidents relating to their organisations. This paper reports the post-implementation evaluation of these users’ experiences in relation to the UPLOADS-ST or UPLOADS-Lite systems. In doing so, it provides useful information to guide the ongoing implementation and refinement of data collection and reporting systems in this setting.
The rationale behind, and processes used, to design and define the specific content of the UPLOADS-ST and UPLOADS-Lite systems are published in full elsewhere (Salmon et al., 2017; Goode et al., 2018; Goode et al., 2017; Goode et al., 2016a). UPLOADS-Lite was a modified version of UPLOADS-ST, following feedback received from users of the latter after a 6 month pilot trial (Goode et al., 2015). Table 1 summarises the key design features of the two systems.
Both the UPLOADS-ST and UPLOADS-Lite were trialled within organisations identified as being part of the Australian led outdoor activities sector. Organisations were invited to participate via peak body and professional membership association newsletters, and relevant social media pages. Interested organisations were asked to invite a senior staff member in a safety-related role to be the primary contact. This person was responsible for undertaking training in the system, collecting and entering all data, and providing training to other staff within their organisations on reporting incidents. Organisations self-selected whether to trial UPLOADS-ST or UPLOADS-Lite based on the initial training session. Those that elected to participate in the UPLOADS-Lite trial identified at baseline that they did not want to do the full contributory factor analysis as part of UPLOADS-ST, usually because they felt they did not have enough incidents to warrant that effort.
This paper reports findings from the post-trial survey of 33 registered organisational users of UPLOADS-ST and 11 registered organisational users of UPLOADS-Lite. The proportion of all organisations that these 44 users represents is unknown, but the trial did involve all of the major national peak bodies, and some state-based organisations, that are driving this industry. The organisations that participated in the trial were self-selected from a broad invite to the sector from the research team. There is therefore a potential sampling bias present in that only organisations with a strong desire to improve safety, or those that did not like or have an existing incident reporting system, participated.
An online survey was administered to representatives of organisations that had used the system to ask them about their views about the system they used. Respondents were the nominated representative of the led outdoor activities organisation participating in the trial. The survey captured information on respondent demographics, experiences with the system, views about the system’s usability and sustainability, beliefs about the security and utility of the data. It was based on the previous survey successfully used in the pilot study to obtain views on use of the initial UPLOADS-ST system (Goode et al., 2018; Goode et al., 2015). The survey was tailored slightly for users of UPLOADS and UPLOADS-Lite, to reflect the relevant system design, and most items were rated on a 6-point Likert scale (unsure, strongly disagree, disagree, neutral, agree, strongly agree).
Because of the nature of the survey responses, and the relatively small numbers of survey respondents, results are presented as the number of responses and the disagree/strongly disagree, agree/strongly agree, and unsure/neutral, response categories were combined. There were insufficient numbers in each group to enable robust statistical comparisons across the two-systems so only descriptive findings are presented.
The overall survey response from UPLOADS-ST users was n = 18 from 33 participating organisations, of whom 14 responded that they used the UPLOADS-ST to enter data during the trial. From an initial 11 organisations who signed-up to use UPLOADS-Lite, six responded to the survey but only four indicated that they entered any data in the system. A mix of mid and large size organisations responded but the profiles of the responding organisations in both groups were similar in terms of size and focus. Respondents were typically male, held both a managerial position and led outdoor activities within their organisation, had 0–5 years’ experience in the led outdoor activities sector, held outdoor qualifications, and were the system administrator for their organisation at the start of the trial (Goode et al., n.d.). There was no information on the characteristics of the non-responders.
Table 2 summarises the respondents’ views about the system. Only four of the UPLOADS-ST users stated that the system was easy to use, and only three agreed that it was easy to enter incident report data. In contrast, the majority of UPLOADS-LITE users who tried the system considered that it was easy to use (n = 3) and entering data was easy (n = 3). The majority of UPLOADS-ST users and all UPLOADS-Lite users agreed/strongly agreed that the security and privacy of the systems were adequate for organisations and led outdoor activities participants.
A majority of respondents (ten for UPLOADS-ST, four for UPLOADS-Lite) indicated they were committed to ongoing data reporting through the system. Seven of the UPLOADS-ST users considered that their organisation’s awareness of the need to report data had improved but only one UPLOADS-Lite user felt the same over the 12 months. Many users of both systems indicated that they also needed to enter their data into other data systems for their organisation’s needs, with the suggestion that this added to their administrative burden. Less than half of respondents (seven UPLOADS-ST, three UPLOADS-Lite) considered that entering data reports was a good use of staff time and resources. Despite these perceived difficulties with the two systems that had been trialled, the vast majority of respondents (12 UPLOADS-ST, six UPLOADS-Lite) believed that having a national incident collection and reporting would make the lead outdoor activity domain safer.
Evaluation of injury data collection and reporting systems is essential to obtain feedback about their overall operation, determine whether their purposes are met, improve their performance and ensure their effectiveness (Ekegren et al., 2014; Bromley et al., 2018; Kerr et al., 2014; Calba et al., 2015; Romaguera et al., 2000; Ekegren et al., 2016; German et al., 2001). Importantly, the attributes of such systems need to be considered from the viewpoints of the end-users who need to use them (Bromley et al., 2018; Auer et al., 2011). This study presents the views of practitioners from the participating organisations in relation to the use of one of two reporting systems, UPLOADS-ST and UPLOADS-Lite, designed for the reporting of incident and injuries in the led outdoor activities sector.
Considering the usability of the systems, more UPLOADS-Lite than UPLOADS-ST users agreed that the system’s software interface was easy to use and that it was easy to enter data. This is not surprising given that UPLOADS-Lite was designed specifically as a less-resource intensive and easier to use version of UPLOADS-ST.
Regarding sustainability of both UPLOADS-ST and UPLOADS-Lite as an approach for ongoing surveillance, the majority of users indicated that their organisation was committed to maintaining data collection and that incident reporting was a good use of staff time and resources. However, the requirement to report adequate information on incidents to enable meaningful analysis and identification of contributory factors across the system needs to be balanced with the workload imposed (Evans et al., 2006). On this aspect, there was stronger support for UPLOADS-Lite. The data reported via UPLOADS-Lite did not provide sufficient information for the contributing organisations to undertake detailed analyses of incidents but those users were still keen for their data to contribute to a national incident database for these purposes and for an external agency to undertake the detailed incident analysis. A key challenge for injury surveillance and incident reporting generally is the development of systems that incur a low data collection burden yet still provide sufficient data to enable informative analyses.
A number of barriers to the sustainability of the systems were also identified. Specifically, the majority of organisations reported they still needed to enter the same data into their other organisational systems, affecting the administrative burden of their roles. In such circumstances, the requirements of externally based surveillance systems are likely to be considered less important to an organisation’s operations and this could influence data completeness and use of the system. A related issue was the lack of resources at the organisational levels available to enter reports and analyse the incident data collected. This could be at least partially due to the requirement to report incidents through multiple channels. Taken together, these three factors pose a significant threat to the sustainability of both the UPLOADS-ST and UPLOADS-Lite systems long term. It will also be important that a strong socio-political environment for the entire led outdoor activity sector provides additional support and incentives for organisations to contribute incident reports.
The utility of collected data is a key factor in sustaining reporting efforts over time (Ekegren et al., 2014; Mitchell et al., 2009; Bromley et al., 2018; Wu et al., 2008). Responses to the survey suggested the 14 organisations using UPLOADS-ST were making little direct use of the data they had been collecting, despite the fact that the system had been designed specifically to address industry-defined needs for such information (Salmon et al., 2017; Goode et al., 2014). It is possible that some of this was due to a perception that a particular participating organisation rarely experienced adverse incidents in its operations and this could have led to a belief that the reporting system was not relevant to them. In addition, it may be that this reflects a knowledge gap in terms of how to analyse and interpret multiple incident data sets. It is likely that organisations will require further education to increase their understanding of the value of reporting analysed data for their organisations, across all levels of risk, and ongoing guidance on how to analyse the incident data and contributory factors, and develop appropriate incident and injury prevention strategies (Salmon et al., 2010; Salmon et al., 2014; Goode et al., 2016b).
There are some limitations to the retrospective self-reported nature of this study. No independent assessment of the system performance and attributes could be made. As our focus was on the user experiences of these, the survey design was appropriate but could have been enhanced with the collection of more qualitative and open-ended responses. Non-responder bias cannot be excluded, as information was not collected on non-responders and only 55% of all organisations involved in the trial responded to the post survey. Not all responders answered every question because they had not all implemented the incident reporting within their organisation during the 12-months. It is possible that some non-responding organisations struggled to allocate internal resources to report injuries and incidents, and had either not implemented their UPLOADS system or had difficulty in completing the post-trial survey. Despite being anonymous, respondents may also have provided more socially acceptable responses for fear of identification, especially as they were responding on behalf of an organisation. Only one person from each organisation completed the survey, and other people from the same organisation may have had different experiences of the system they trialled. However, this possibility was minimised because the survey respondent was generally the major UPLOADS administrator for that organisation, and so would be most familiar with it. The sample size of respondents was not large and so the results may not be able to be generalised to all led outdoor activity providers, though the participants were from all organisation types and sizes across Australia. There may also be potentially important barriers towards the system use that were not included in the survey, which was designed to be succinct and not over burdensome for the respondents. Nonetheless, this study offers a foundation for future monitoring and implementation of the UPLOADS-ST and UPLOADS-Lite systems and has implications for the implementation of injury and incident surveillance systems more widely. Future research is warranted to determine the barriers and motivators of end-users towards the use of incident reporting systems, as a first step towards being able to increase end-user engagement with them into the future.
The capacity of any surveillance system to describe patterns of incidents and their contributory factors accurately is critical in order to develop effective incident prevention strategies (Finch, 1997; Calba et al., 2015; Ekegren et al., 2016). It is therefore essential that reporting systems correctly report the occurrence of incidents and associated injuries, as well as their contributory factors. Evaluation of the effectiveness of incident reporting systems is therefore critical. Guidelines for evaluation of a good surveillance system, include a broad selection of attributes (German et al., 2001). These need to be adapted for specific situations and tailored to each evaluation. While data reporting systems have the potential to provide key insights into injuries and incidents to inform their prevention, they rely on organisations using them. Their end users’ experiences and the resources of contributing organisations, in turn, influence this. Identified barriers to the sustainability of both the UPLOADS-ST and UPLOADS-Lite systems included other reporting responsibilities within organisations, and a lack of infrastructure available to assist with entering data reports and analyse the incident data collected required by UPLOADS-ST. While surveillance systems have the potential to provide key insights into injuries and incidents to improve risk management, this study illustrates that their use by relevant sector stakeholders is highly dependent on users’ experiences and the resources of contributing organisations. Organisations can vary considerably in terms of their commitment to data collection and reporting incidents and thus their capacity to engage in such activities. These activities are crucial requirements of a systems-based approach to prevention (Goode et al., 2018) and require ongoing resourcing. Although development of UPLOADS was driven by a cohort of led outdoor activity stakeholders, previous incident reporting systems in this domain, such as the NZ National Incident Database (Salmon et al., 2014), have not had sustained success because of a lack of continuing funding to support ongoing training and data collection, analysis, and dissemination functions. Future efforts are needed to further increase buy-in to the systems approach in the led outdoor activity setting, and increasing the direct value and worth of incident reporting, whilst also reducing the administrative burden of effective data collection systems.
Availability of data and materials
Understanding and Preventing Led Outdoor Accidents Data System
Modified version of the Understanding and Preventing Led Outdoor Accidents Data System Software Tool
The Understanding and Preventing Led Outdoor Accidents Data System Software Tool
Auer AM, Dobmeier TM, Haglund BJA, Tillgren P. The relevance of WHO injury surveillance guidelines for evaluation: learning from the aboriginal community-centered injury surveillance system (ACCISS) and two institution-based systems. BMC Public Health. 2011;11:744. https://doi.org/10.1186/471-2458-11-744.
Bromley S, Drew M, Talpey S, McIntosh A, Finch C. Collecting health and exposure data in Australian Olympic Combat Sports: Feasibility study utilizing an electronic system. Journal of Medical Internet Research Human Factors. 2018;5(4):e27. [Published on line 9/10/2018 as doi: 10.196/humanfactors.9541].
Calba C, Goutard FL, Hoinville L, Hendrikx P, Lindberg A, Saegerman C, et al. Surveillance systems evaluation: a systematic review of the existing approaches. BMC Public Health. 2015;15(1):448. https://doi.org/10.1186/s12889-015-1791-5.
Dickson TJ. Learning from injury surveillance and incident analysis Risk Management in the Outdoors: a whole-of-organisation approach for education. In: Sport and recreation. Cambridge, GB: Cambridge University Press; 2012. p. 204–30.
Ekegren C, Gabbe B, Finch C. Sports injury surveillance systems: a review of methods and data quality. Sports Med. 2016;46(1):49–65. [First published online 30 September 2015 as. https://doi.org/10.1007/s40279-015-0410-z].
Ekegren CL, Donaldson A, Gabbe BJ, Finch CF. Implementing injury surveillance systems alongside injury prevention programs: evaluation of an online surveillance system in a community setting. Injury Epidemiology. 2014;1(1):19. 0.1186/s40621-014-0019-y.
Evans SM, Berry JG, Smith BJ, Esterman A, Selim P, O'Shaughnessy J, et al. Attitudes and barriers to incident reporting: a collaborative hospital study. Quality and Safety in Health Care. 2006;15(1):39–43.
Finch C, Staines C. Guidance for sports injury surveillance: the 20-year influence of the Australian sports injury data dictionary. Injury Prevention. 2018;24(3):72–380. [First published Online First 27/12/2017 as. https://doi.org/10.1136/injuryprev-2017-042580.
Finch CF. An overview of some definitional issues for sports injury surveillance. Sports Med. 1997;24(3):157–63.
German RR, Lee LM, Horan JM, Milstein RL, Pertowski CA, Waller MN. Updated guidelines for evaluating public health surveillance systems: recommendations from the guidelines working group. Morbidity and Mortality Weekly Recommendations and Reports. 2001;50(Rr-13:1–35.
Goode N, Finch C, Cassell E, Lenné M, Salmon P. What would you like? Identifying the required characteristics of an industry-wide incident reporting and learning system for the led outdoor activity sector. Aust J Outdoor Educ. 2014;17(2):2–15.
Goode N, Read GJM, van Mulken MRH, Clacy A, Salmon PM. Designing system reforms: using a systems approach to translate incident analyses into prevention strategies. Front Psychol. 2016b;7. https://doi.org/10.3389/fpsyg.2016.01974.
Goode N, Salmon P, Lenne M, CF F. Translating systems thinking into practice: a guide to developing incident reporting systems. https://www.crcpress.com/Translating-Systems-Thinking-into-Practice-A-Guide-to-Developing-Incident/Goode-Salmon-Lenne-Finch/p/book/9781472436917. Boca Raton, Florida: CRC Press; 2018.
Goode N, Salmon PM, Lenné MG, Finch CF. The UPLOADS project: development of an Australian national incident dataset for led outdoor activities. Wilderness & Environmental Medicine. 2015;26(4):574–6. https://doi.org/10.1016/j.wem.2015.04.006.
Goode N, Salmon PM, Taylor NZ, Lenné MG, Finch CF. Lost in translation: the validity of a systemic accident analysis method embedded in an incident reporting software tool. Theor Issues Ergon Sci. 2016a;17(5–6):483–506.
Goode N, Salmon PM, Taylor NZ, Lenné MG, Finch CF. Developing a contributing factor classification scheme for Rasmussen's AcciMap: reliability and validity evaluation. Appl Ergon. 2017;64:14–26.
Goode N, Shaw L, Finch CF, Salmon P. Challenges of translating Rasmussen's Accimap into a usable, sustainable and useful incident reporting system: end user attitudes follwoing 12-months implementation. Cognition, Tcehnology & Work. 2019. In press.
Holder Y, Peden M, Krug E, Lund J, Gururaj G, Kobusingye O. Injury surveillance guidelines. Geneva: World Health Organization. Available from http://apps.who.int/iris/bitstream/10665/42451/1/9241591331.pdf. (Accessed 4/01/2019); 2001.
Kerr Z, Dompier T, Snook E, Marshall S, Klossner D, Hainline B, et al. National Collegiate Athletic Association Injury Surveillance System: review of methods for 2004–2005 through 2013–2014 data collection. J Athl Train. 2014;49(4):552–60. https://doi.org/10.4085/1062-6050-49.3.58.
Mitchell RJ, Williamson AM, O'Connor R. The development of an evaluation framework for injury surveillance systems. BMC Public Health. 2009;9:260. https://doi.org/10.1186/471-2458-9-260.
Rasmussen J. Risk management in a dynamic society: a modelling problem. Saf Sci. 1997;27(2):183–213.
Romaguera RA, German RR, Klaucke D. Evaluating public health surveillance. Principles and Practice of Public Health Surveillance. 2000;2:176–193.
Salmon P, Goode N, Lenné M, Finch C, Cassell E. Injury causation in the great outdoors: a systems analysis of led outdoor activity injury incidents. Accid Anal Prev. 2014;63:111–20.
Salmon P, Williamson A, Lenne M, Mitsopoulos-Rubens E, Rudin-Brown CM. Systems-based accident analysis in the led outdoor activity domain: application and evaluation of a risk management framework. Ergonomics. 2010;53(8):927–39.
Salmon PM, Goode N, Taylor N, Lenne MG, Dallat CE, Finch CF. Rasmussen's legacy in the great outdoors: a new incident reporting and learning system for led outdoor activities. Appl Ergon 2017;59(Pt B):637–648.
Service Skills Australia. Envrionmental scan 2015: sport, fitness and recreation. Sydney, Australia: Service Skills Australia; 2015. Available from http://hdl.voced.edu.au/10707/372413. (Accessed 14 July 2019).
Wu JH, Shen WS, Lin LM, Greenes RA, Bates DW. Testing the technology acceptance model for evaluating healthcare professionals' intention to use an adverse event reporting system. Int J Qual Health Care. 2008;20(2):123–9.
This project was supported by Linkage Project funding from the Australia Research Council (ARC) in partnership with Australian Camps Association, Outdoor Council of Australia, The Outdoor Education Group, Sport and Recreation Victoria, Victorian YMCA Accommodation Services Pty Ltd., Outdoors Victoria, Outdoor Recreation Industry Council (Outdoors NSW), Outdoors WA, Outdoors South Australia, Queensland Outdoor Recreation Federation, Wilderness Escape Outdoor Adventures, Venture Corporate Recharge, and Christian Venues Association (LP110100037). PS’s contribution was funded through his current Australian Research Council Future Fellowship (FT140100681). NG’s contribution was funded through the University of the Sunshine Coast. LS was employed through the ARC Linkage Grant for this work.
Ethics approval and consent to participate
The University of the Sunshine Coast Human Ethics Committee approved the study. Consent to participate was indicated by completion of the online survey.
Consent for publication
All authors give their consent for this paper to be published.
The authors declare that they have no competing interests.
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
About this article
Cite this article
Finch, C.F., Goode, N., Shaw, L. et al. End-user experiences with two incident and injury reporting systems designed for led outdoor activities - challenges for implementation of future data systems. Inj. Epidemiol. 6, 39 (2019). https://doi.org/10.1186/s40621-019-0214-y
- Injury surveillance
- Incident reporting
- Data systems