Presentation
PS4 - AVIAN-S: A Natural Language Processing Model for Analyzing Safety Event Reports
DescriptionMost large and complex organizations rely on voluntary safety reporting programs (VSRPs) to understand risk within their operations. Insights from these first-hand accounts can lead to significant safety and efficiency improvements. Subject matter experts often read and analyze these reports by labeling factors of interest to derive safety insights. The significant labor resources and expertise required for this analysis can limit the insights an organization is able to obtain from its VSRP data. Rapid advances in Machine Learning (ML) and natural language processing (NLP) have created the potential to address these challenges.
The AVIAN-S is a novel ML model developed and trained on over 70,000 rows of manually labeled safety factors across 18,000 narrative-based safety reports. This model uses machine learning and natural language processing (NLP) to automate the task of labeling safety reporting data and codifying report narratives according to a structured list of human factors topics. The model is built using publicly available, de-identified safety reports provided through NASA’s Aviation Safety Reporting System. While the large training dataset underlying AVIAN-S is based on aviation reports, a significant portion of the trained factors are generalizable to other domains.
The language used by safety event reporters to describe the impact of factors such as fatigue, checklist usage, workload, communication, staffing, time pressure, and expectation bias is likely not domain-specific. These factor labels align with common healthcare models such as the causes and type sections of the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Patient Safety Event Taxonomy (e.g., Selection, training, staffing, organizational culture, procedures, rule-based errors). Results to date demonstrate the model’s ability to identify safety factors with 89% - 97% real-world measured accuracy.
This presentation will provide an exploratory analysis of the generalizability of model results, including a comparison of the underlying model taxonomy to common healthcare models (e.g., JCAHO Patient Safety Event Taxonomy), exemplar model factor labeling results, and existing accuracy data across 20,000 safety reports.
Additionally, lessons learned from developing and finetuning the AVIAN-S model for domain-specific applications will be discussed. This includes adapting a base language model to domain-specific language, identifying relevant metrics for assessing model performance with safety reports and preparing a sufficient training dataset. These findings are highly relevant to researchers developing a similar domain-specific NLP model for healthcare.
The AVIAN-S is a novel ML model developed and trained on over 70,000 rows of manually labeled safety factors across 18,000 narrative-based safety reports. This model uses machine learning and natural language processing (NLP) to automate the task of labeling safety reporting data and codifying report narratives according to a structured list of human factors topics. The model is built using publicly available, de-identified safety reports provided through NASA’s Aviation Safety Reporting System. While the large training dataset underlying AVIAN-S is based on aviation reports, a significant portion of the trained factors are generalizable to other domains.
The language used by safety event reporters to describe the impact of factors such as fatigue, checklist usage, workload, communication, staffing, time pressure, and expectation bias is likely not domain-specific. These factor labels align with common healthcare models such as the causes and type sections of the Joint Commission on Accreditation of Healthcare Organizations (JCAHO) Patient Safety Event Taxonomy (e.g., Selection, training, staffing, organizational culture, procedures, rule-based errors). Results to date demonstrate the model’s ability to identify safety factors with 89% - 97% real-world measured accuracy.
This presentation will provide an exploratory analysis of the generalizability of model results, including a comparison of the underlying model taxonomy to common healthcare models (e.g., JCAHO Patient Safety Event Taxonomy), exemplar model factor labeling results, and existing accuracy data across 20,000 safety reports.
Additionally, lessons learned from developing and finetuning the AVIAN-S model for domain-specific applications will be discussed. This includes adapting a base language model to domain-specific language, identifying relevant metrics for assessing model performance with safety reports and preparing a sufficient training dataset. These findings are highly relevant to researchers developing a similar domain-specific NLP model for healthcare.
Event Type
Poster Presentation
TimeMonday, March 254:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives