BEGIN:VCALENDAR
VERSION:2.0
PRODID:Linklings LLC
BEGIN:VTIMEZONE
TZID:America/Chicago
X-LIC-LOCATION:America/Chicago
BEGIN:DAYLIGHT
TZOFFSETFROM:-0600
TZOFFSETTO:-0500
TZNAME:CDT
DTSTART:19700308T020000
RRULE:FREQ=YEARLY;BYMONTH=3;BYDAY=2SU
END:DAYLIGHT
BEGIN:STANDARD
TZOFFSETFROM:-0500
TZOFFSETTO:-0600
TZNAME:CST
DTSTART:19701101T020000
RRULE:FREQ=YEARLY;BYMONTH=11;BYDAY=1SU
END:STANDARD
END:VTIMEZONE
BEGIN:VEVENT
DTSTAMP:20240325T185835Z
LOCATION:Salon C
DTSTART;TZID=America/Chicago:20240325T164500
DTEND;TZID=America/Chicago:20240325T181500
UID:HFESHCS_2024 International Symposium on Human Factors and Ergonomics i
 n Health Care_sess111_POST244@linklings.com
SUMMARY:PS4 - AVIAN-S: A Natural Language Processing Model for Analyzing S
 afety Event Reports
DESCRIPTION:Poster Presentation\n\nMichael Sawyer, Katherine Berry, Edward
  Bynum, Jordan Hinson, and Amelia Kinsella (Fort Hill Group)\n\nMost large
  and complex organizations rely on voluntary safety reporting programs (VS
 RPs) to understand risk within their operations. Insights from these first
 -hand accounts can lead to significant safety and efficiency improvements.
  Subject matter experts often read and analyze these reports by labeling f
 actors of interest to derive safety insights. The significant labor resour
 ces and expertise required for this analysis can limit the insights an org
 anization is able to obtain from its VSRP data. Rapid advances in Machine 
 Learning (ML) and natural language processing (NLP) have created the poten
 tial to address these challenges. \nThe AVIAN-S is a novel ML model develo
 ped and trained on over 70,000 rows of manually labeled safety factors acr
 oss 18,000 narrative-based safety reports. This model uses machine learnin
 g and natural language processing (NLP) to automate the task of labeling s
 afety reporting data and codifying report narratives according to a struct
 ured list of human factors topics. The model is built using publicly avail
 able, de-identified safety reports provided through NASA’s Aviation Safety
  Reporting System. While the large training dataset underlying AVIAN-S is 
 based on aviation reports, a significant portion of the trained factors ar
 e generalizable to other domains. \n\nThe language used by safety event re
 porters to describe the impact of factors such as fatigue, checklist usage
 , workload, communication, staffing, time pressure, and expectation bias i
 s likely not domain-specific. These factor labels align with common health
 care models such as the causes and type sections of the Joint Commission o
 n Accreditation of Healthcare Organizations (JCAHO) Patient Safety Event T
 axonomy (e.g., Selection, training, staffing, organizational culture, proc
 edures, rule-based errors). Results to date demonstrate the model’s abilit
 y to identify safety factors with 89% - 97% real-world measured accuracy.\
 nThis presentation will provide an exploratory analysis of the generalizab
 ility of model results, including a comparison of the underlying model tax
 onomy to common healthcare models (e.g., JCAHO Patient Safety Event Taxono
 my), exemplar model factor labeling results, and existing accuracy data ac
 ross 20,000 safety reports.   \n\nAdditionally, lessons learned from devel
 oping and finetuning the AVIAN-S model for domain-specific applications wi
 ll be discussed. This includes adapting a base language model to domain-sp
 ecific language, identifying relevant metrics for assessing model performa
 nce with safety reports and preparing a sufficient training dataset. These
  findings are highly relevant to researchers developing a similar domain-s
 pecific NLP model for healthcare.\n\nTrack: Digital Health, Simulation and
  Education, Hospital Environments, Medical and Drug Delivery Devices, Pati
 ent Safety Research and Initiatives
END:VEVENT
END:VCALENDAR