Presentation
Transforming Healthcare through Integrating Generative Large Language Models and Resilience Engineering Tools: Applications in Medical Education
SessionDevelopment Approaches (DH5)
DescriptionSummary:
This research explores the transformative potential of generative large language models (LLMs) and Resilience Engineering Tools in healthcare to improve medical education. By harnessing the power of LLMs like LLaMA and Alpaca, renowned for their proficiency in natural language understanding and generation, as well as utilizing the Resilience Engineering Tool to Improve Patient Safety (RETIPS) to collect real-world narratives of adaptation in everyday work from frontline caregivers, this research offers innovative prospects for healthcare education (Hegde et al., 2020).
Motivation and Background:
In medical education, the conventional focus primarily revolves around imparting standard procedures, best practices, and episodic learning via approaches such as case studies presented during morbidity & mortality (M&M) conferences. While these are invaluable for conveying fundamental concepts in specific challenging situations, there is a significant gap in the training curriculum. This gap pertains to the absence of exposure to how practitioners navigate the complexities of everyday situations that are considered 'normal' or routine. This encompasses coping with efficiency pressures, maneuvering constraints, navigating resource shortages, and effectively cooperating and coordinating with peers. Addressing this educational gap would serve as a valuable addition to the training curriculum, complementing the current emphasis on standard procedures and episodic learning.
Furthermore, the importance of bridging this educational gap becomes acutely evident during large-scale disruptive events, such as the COVID-19 pandemic. Under such circumstances, the lack of established best practices and evolved heuristics poses a significant challenge to training and preparedness. There’s an urgent need to quickly learn from frontline healthcare workers and disseminate lessons, including innovative solutions, throughout the workforce.
In response to these needs, RETIPS was implemented in the radiology department of a major children’s hospital. This tool allowed frontline healthcare staff to share narratives of their everyday challenges and successful adaptations during the early stages of the COVID-19 pandemic. This initiative resulted in the generation of fifty-eight reports that offered critical insights into adaptive patterns and actionable strategies. However, the integration of these reports into medical education, particularly for quality improvement training for residents, has been limited by the relatively small size of the available dataset.
Beyond these pedagogical challenges, the potential of LLMs in healthcare remains largely untapped due to concerns related to cost, healthcare data privacy, and complex security regulations. This discourages the extensive utilization of third-party services like OpenAI’s ChatGPT. In the context of text classification, there is a growing need for accurate and efficient categorization, especially when working with limited data. In this regard, LLMs offer valuable assistance by generating synthetic responses to real survey questions and enhancing the categorization process. Moreover, medical education increasingly demands engaging and tailored learning materials. The extraction of insights from the narratives of frontline healthcare workers using LLMs presents a promising opportunity to enrich educational content. Ultimately, our focus is to support medical education with more of the 'everyday' adaptive patterns content, by applying LLMs to RETIPS responses.
Approach:
Our methodology fits within an envisioned framework. It involves collecting data through RETIPS and similar self-reported narratives from frontline staff. Using generative LLMs with data augmentation, we created a corpus of scenarios that reflect characteristics of everyday adaptive work similar to those of the real narratives. The iterative process of generating realistic and representative synthetic responses, however, required significant human involvement in terms of fine-tuning and revision. The presentation will expand on the human-AI interaction loops at the AI-developmental stage, with instances of challenges overcome and improvements made. These include curbing the LLM's 'hallucinations', removing inconsistencies in the output format, and improving representativeness. Insights from our experience will be offered, that could be helpful in guiding educators, AI developers, and other stakeholders, in developing LLMs to enhance data quality and representativeness. These scenarios find applications in various aspects of medical education, like quality and safety analysis and personalized study content creation.
Importance of this work and takeaway points:
The list of key aspects in medical education that this research addresses:
• Realistic Narratives: Integrating LLMs with RETIPS enhances the development and delivery of medical education by augmenting standard procedures with real-world narratives.
• Everyday Situation Training: Bridging the educational gap by exposing students to everyday healthcare work complexities, including dealing with efficiency pressures, maneuvering constraints, and resource shortages.
• Adaptive Learning: Offering adaptable learning materials that encourage adaptive thinking and action among healthcare professionals.
• Large-Scale Disruption Response: Learning from disruptive events, such as the COVID-19 pandemic, by quickly extracting actionable insights from frontline healthcare workers' narratives and disseminating them throughout the healthcare workforce.
• Quality Improvement Training: Enhancing quality improvement training for residents by disseminating effective strategies evolved locally in everyday work that have applicability in similar settings or more expansive operational areas.
• Cost and Data Privacy Challenges: The case for use of LLMs in healthcare to address cost and data privacy concerns.
• Accurate Text Classification: Improving text classification accuracy by utilizing LLMs for generating synthetic responses to real survey questions.
• Data-Driven Learning Materials: Developing innovative, data-driven learning materials that are tailored, engaging, and based on real-world experiences, enriching medical education.
Limitations and future work:
One primary limitation is the small size of the RETIPS dataset, comprising 58 responses. This may impact the study's reliability and generalizability, particularly when compared to cases with larger datasets. However, the small size facilitates focused exploration of the data augmentation problem, which is crucial for its value in small datasets.
Another limitation is the exclusivity of data collection to RETIPS surveys administered to radiology staff at a single hospital. While necessary for research objectives, the models' performance may vary in other healthcare domains and outside medicine.
To ensure broad accessibility, we focus on language models with at most 30 billion parameters, excluding larger models like BLOOM (176 billion parameters). While this choice facilitates accessibility, it may limit performance enhancement. Future studies could explore larger datasets for richer training data and experiment with different LLMs for potential improvements.
Considering the rapid evolution of text-generation LLMs, future research should experiment with newer models, incorporating architectural and training advancements. Hyperparameter tuning of language models and classifiers presents a potential area for further investigation to enhance performance.
Despite these limitations, working with small data in a privacy-conscious environment using accessible AI tools aligns with the challenges many face in healthcare research. We aim to provide valuable insights into employing AI tools in such settings.
Future work will involve evaluation of the synthetic reports by hospital stakeholders, including educators, residents and trainees, and quality and safety personnel.
Reference
Hegde, S., Hettinger, A. Z., Fairbanks, R. J., Wreathall, J., Krevat, S. A., Jackson, C. D., & Bisantz, A. M. (2020). Qualitative findings from a pilot stage implementation of a novel organizational learning tool toward operationalizing the Safety-II paradigm in health care. Applied ergonomics, 82, 102913.
This research explores the transformative potential of generative large language models (LLMs) and Resilience Engineering Tools in healthcare to improve medical education. By harnessing the power of LLMs like LLaMA and Alpaca, renowned for their proficiency in natural language understanding and generation, as well as utilizing the Resilience Engineering Tool to Improve Patient Safety (RETIPS) to collect real-world narratives of adaptation in everyday work from frontline caregivers, this research offers innovative prospects for healthcare education (Hegde et al., 2020).
Motivation and Background:
In medical education, the conventional focus primarily revolves around imparting standard procedures, best practices, and episodic learning via approaches such as case studies presented during morbidity & mortality (M&M) conferences. While these are invaluable for conveying fundamental concepts in specific challenging situations, there is a significant gap in the training curriculum. This gap pertains to the absence of exposure to how practitioners navigate the complexities of everyday situations that are considered 'normal' or routine. This encompasses coping with efficiency pressures, maneuvering constraints, navigating resource shortages, and effectively cooperating and coordinating with peers. Addressing this educational gap would serve as a valuable addition to the training curriculum, complementing the current emphasis on standard procedures and episodic learning.
Furthermore, the importance of bridging this educational gap becomes acutely evident during large-scale disruptive events, such as the COVID-19 pandemic. Under such circumstances, the lack of established best practices and evolved heuristics poses a significant challenge to training and preparedness. There’s an urgent need to quickly learn from frontline healthcare workers and disseminate lessons, including innovative solutions, throughout the workforce.
In response to these needs, RETIPS was implemented in the radiology department of a major children’s hospital. This tool allowed frontline healthcare staff to share narratives of their everyday challenges and successful adaptations during the early stages of the COVID-19 pandemic. This initiative resulted in the generation of fifty-eight reports that offered critical insights into adaptive patterns and actionable strategies. However, the integration of these reports into medical education, particularly for quality improvement training for residents, has been limited by the relatively small size of the available dataset.
Beyond these pedagogical challenges, the potential of LLMs in healthcare remains largely untapped due to concerns related to cost, healthcare data privacy, and complex security regulations. This discourages the extensive utilization of third-party services like OpenAI’s ChatGPT. In the context of text classification, there is a growing need for accurate and efficient categorization, especially when working with limited data. In this regard, LLMs offer valuable assistance by generating synthetic responses to real survey questions and enhancing the categorization process. Moreover, medical education increasingly demands engaging and tailored learning materials. The extraction of insights from the narratives of frontline healthcare workers using LLMs presents a promising opportunity to enrich educational content. Ultimately, our focus is to support medical education with more of the 'everyday' adaptive patterns content, by applying LLMs to RETIPS responses.
Approach:
Our methodology fits within an envisioned framework. It involves collecting data through RETIPS and similar self-reported narratives from frontline staff. Using generative LLMs with data augmentation, we created a corpus of scenarios that reflect characteristics of everyday adaptive work similar to those of the real narratives. The iterative process of generating realistic and representative synthetic responses, however, required significant human involvement in terms of fine-tuning and revision. The presentation will expand on the human-AI interaction loops at the AI-developmental stage, with instances of challenges overcome and improvements made. These include curbing the LLM's 'hallucinations', removing inconsistencies in the output format, and improving representativeness. Insights from our experience will be offered, that could be helpful in guiding educators, AI developers, and other stakeholders, in developing LLMs to enhance data quality and representativeness. These scenarios find applications in various aspects of medical education, like quality and safety analysis and personalized study content creation.
Importance of this work and takeaway points:
The list of key aspects in medical education that this research addresses:
• Realistic Narratives: Integrating LLMs with RETIPS enhances the development and delivery of medical education by augmenting standard procedures with real-world narratives.
• Everyday Situation Training: Bridging the educational gap by exposing students to everyday healthcare work complexities, including dealing with efficiency pressures, maneuvering constraints, and resource shortages.
• Adaptive Learning: Offering adaptable learning materials that encourage adaptive thinking and action among healthcare professionals.
• Large-Scale Disruption Response: Learning from disruptive events, such as the COVID-19 pandemic, by quickly extracting actionable insights from frontline healthcare workers' narratives and disseminating them throughout the healthcare workforce.
• Quality Improvement Training: Enhancing quality improvement training for residents by disseminating effective strategies evolved locally in everyday work that have applicability in similar settings or more expansive operational areas.
• Cost and Data Privacy Challenges: The case for use of LLMs in healthcare to address cost and data privacy concerns.
• Accurate Text Classification: Improving text classification accuracy by utilizing LLMs for generating synthetic responses to real survey questions.
• Data-Driven Learning Materials: Developing innovative, data-driven learning materials that are tailored, engaging, and based on real-world experiences, enriching medical education.
Limitations and future work:
One primary limitation is the small size of the RETIPS dataset, comprising 58 responses. This may impact the study's reliability and generalizability, particularly when compared to cases with larger datasets. However, the small size facilitates focused exploration of the data augmentation problem, which is crucial for its value in small datasets.
Another limitation is the exclusivity of data collection to RETIPS surveys administered to radiology staff at a single hospital. While necessary for research objectives, the models' performance may vary in other healthcare domains and outside medicine.
To ensure broad accessibility, we focus on language models with at most 30 billion parameters, excluding larger models like BLOOM (176 billion parameters). While this choice facilitates accessibility, it may limit performance enhancement. Future studies could explore larger datasets for richer training data and experiment with different LLMs for potential improvements.
Considering the rapid evolution of text-generation LLMs, future research should experiment with newer models, incorporating architectural and training advancements. Hyperparameter tuning of language models and classifiers presents a potential area for further investigation to enhance performance.
Despite these limitations, working with small data in a privacy-conscious environment using accessible AI tools aligns with the challenges many face in healthcare research. We aim to provide valuable insights into employing AI tools in such settings.
Future work will involve evaluation of the synthetic reports by hospital stakeholders, including educators, residents and trainees, and quality and safety personnel.
Reference
Hegde, S., Hettinger, A. Z., Fairbanks, R. J., Wreathall, J., Krevat, S. A., Jackson, C. D., & Bisantz, A. M. (2020). Qualitative findings from a pilot stage implementation of a novel organizational learning tool toward operationalizing the Safety-II paradigm in health care. Applied ergonomics, 82, 102913.
Event Type
Oral Presentations
TimeTuesday, March 2611:30am - 12:00pm CDT
LocationSalon A-2
Digital Health