Presentation
DH15 - User Centered Design Approach to Improve Clinical Evaluations for Non Accidental Trauma
DescriptionBackground: Non-accidental trauma (NAT) is a leading cause of injury and death during early childhood, and approximately a third of NAT cases involve fractures. At our local hospital system between January 2019 and June 2022, up to 35% of children who returned to the emergency department (ED) with high-risk injuries had incomplete child maltreatment evaluations during a prior encounter. Between January 2022 and September 2022, up to 29% of children who presented to the ED initially with high-risk fractures each month received incomplete child maltreatment evaluations. Significant variability still exists in recognition as well as subsequent evaluation of NAT and high-risk fractures, and NAT cases missed from earlier medical encounters have increased risk for mortality and severe injuries.
Goal: In this study, we seek to improve the recognition of fracture-related NAT (long bone and skull fractures) and the subsequent completion of child maltreatment evaluations through the design and implementation of clinical decision support (CDS) for providers in the ED.
Methodology: To standardize practices for the recognition and evaluation of high-risk fractures, we developed three interruptive alerts for children at high-risk ages who present to the ED with either long bone fractures, skull fractures, or fractures requiring transfer from an outside hospital system. The identification of fractures relied on pattern recognition within radiology reports that evaluated frequently used expressions. To enhance the CDS system, we integrated recommendations and orders for skeletal surveys, social work consultations, and relevant labs and imaging into the alerts by aligning with both existing literature and local consensus among healthcare leadership.
Our approach also incorporated User-Centered Design (UCD) principles and in-situ testing to refine the CDS system. ED providers engaged in two distinct use cases that mirrored missed sentinel fractures identified by the child protection team. One case involved a skull fracture, and the other involved a long bone fracture. Multiple use cases were created, and the cases would either warrant additional NAT evaluations or not. Then the cases would be randomly assigned to ED providers. Providers vocalized their thoughts while navigating the system and completing orders for the simulated patients within a high-fidelity electronic health record environment. After each session, we gathered feedback on the CDS design and iteratively improved the CDS until no further feedback for improvements was provided. During this formative testing, participant feedback was validated at the end of each session and crucial insights were shared with the informatics team. While we opted for in-situ testing without screen capture or audio/video recordings, this choice was deliberate, allowing us to test the system within its operational context effectively. Following rigorous formative testing, we finalized the algorithmic representations for each fracture type and fine-tuned the design of the CDS. This comprehensive approach ensured the robustness and usability of our system, providing a standardized and efficient method for recognizing and evaluating high-risk, fracture-related NAT cases in clinical practice.
Measures: Our primary outcome measure was the proportion of completed NAT evaluations in patients who presented with high-risk fractures. The complete NAT evaluation was also examined and separated into its components. At Children’s Healthcare of Atlanta (CHOA), the child advocacy/child protection and trauma teams screen all potential traumas and highlight potential NAT cases for manual review. Even if a case was ultimately not identified as NAT, the team would nonetheless flag whether all recommended evaluations (e.g. skeletal survey, social work consult, and intraabdominal injury laboratory screen) were performed based on the documented details. We utilized descriptive statistics to compare missed evaluations pre- and post-implementation.
Formative Testing Results: Nine participants (7 Attendings, 2 Residents) participated in the formative testing. In 15 out of the 18 use cases (83%), participants performed the appropriate workups in a simulated setting. One participant did not order a skeletal survey when it was required by the algorithm and mentioned they would discuss the case with the child protection team before placing orders. Several participants also voiced that the algorithm was prescriptive and took away the autonomy and clinical judgments of the physicians. Two participants ordered a skeletal survey when it was not required per the algorithm, as they considered a parietal skull fracture to be high-risk for NAT although the algorithm delineated such fractures to be low-risk for NAT. A few participants also suggested using the phrase “At least one high-risk feature” in the algorithm representation instead of “All low-risk factors” to avoid the language of a double negative. Participants reflected on the alert design and asked for additional acknowledgment reasons including, “No workup indicated”, positive affirmation for completing the workup, order sets for NAT from the alerts, and “A witnessed fall” explanation for cases of skull fractures. Participants also mentioned challenges with obtaining a skeletal survey in the ED and wanted ED leadership and radiology to refine this process.
Implementation and Evaluation Results: The CDS alert system was run in the background to ensure appropriate triggering before being officially implemented on 10/19/22. We monitored the alert data until 10/12/2023. The alert for skull fractures was fired 602 times with at least one new order selected 295 times (49%). The long bone fracture alert was fired 216 times with order placements in 107 cases (49%), and the alert for outside hospital fractures was fired 515 times with 162 orders placed (31%).
Outcome Measures: During the pre-intervention period (1/1/2022 to 10/18/2022), 482 cases with high-risk fractures were identified by the child protection team. Based on chart reviews, 10% of those high-risk cases were identified with incomplete NAT evaluations. The missed evaluation rates were significantly lower (p<0.05) at 5% (29/559) during the post-intervention period (10/19/22- 6/30/23).
Goal: In this study, we seek to improve the recognition of fracture-related NAT (long bone and skull fractures) and the subsequent completion of child maltreatment evaluations through the design and implementation of clinical decision support (CDS) for providers in the ED.
Methodology: To standardize practices for the recognition and evaluation of high-risk fractures, we developed three interruptive alerts for children at high-risk ages who present to the ED with either long bone fractures, skull fractures, or fractures requiring transfer from an outside hospital system. The identification of fractures relied on pattern recognition within radiology reports that evaluated frequently used expressions. To enhance the CDS system, we integrated recommendations and orders for skeletal surveys, social work consultations, and relevant labs and imaging into the alerts by aligning with both existing literature and local consensus among healthcare leadership.
Our approach also incorporated User-Centered Design (UCD) principles and in-situ testing to refine the CDS system. ED providers engaged in two distinct use cases that mirrored missed sentinel fractures identified by the child protection team. One case involved a skull fracture, and the other involved a long bone fracture. Multiple use cases were created, and the cases would either warrant additional NAT evaluations or not. Then the cases would be randomly assigned to ED providers. Providers vocalized their thoughts while navigating the system and completing orders for the simulated patients within a high-fidelity electronic health record environment. After each session, we gathered feedback on the CDS design and iteratively improved the CDS until no further feedback for improvements was provided. During this formative testing, participant feedback was validated at the end of each session and crucial insights were shared with the informatics team. While we opted for in-situ testing without screen capture or audio/video recordings, this choice was deliberate, allowing us to test the system within its operational context effectively. Following rigorous formative testing, we finalized the algorithmic representations for each fracture type and fine-tuned the design of the CDS. This comprehensive approach ensured the robustness and usability of our system, providing a standardized and efficient method for recognizing and evaluating high-risk, fracture-related NAT cases in clinical practice.
Measures: Our primary outcome measure was the proportion of completed NAT evaluations in patients who presented with high-risk fractures. The complete NAT evaluation was also examined and separated into its components. At Children’s Healthcare of Atlanta (CHOA), the child advocacy/child protection and trauma teams screen all potential traumas and highlight potential NAT cases for manual review. Even if a case was ultimately not identified as NAT, the team would nonetheless flag whether all recommended evaluations (e.g. skeletal survey, social work consult, and intraabdominal injury laboratory screen) were performed based on the documented details. We utilized descriptive statistics to compare missed evaluations pre- and post-implementation.
Formative Testing Results: Nine participants (7 Attendings, 2 Residents) participated in the formative testing. In 15 out of the 18 use cases (83%), participants performed the appropriate workups in a simulated setting. One participant did not order a skeletal survey when it was required by the algorithm and mentioned they would discuss the case with the child protection team before placing orders. Several participants also voiced that the algorithm was prescriptive and took away the autonomy and clinical judgments of the physicians. Two participants ordered a skeletal survey when it was not required per the algorithm, as they considered a parietal skull fracture to be high-risk for NAT although the algorithm delineated such fractures to be low-risk for NAT. A few participants also suggested using the phrase “At least one high-risk feature” in the algorithm representation instead of “All low-risk factors” to avoid the language of a double negative. Participants reflected on the alert design and asked for additional acknowledgment reasons including, “No workup indicated”, positive affirmation for completing the workup, order sets for NAT from the alerts, and “A witnessed fall” explanation for cases of skull fractures. Participants also mentioned challenges with obtaining a skeletal survey in the ED and wanted ED leadership and radiology to refine this process.
Implementation and Evaluation Results: The CDS alert system was run in the background to ensure appropriate triggering before being officially implemented on 10/19/22. We monitored the alert data until 10/12/2023. The alert for skull fractures was fired 602 times with at least one new order selected 295 times (49%). The long bone fracture alert was fired 216 times with order placements in 107 cases (49%), and the alert for outside hospital fractures was fired 515 times with 162 orders placed (31%).
Outcome Measures: During the pre-intervention period (1/1/2022 to 10/18/2022), 482 cases with high-risk fractures were identified by the child protection team. Based on chart reviews, 10% of those high-risk cases were identified with incomplete NAT evaluations. The missed evaluation rates were significantly lower (p<0.05) at 5% (29/559) during the post-intervention period (10/19/22- 6/30/23).
Event Type
Poster Presentation
TimeTuesday, March 264:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives