Presentation
Real-Time Monitoring of a Family-Driven Tobacco Cessation Clinical Decision Support System
SessionMethods (DH6)
DescriptionIntroduction
Secondhand smoke (SHS) exposure presents a significant public health risk, particularly for children. Over 40% of the pediatric population aged 3 to 11 years in the United States is exposed to SHS. Household smokers (e.g., mother, father, or grandparents) are the primary source of SHS exposure for children. Delivery of evidence-based tobacco cessation assistance to parents and other caregivers (hereafter referred to as parents) can reduce their smoking rates. Pediatricians are uniquely positioned to deliver tobacco cessation interventions to parents, but barriers exist to consistent and effective delivery. Clinical decision support (CDS) can potentially overcome some of these barriers.
CDS, however, is not without risk. These systems are highly susceptible to malfunction, which can persist undetected for long periods, and, in some cases, the underlying causes of malfunctions are never identified. To ensure the continued safe and effective use of these systems, appropriate monitoring must be implemented. Unfortunately, current monitoring techniques have focused on detecting failures in alerts configured using native electronic health record (EHR) functionality and typically only monitor alert-firing rates. Malfunctions from external CDS systems integrated into the EHR (e.g., CDS Hooks and SMART) present new complexities and failure modes not captured by existing methods. Therefore, it is essential that the capabilities of our monitoring systems are also expanded. As a prototype for this type of approach, our objective was to develop a real-time monitoring system to detect potential failures from a distributed CDS architecture and, importantly, correct any resulting deficiencies in the delivery of care.
Methods
In 2021, our team expanded on our previous work of connecting the parents of our patients to tobacco cessation services. Using pre-visit questionnaires, the new system deployed at Children's Hospital of Philadelphia screens parents for tobacco use, provides motivational messages based on behavioral economic theory (providing nudges to act), and, if requested, automatically connects them to three treatment services: (1) nicotine replacement therapy (NRT); (2) phone-based counseling; and (3) text message-based counseling. Clinicians interact with the system through a web-based application embedded in the EHR, which primarily uses FHIR for retrieving data.
To develop the monitoring system, we utilized a two-staged approach: (1) an assessment of our CDS's potential failure modes; and (2) an evaluation of possible monitoring architectures. To identify potential failure mechanisms, we performed a failure modes and effects analysis (FMEA), which is a proactive method for assessing risk within a system. Each system process identified during the analysis was evaluated on three measures (frequency, severity, and ability to detect) to determine its risk level (criticality within FMEA). Informed by the results of the FMEA, we evaluated multiple monitoring architectures to identify, alert in real-time, and correct system failures. Since our FMEA included factors beyond technology (e.g., users, workflow, policy), it was important to incorporate these additional dimensions in the design of our monitoring architecture. To do this, we used an established sociotechnical model, which considers components beyond technology including, content, human-computer interfaces, people, as well as policies.
Results
Based on a review of our CDS architecture and interviews with developers, EHR analysts, improvement advisors, and external partners, the FMEA identified several potential failure mechanisms. The top five, in order of criticality, were: (1) parent data entry error; (2) network error; (3) FHIR server unavailable; (4) encryption service failure; and (5) clinician data entry error. Most failure modes seriously impacted the CDS system's ability to connect parents to treatment. Given the public health significance related to SHS exposure, most were considered high-severity issues (severity score of 4 or 5). We then used the sociotechnical model to help identify the necessary components of our monitoring infrastructure and their interactions. For example, to rectify potential care deficiencies as a result of a CDS failure, appropriate EHR security (external regulations and internal policy) granting access to specific data in a patient's chart (content) was necessary for certain project team members (personnel).
Our monitoring solution collects information from multiple system components every minute and with each web request. All identified failures (e.g., unresponsive server) are immediately sent to the lead developer via text message and email. Depending on the failure, the developer engages the project manager or clinical champion, who may review the patient's chart, to resolve the issue. To identify potential failures occurring in the systems of our treatment partners, we exchanged weekly counts of connections to compare expected connections per week with actual connections per week.
The CDS monitoring system has been in production since July 2021. Over the study period (19 months), the CDS tool screened 79,926 families for tobacco use and connected 2,382 individuals to 5,098 treatment options. During this time, five types of system failures occurred: (1) network connectivity issue causing failures in 149 visits; (2) FHIR server downtime (total of 33 minutes) causing failures in 144 visits; (3) FHIR web request errors causing failures in 10 visits; (4) parent data entry errors impacting 171 treatment connections; and (5) treatment partner system error impacting 23 treatment connections. No failures occurred as a result of clinician data entry. Our monitoring system identified and resolved 94% (n=465) of all malfunctions. The remaining 6% (n=32) were properly formatted, but incorrect date of birth values entered by parents, which were identified by our NRT partner. In all cases, we successfully connected parents to the appropriate treatment options. Due to real-time notifications, all system-identified failures were acted upon within 5 minutes of occurring and corrected before being reported by end-users. The error in our treatment partner's system was identified and resolved within four days.
Discussion
We developed a real-time CDS monitoring system that can support multiple distributed system architectures (e.g., CDS Hooks and SMART) and data exchange methods (e.g., FHIR). During the project period, our tool successfully identified 94% of system errors, with the other 6% identified by our treatment partner. All errors were resolved before being reported by end-users, most within minutes of occurring. This represents a substantial improvement from existing methods, which often require a one-day delay, some taking months to resolve. Additionally, by using a sociotechnical model in the design of our monitoring system, we were not only able to detect malfunctions, but also correct care deficiencies that may have occurred as a result. Our work uncovered three particularly concerning results. First, even small downtimes of system components (e.g., FHIR server or network connectivity) can have a large-scale impact on CDS operations. The other two relate to the FMEA. Our initial analysis did not include failures from our external partners. Additionally, even though the FMEA identified parent data entry errors as a potential failure point, 32 of these failures went undetected by our monitoring system. These results underscore the challenges of monitoring distributed systems. Our work also highlights the importance of resolving system failures in real time, which is particularly important for interventions grounded in behavioral economics theory. These interventions take advantage of specific, typically brief, windows of time where a participant is appropriately engaged to accept an intervention.
Conclusion
Current monitoring methods are inadequate for distributed web-based CDS architectures. Our monitoring system rapidly identified, alerted in real-time, and resolved system failures before being reported by end users. This was largely due to our sociotechnical approach, which considered several components (e.g., human and technology) and their interactions within and across organizational boundaries. To ensure the ongoing safe and effective use of CDS systems, it is important that continued emphasis is placed not only on expanding CDS delivery models, but also on methods to monitor performance and correct failures. Our approach represents one model for achieving this goal.
Secondhand smoke (SHS) exposure presents a significant public health risk, particularly for children. Over 40% of the pediatric population aged 3 to 11 years in the United States is exposed to SHS. Household smokers (e.g., mother, father, or grandparents) are the primary source of SHS exposure for children. Delivery of evidence-based tobacco cessation assistance to parents and other caregivers (hereafter referred to as parents) can reduce their smoking rates. Pediatricians are uniquely positioned to deliver tobacco cessation interventions to parents, but barriers exist to consistent and effective delivery. Clinical decision support (CDS) can potentially overcome some of these barriers.
CDS, however, is not without risk. These systems are highly susceptible to malfunction, which can persist undetected for long periods, and, in some cases, the underlying causes of malfunctions are never identified. To ensure the continued safe and effective use of these systems, appropriate monitoring must be implemented. Unfortunately, current monitoring techniques have focused on detecting failures in alerts configured using native electronic health record (EHR) functionality and typically only monitor alert-firing rates. Malfunctions from external CDS systems integrated into the EHR (e.g., CDS Hooks and SMART) present new complexities and failure modes not captured by existing methods. Therefore, it is essential that the capabilities of our monitoring systems are also expanded. As a prototype for this type of approach, our objective was to develop a real-time monitoring system to detect potential failures from a distributed CDS architecture and, importantly, correct any resulting deficiencies in the delivery of care.
Methods
In 2021, our team expanded on our previous work of connecting the parents of our patients to tobacco cessation services. Using pre-visit questionnaires, the new system deployed at Children's Hospital of Philadelphia screens parents for tobacco use, provides motivational messages based on behavioral economic theory (providing nudges to act), and, if requested, automatically connects them to three treatment services: (1) nicotine replacement therapy (NRT); (2) phone-based counseling; and (3) text message-based counseling. Clinicians interact with the system through a web-based application embedded in the EHR, which primarily uses FHIR for retrieving data.
To develop the monitoring system, we utilized a two-staged approach: (1) an assessment of our CDS's potential failure modes; and (2) an evaluation of possible monitoring architectures. To identify potential failure mechanisms, we performed a failure modes and effects analysis (FMEA), which is a proactive method for assessing risk within a system. Each system process identified during the analysis was evaluated on three measures (frequency, severity, and ability to detect) to determine its risk level (criticality within FMEA). Informed by the results of the FMEA, we evaluated multiple monitoring architectures to identify, alert in real-time, and correct system failures. Since our FMEA included factors beyond technology (e.g., users, workflow, policy), it was important to incorporate these additional dimensions in the design of our monitoring architecture. To do this, we used an established sociotechnical model, which considers components beyond technology including, content, human-computer interfaces, people, as well as policies.
Results
Based on a review of our CDS architecture and interviews with developers, EHR analysts, improvement advisors, and external partners, the FMEA identified several potential failure mechanisms. The top five, in order of criticality, were: (1) parent data entry error; (2) network error; (3) FHIR server unavailable; (4) encryption service failure; and (5) clinician data entry error. Most failure modes seriously impacted the CDS system's ability to connect parents to treatment. Given the public health significance related to SHS exposure, most were considered high-severity issues (severity score of 4 or 5). We then used the sociotechnical model to help identify the necessary components of our monitoring infrastructure and their interactions. For example, to rectify potential care deficiencies as a result of a CDS failure, appropriate EHR security (external regulations and internal policy) granting access to specific data in a patient's chart (content) was necessary for certain project team members (personnel).
Our monitoring solution collects information from multiple system components every minute and with each web request. All identified failures (e.g., unresponsive server) are immediately sent to the lead developer via text message and email. Depending on the failure, the developer engages the project manager or clinical champion, who may review the patient's chart, to resolve the issue. To identify potential failures occurring in the systems of our treatment partners, we exchanged weekly counts of connections to compare expected connections per week with actual connections per week.
The CDS monitoring system has been in production since July 2021. Over the study period (19 months), the CDS tool screened 79,926 families for tobacco use and connected 2,382 individuals to 5,098 treatment options. During this time, five types of system failures occurred: (1) network connectivity issue causing failures in 149 visits; (2) FHIR server downtime (total of 33 minutes) causing failures in 144 visits; (3) FHIR web request errors causing failures in 10 visits; (4) parent data entry errors impacting 171 treatment connections; and (5) treatment partner system error impacting 23 treatment connections. No failures occurred as a result of clinician data entry. Our monitoring system identified and resolved 94% (n=465) of all malfunctions. The remaining 6% (n=32) were properly formatted, but incorrect date of birth values entered by parents, which were identified by our NRT partner. In all cases, we successfully connected parents to the appropriate treatment options. Due to real-time notifications, all system-identified failures were acted upon within 5 minutes of occurring and corrected before being reported by end-users. The error in our treatment partner's system was identified and resolved within four days.
Discussion
We developed a real-time CDS monitoring system that can support multiple distributed system architectures (e.g., CDS Hooks and SMART) and data exchange methods (e.g., FHIR). During the project period, our tool successfully identified 94% of system errors, with the other 6% identified by our treatment partner. All errors were resolved before being reported by end-users, most within minutes of occurring. This represents a substantial improvement from existing methods, which often require a one-day delay, some taking months to resolve. Additionally, by using a sociotechnical model in the design of our monitoring system, we were not only able to detect malfunctions, but also correct care deficiencies that may have occurred as a result. Our work uncovered three particularly concerning results. First, even small downtimes of system components (e.g., FHIR server or network connectivity) can have a large-scale impact on CDS operations. The other two relate to the FMEA. Our initial analysis did not include failures from our external partners. Additionally, even though the FMEA identified parent data entry errors as a potential failure point, 32 of these failures went undetected by our monitoring system. These results underscore the challenges of monitoring distributed systems. Our work also highlights the importance of resolving system failures in real time, which is particularly important for interventions grounded in behavioral economics theory. These interventions take advantage of specific, typically brief, windows of time where a participant is appropriately engaged to accept an intervention.
Conclusion
Current monitoring methods are inadequate for distributed web-based CDS architectures. Our monitoring system rapidly identified, alerted in real-time, and resolved system failures before being reported by end users. This was largely due to our sociotechnical approach, which considered several components (e.g., human and technology) and their interactions within and across organizational boundaries. To ensure the ongoing safe and effective use of CDS systems, it is important that continued emphasis is placed not only on expanding CDS delivery models, but also on methods to monitor performance and correct failures. Our approach represents one model for achieving this goal.
Event Type
Oral Presentations
TimeTuesday, March 262:30pm - 3:00pm CDT
LocationSalon A-2
Digital Health