Presentation
Early Investments for Teaming Dividends: A Human-Centered Approach to a Patient Decompensation Prediction Algorithm
SessionDevelopment Approaches (DH5)
DescriptionDespite the excitement surrounding artificial intelligence (AI) and machine learning (ML) in healthcare (Rajpurkar et al., 2022), responsible utilization of AI/ML technologies in high-stakes domains like healthcare requires more than simply high-performing algorithms, but also mechanisms for effectively interacting with domain practitioners, like algorithm explainability (Samek et al., 2017). However, guidance on how to design and develop algorithms which support effective human-machine interactions remains an acknowledged gap in the research on human-machine teaming (NASEM, 2021). Several of our design strategies empirically tested with healthcare professionals (Morey, 2021, 2023; Rayo et al., 2022) have shown promising potential to support effective human-machine interactions, like the ability of people to accurately assess when algorithms are misaligned or misfit for the situation (Rayo et al., 2020). Nevertheless, these publications to date have focused predominantly on the design of the data visualizations and resultant human-machine interactions after an algorithm has been created, with little detail about the development of the algorithms themselves which enabled these interactions. The purpose of this paper is to describe our approach to the design and development of AI/ML algorithms in high-stakes domains, using two patient decompensation algorithms as examples (Morey, 2021, 2023). We believe that an investment in the potential for teaming early in the algorithm development process, not just the pursuit of high-performing algorithms, will pay dividends for the eventual effectiveness human-machine team.
This paper will outline our approach to algorithm development, including (1) our process for developing two high-performing algorithms for predicting patient decompensation events five minutes into the future (Morey, 2021, 2023) and (2) our early strategic investments in algorithm development that later paid dividends for human-machine teaming. To address the former, we will include the types and granularity of data we obtained, the features we computed prior to training the model, the decision criteria we utilized for training and pruning the model features, the significant features that were retained in the models, and the performance of the models after cross-validation. To address the later, we will discuss two strategies we employed in developing these algorithms which greatly aided our ability to later design complex human-machine interactions. First, we designed the algorithms to predict patient decompensation events five minutes into the future, which is further time horizon than traditional clinical alarms. Although predicting further into the future is inherently more uncertain and inevitably limits the ceiling of algorithm performance, we believe that future-oriented algorithms are better matched to the tempo of work and time pressure that clinicians face, making these kinds of algorithms better suited for how clinicians interact with patients. Second, we deliberately designed algorithm features that would be interpretable by people and even designed some algorithm features that explicitly aligned with how clinicians themselves anticipate patient decompensation (Horwood, Moffatt-Bruce, et al., 2018; Horwood, Rayo, et al., 2018). We were later able to exploit the interpretability of these features to create custom visualizations for each specific feature which closely aligned with the computations of the algorithm itself (Morey, 2021, 2023).
From a strictly algorithmic viewpoint, these decisions might appear counterproductive. It is likely possible that we could have achieved higher algorithm performance had we not restricted our algorithm development in such a way. However, knowing that these algorithms would not and cannot be deployed autonomously in healthcare (i.e., without human input, oversight, interaction, etc.), we have seen growing evidence that these decisions greatly augmented the capabilities of the algorithm-clinician team, even if they hindered the algorithms themselves, and were ultimately beneficial to the joint human-machine team for better patient outcomes. We believe that taking this human-machine teaming perspective early in the algorithm development process is the key to unlocking the potential of powerful AI/ML technologies that contribute to human decision-making so that the joint human-machine team can perform better than either people or algorithms could perform alone (Woods, 1985).
References:
Horwood, C. R., Moffatt-Bruce, S. D., Fitzgerald, M., & Rayo, M. F. (2018). A qualitative analysis of clinical decompensation in the surgical patient: Perceptions of nurses and physicians. Surgery, 164(6), 1311–1315. https://doi.org/10.1016/j.surg.2018.06.006
Horwood, C. R., Rayo, M. F., Fitzgerald, M., Balkin, E. A., & Moffatt-Bruce, S. D. (2018). Gaps Between Alarm Capabilities and Decision-making Needs: An Observational Study of Detecting Patient Decompensation. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 7, 112–116. https://doi.org/10.1177/2327857918071028
Morey, D. A. (2021). Breaking away from brittle machines: Evaluating simultaneous inference and data (SID) displays to facilitate machine fitness assessment. Master’s Thesis, The Ohio State University.
Morey, D. A. (2023). Jointness Still Matters: Adding AI Without Designing for Joint Activity Likely Degrades Performance [Dissertation, The Ohio State University]. http://rave.ohiolink.edu/etdc/view?acc_num=osu1689881674186536
NASEM. (2021). Human-AI Teaming: State-of-the-Art and Research Needs. National Academies of Sciences, Engineering, and Medicine. https://doi.org/10.17226/26355
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28(1), Article 1. https://doi.org/10.1038/s41591-021-01614-0
Rayo, M. F., Fitzgerald, M. C., Gifford, R. C., Morey, D. A., Reynolds, M. E., D’Annolfo, K., & Jefferies, C. M. (2020). The Need for Machine Fitness Assessment: Enabling Joint Human-Machine Performance in Consumer Health Technologies. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 9, 40–42. https://doi.org/10.1177/2327857920091041
Rayo, M. F., Horwood, C. R., Fitzgerald, M. C., Grayson, M. R., Abdel-Rasoul, M., & Moffatt-Bruce, S. D. (2022). Situated Visual Alarm Displays Support Machine Fitness Assessment for Nonexplainable Automation. IEEE Transactions on Human-Machine Systems, 52(5), 984–993. https://doi.org/10.1109/THMS.2022.3155714
Samek, W., Wiegand, T., & Müller, K.-R. (2017). Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (arXiv:1708.08296). arXiv. https://doi.org/10.48550/arXiv.1708.08296
Woods, D. D. (1985). Cognitive Technologies: The Design of Joint Human-Machine Cognitive Systems. AI Magazine, 6(4), 1–7. https://doi.org/10.1609/aimag.v6i4.511
This paper will outline our approach to algorithm development, including (1) our process for developing two high-performing algorithms for predicting patient decompensation events five minutes into the future (Morey, 2021, 2023) and (2) our early strategic investments in algorithm development that later paid dividends for human-machine teaming. To address the former, we will include the types and granularity of data we obtained, the features we computed prior to training the model, the decision criteria we utilized for training and pruning the model features, the significant features that were retained in the models, and the performance of the models after cross-validation. To address the later, we will discuss two strategies we employed in developing these algorithms which greatly aided our ability to later design complex human-machine interactions. First, we designed the algorithms to predict patient decompensation events five minutes into the future, which is further time horizon than traditional clinical alarms. Although predicting further into the future is inherently more uncertain and inevitably limits the ceiling of algorithm performance, we believe that future-oriented algorithms are better matched to the tempo of work and time pressure that clinicians face, making these kinds of algorithms better suited for how clinicians interact with patients. Second, we deliberately designed algorithm features that would be interpretable by people and even designed some algorithm features that explicitly aligned with how clinicians themselves anticipate patient decompensation (Horwood, Moffatt-Bruce, et al., 2018; Horwood, Rayo, et al., 2018). We were later able to exploit the interpretability of these features to create custom visualizations for each specific feature which closely aligned with the computations of the algorithm itself (Morey, 2021, 2023).
From a strictly algorithmic viewpoint, these decisions might appear counterproductive. It is likely possible that we could have achieved higher algorithm performance had we not restricted our algorithm development in such a way. However, knowing that these algorithms would not and cannot be deployed autonomously in healthcare (i.e., without human input, oversight, interaction, etc.), we have seen growing evidence that these decisions greatly augmented the capabilities of the algorithm-clinician team, even if they hindered the algorithms themselves, and were ultimately beneficial to the joint human-machine team for better patient outcomes. We believe that taking this human-machine teaming perspective early in the algorithm development process is the key to unlocking the potential of powerful AI/ML technologies that contribute to human decision-making so that the joint human-machine team can perform better than either people or algorithms could perform alone (Woods, 1985).
References:
Horwood, C. R., Moffatt-Bruce, S. D., Fitzgerald, M., & Rayo, M. F. (2018). A qualitative analysis of clinical decompensation in the surgical patient: Perceptions of nurses and physicians. Surgery, 164(6), 1311–1315. https://doi.org/10.1016/j.surg.2018.06.006
Horwood, C. R., Rayo, M. F., Fitzgerald, M., Balkin, E. A., & Moffatt-Bruce, S. D. (2018). Gaps Between Alarm Capabilities and Decision-making Needs: An Observational Study of Detecting Patient Decompensation. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 7, 112–116. https://doi.org/10.1177/2327857918071028
Morey, D. A. (2021). Breaking away from brittle machines: Evaluating simultaneous inference and data (SID) displays to facilitate machine fitness assessment. Master’s Thesis, The Ohio State University.
Morey, D. A. (2023). Jointness Still Matters: Adding AI Without Designing for Joint Activity Likely Degrades Performance [Dissertation, The Ohio State University]. http://rave.ohiolink.edu/etdc/view?acc_num=osu1689881674186536
NASEM. (2021). Human-AI Teaming: State-of-the-Art and Research Needs. National Academies of Sciences, Engineering, and Medicine. https://doi.org/10.17226/26355
Rajpurkar, P., Chen, E., Banerjee, O., & Topol, E. J. (2022). AI in health and medicine. Nature Medicine, 28(1), Article 1. https://doi.org/10.1038/s41591-021-01614-0
Rayo, M. F., Fitzgerald, M. C., Gifford, R. C., Morey, D. A., Reynolds, M. E., D’Annolfo, K., & Jefferies, C. M. (2020). The Need for Machine Fitness Assessment: Enabling Joint Human-Machine Performance in Consumer Health Technologies. Proceedings of the International Symposium on Human Factors and Ergonomics in Health Care, 9, 40–42. https://doi.org/10.1177/2327857920091041
Rayo, M. F., Horwood, C. R., Fitzgerald, M. C., Grayson, M. R., Abdel-Rasoul, M., & Moffatt-Bruce, S. D. (2022). Situated Visual Alarm Displays Support Machine Fitness Assessment for Nonexplainable Automation. IEEE Transactions on Human-Machine Systems, 52(5), 984–993. https://doi.org/10.1109/THMS.2022.3155714
Samek, W., Wiegand, T., & Müller, K.-R. (2017). Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models (arXiv:1708.08296). arXiv. https://doi.org/10.48550/arXiv.1708.08296
Woods, D. D. (1985). Cognitive Technologies: The Design of Joint Human-Machine Cognitive Systems. AI Magazine, 6(4), 1–7. https://doi.org/10.1609/aimag.v6i4.511
Event Type
Oral Presentations
TimeTuesday, March 2611:00am - 11:30am CDT
LocationSalon A-2
Digital Health