Presentation
MDD30 - Viability of Proxy Users for Usability Testing of Neonatal Devices for Use in Sub-Saharan Africa: A Qualitative Analysis of Use Errors
DescriptionPerforming usability testing with proxy users has been presented as an option for more efficient design and evaluation processes for medical devices designed for use in low resource settings. Proxy users could remove challenges associated with performing usability testing within low-resource settings, such as the burdens of time and financial resources associated with international travel and research.
In our work as human factors researchers, we focus on the benefits of usability testing in identifying medical devices that can be efficiently and effectively operated by medical professionals within a variety of medical facilities. We have conducted usability studies that capture unique perspectives offered by people in different working environments, such as low-resource hospitals, to determine the suitability of a device for use.
An understanding about how proxy user groups perform as substitutes for a target user group in usability testing could benefit broader usability testing and research. During the COVID-19 pandemic, for example, when travel was suspended to geographical regions where we perform usability testing, we considered more deeply the question of using proxy user groups to avoid delaying testing. Identifying a set of users in Houston to form a reasonably accurate and viable proxy user group that is representative of nurses in Malawi could measure the usability of a US-developed device throughout the iterative design cycle without traveling to hospitals in Malawi while remaining confident that usability results found with the Houston proxy user group are an appropriate substitute.
To explore proxy user group use, we examined usability testing we conducted on a variety of neonatal medical technologies with nurses from both the United States and Malawi. Our research design allowed us to compare the findings from both participant pools to assess the feasibility of utilizing a proxy user group based in the US to represent our target users in sub-Saharan Africa for future usability research.
This research was conducted with the Newborn Essential Solutions and Technologies (NEST360) program at the Rice360 Institute for Global Health Technologies. NEST360 supports African governments in their efforts to reduce neonatal mortality through improving neonatal care in African hospitals. These efforts include identifying affordable technologies suitable for use in these low-resource, hospital settings by performing technical testing, environment testing, and usability testing.
Previous research efforts investigated the nature of these two user groups by analyzing and comparing the quantitative data from usability studies. Results showed that the two groups were approximately equivalent for SUS scores, workload scores, and overall task success rates. This cautiously suggested to researchers that the Houston user group was a reasonable proxy for usability testing. However, the small sample size limited the amount of quantitative analysis and interpretation that could be done with this data. Conducting a comparative error analysis between both user groups provides a holistic, qualitative approach to the collected data that helps to answer whether the proxy group is behaving and responding in the same manner as the target users. For example, this approach can reveal if Houston users commit the same types of use errors that Malawi users commit with neonatal devices. Furthermore, one could determine whether they recover from those errors or fail in equal numbers.
Summative usability testing of seven neonatal medical devices was conducted with nurses in both Blantyre, Malawi, and Houston, Texas, USA. These devices belonged to three categories of neonatal devices: bilirubinometers, conductive warmers, and continuous temperature monitors. Expected target users were nurses recruited from the newborn care unit of Queen Elizabeth Central Hospital in Blantyre, Malawi. The proxy user group in the United States was primarily selected to match the Malawian participants' education level and number of years of medical experience.
After viewing brief training videos for each device, participants were asked to complete various tasks with each device to simulate their use in a clinical setting. Any use errors were observed and recorded in addition to the success rate of each task. Each error committed was described and tagged to the device name, task, and user group (Houston/Blantyre) from which it was identified. The frequency of each error was also recorded to ensure proper accounting of all errors but was not analyzed further. For this analysis, any error that occurred at least once was noted as a potential usability issue that should be investigated. It was reasonable, however, to assess the likelihood that users will recover from an error to determine the severity of that error, which could help support and inform priorities during the design/redesign phase. As such, each use error was coded as a recovered or non-recovered error (failure). Ultimately, results from 13 Houstonians and 17 Malawians were included in the analysis.
There were 71 total distinct errors that occurred across both user groups at least once. Seventeen (23.9%) of those errors were only observed in Houston, meaning 54 errors were observed in Blantyre. Of those 54 errors, just 29 (53.7%) occurred in both Houston and Blantyre user groups. Consequently, there were 25 unique errors from Blantyre (46.3%) that did not occur in Houston. These results suggest that if researchers had only collected data from proxy users in Houston, approximately half of the errors that subsequently occurred in Malawi would not have been identified by the proxy group testing.
Further inspection of the common errors shared between both user groups reveals details about the severity of these errors. Of the 29 common errors, 20 (69%) matched the severity between both groups. Additionally, eight of the common errors identified (28%) were recovered errors in Houston but non-recovered errors in Blantyre, while only one (3%) of the common errors was found to be a non-recovered error in Houston and a recovered error in Blantyre. In other words, 27% of the common errors identified tended to be more severe (i.e., did not/could not recover) in Blantyre than in Houston. Only 3% of the common errors identified in Houston were more severe. This discrepancy suggests that recovered errors elicited in Houston were more likely to become non-recovered errors in Blantyre than the other way around.
Ultimately, the findings from this study indicate that the user group in Houston, as currently defined, is an insufficient proxy for testing the usability of neonatal medical devices intended to be used by Malawian nurses. If researchers and device manufacturers were to move forward with this same proxy group for future device testing and discontinue testing with the target users, it is reasonable to assume that they would be missing important data about errors that would likely occur within their target user group. Perhaps most importantly, this research shows that while proxy usability testing can be advantageous, considerable caution should be exercised when selecting proxy users and generalizing their results.
In our work as human factors researchers, we focus on the benefits of usability testing in identifying medical devices that can be efficiently and effectively operated by medical professionals within a variety of medical facilities. We have conducted usability studies that capture unique perspectives offered by people in different working environments, such as low-resource hospitals, to determine the suitability of a device for use.
An understanding about how proxy user groups perform as substitutes for a target user group in usability testing could benefit broader usability testing and research. During the COVID-19 pandemic, for example, when travel was suspended to geographical regions where we perform usability testing, we considered more deeply the question of using proxy user groups to avoid delaying testing. Identifying a set of users in Houston to form a reasonably accurate and viable proxy user group that is representative of nurses in Malawi could measure the usability of a US-developed device throughout the iterative design cycle without traveling to hospitals in Malawi while remaining confident that usability results found with the Houston proxy user group are an appropriate substitute.
To explore proxy user group use, we examined usability testing we conducted on a variety of neonatal medical technologies with nurses from both the United States and Malawi. Our research design allowed us to compare the findings from both participant pools to assess the feasibility of utilizing a proxy user group based in the US to represent our target users in sub-Saharan Africa for future usability research.
This research was conducted with the Newborn Essential Solutions and Technologies (NEST360) program at the Rice360 Institute for Global Health Technologies. NEST360 supports African governments in their efforts to reduce neonatal mortality through improving neonatal care in African hospitals. These efforts include identifying affordable technologies suitable for use in these low-resource, hospital settings by performing technical testing, environment testing, and usability testing.
Previous research efforts investigated the nature of these two user groups by analyzing and comparing the quantitative data from usability studies. Results showed that the two groups were approximately equivalent for SUS scores, workload scores, and overall task success rates. This cautiously suggested to researchers that the Houston user group was a reasonable proxy for usability testing. However, the small sample size limited the amount of quantitative analysis and interpretation that could be done with this data. Conducting a comparative error analysis between both user groups provides a holistic, qualitative approach to the collected data that helps to answer whether the proxy group is behaving and responding in the same manner as the target users. For example, this approach can reveal if Houston users commit the same types of use errors that Malawi users commit with neonatal devices. Furthermore, one could determine whether they recover from those errors or fail in equal numbers.
Summative usability testing of seven neonatal medical devices was conducted with nurses in both Blantyre, Malawi, and Houston, Texas, USA. These devices belonged to three categories of neonatal devices: bilirubinometers, conductive warmers, and continuous temperature monitors. Expected target users were nurses recruited from the newborn care unit of Queen Elizabeth Central Hospital in Blantyre, Malawi. The proxy user group in the United States was primarily selected to match the Malawian participants' education level and number of years of medical experience.
After viewing brief training videos for each device, participants were asked to complete various tasks with each device to simulate their use in a clinical setting. Any use errors were observed and recorded in addition to the success rate of each task. Each error committed was described and tagged to the device name, task, and user group (Houston/Blantyre) from which it was identified. The frequency of each error was also recorded to ensure proper accounting of all errors but was not analyzed further. For this analysis, any error that occurred at least once was noted as a potential usability issue that should be investigated. It was reasonable, however, to assess the likelihood that users will recover from an error to determine the severity of that error, which could help support and inform priorities during the design/redesign phase. As such, each use error was coded as a recovered or non-recovered error (failure). Ultimately, results from 13 Houstonians and 17 Malawians were included in the analysis.
There were 71 total distinct errors that occurred across both user groups at least once. Seventeen (23.9%) of those errors were only observed in Houston, meaning 54 errors were observed in Blantyre. Of those 54 errors, just 29 (53.7%) occurred in both Houston and Blantyre user groups. Consequently, there were 25 unique errors from Blantyre (46.3%) that did not occur in Houston. These results suggest that if researchers had only collected data from proxy users in Houston, approximately half of the errors that subsequently occurred in Malawi would not have been identified by the proxy group testing.
Further inspection of the common errors shared between both user groups reveals details about the severity of these errors. Of the 29 common errors, 20 (69%) matched the severity between both groups. Additionally, eight of the common errors identified (28%) were recovered errors in Houston but non-recovered errors in Blantyre, while only one (3%) of the common errors was found to be a non-recovered error in Houston and a recovered error in Blantyre. In other words, 27% of the common errors identified tended to be more severe (i.e., did not/could not recover) in Blantyre than in Houston. Only 3% of the common errors identified in Houston were more severe. This discrepancy suggests that recovered errors elicited in Houston were more likely to become non-recovered errors in Blantyre than the other way around.
Ultimately, the findings from this study indicate that the user group in Houston, as currently defined, is an insufficient proxy for testing the usability of neonatal medical devices intended to be used by Malawian nurses. If researchers and device manufacturers were to move forward with this same proxy group for future device testing and discontinue testing with the target users, it is reasonable to assume that they would be missing important data about errors that would likely occur within their target user group. Perhaps most importantly, this research shows that while proxy usability testing can be advantageous, considerable caution should be exercised when selecting proxy users and generalizing their results.
Event Type
Poster Presentation
TimeTuesday, March 264:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives