Presentation
MDD9 - Causes of Test Artifacts in Usability Studies and Best Practices to Avoid Them
DescriptionUsability studies are typically performed outside of the actual use environment, a key factor in the interaction between a user and a medical device.
FDA guidance indicates that test conditions (including the study environment) must be sufficiently realistic to represent actual conditions of use. Thus, replicating the furnishings from a clinical or non-clinical (e.g., home) setting and the use of realistic manikins are key steps that researchers take to aid in the study’s realism. However, there continues to be a considerable portion of observed use events that stem from the simulated environment, i.e., test artifacts. This leads way to the central question of this submission: Why do we continue to see test artifacts from studies performed in “realistic” simulated environments?
These test artifacts are reportable findings that can obscure the conclusion of the study. Additionally, they highlight gaps between the test conditions and actual conditions. A large gap could indicate that the participants are no longer interacting with the device as they would in actual use as they become increasingly influenced by their simulated surroundings.
Through analysis of past usability studies three main groups of artifacts have been determined. These account for over 80% of test artifacts from simulated use studies on injection devices.
• Injection pad related events:
o Accounting for 90% of injection pad events, the two most commonly observed events are participants not cleaning the injection site and not pinching the injection site before needle insertion.
• Timing related events:
o Participants will often pass through steps that require a waiting or hold period, assuming that this is not necessary under the testing conditions.
• Study environment related events:
o The most commonly observed event is participants not noticing the needed material in their environment and omitting a use step.
Understanding how participants perceive the test conditions helps to describe why the above events have emerged as the most common test artifacts. Test conditions function as stimuli to the participants’ senses. They most notably play on their sense of sight, touch, and hearing. Using these senses as rationale we provide a set of best practices to reduce the number of test artifacts that goes past common practice currently seen in usability testing.
Utilizing the scenario introduction is a good way to draw attention to the injection pad and familiarize the participant with the attachment of it. In studies where the pad's was clearly explained vs. not there was a reduction in the amount of test artifacts caused by participants not cleaning/pinching the pad. Additionally, choice of pad is critical and a hyper realistic pad that looks and feels like skin should be chosen. However, it is important for the size of the pad to not exceed 4”x4” as at this point it becomes too bulky to appear realistic to the participants.
To prevent participants from passing through waiting periods it is important to show the passage of time within the study room. A working clock, or timing device allows participants to see and hear an indication of time. Additionally, the scenario introduction can be used to clarify that the participant can take their time and should not feel rushed.
Finally, the use of real sinks and fridges should be used whenever possible, and all study supplies should be made apparent to the participant. They should be familiarized with their surroundings as they would at home (or clinic for HCPs). When study supplies were pointed out prior to the simulated use scenario to the participant there was a considerable increase in participants’ washing their hands and utilizing all appropriate materials.
FDA guidance indicates that test conditions (including the study environment) must be sufficiently realistic to represent actual conditions of use. Thus, replicating the furnishings from a clinical or non-clinical (e.g., home) setting and the use of realistic manikins are key steps that researchers take to aid in the study’s realism. However, there continues to be a considerable portion of observed use events that stem from the simulated environment, i.e., test artifacts. This leads way to the central question of this submission: Why do we continue to see test artifacts from studies performed in “realistic” simulated environments?
These test artifacts are reportable findings that can obscure the conclusion of the study. Additionally, they highlight gaps between the test conditions and actual conditions. A large gap could indicate that the participants are no longer interacting with the device as they would in actual use as they become increasingly influenced by their simulated surroundings.
Through analysis of past usability studies three main groups of artifacts have been determined. These account for over 80% of test artifacts from simulated use studies on injection devices.
• Injection pad related events:
o Accounting for 90% of injection pad events, the two most commonly observed events are participants not cleaning the injection site and not pinching the injection site before needle insertion.
• Timing related events:
o Participants will often pass through steps that require a waiting or hold period, assuming that this is not necessary under the testing conditions.
• Study environment related events:
o The most commonly observed event is participants not noticing the needed material in their environment and omitting a use step.
Understanding how participants perceive the test conditions helps to describe why the above events have emerged as the most common test artifacts. Test conditions function as stimuli to the participants’ senses. They most notably play on their sense of sight, touch, and hearing. Using these senses as rationale we provide a set of best practices to reduce the number of test artifacts that goes past common practice currently seen in usability testing.
Utilizing the scenario introduction is a good way to draw attention to the injection pad and familiarize the participant with the attachment of it. In studies where the pad's was clearly explained vs. not there was a reduction in the amount of test artifacts caused by participants not cleaning/pinching the pad. Additionally, choice of pad is critical and a hyper realistic pad that looks and feels like skin should be chosen. However, it is important for the size of the pad to not exceed 4”x4” as at this point it becomes too bulky to appear realistic to the participants.
To prevent participants from passing through waiting periods it is important to show the passage of time within the study room. A working clock, or timing device allows participants to see and hear an indication of time. Additionally, the scenario introduction can be used to clarify that the participant can take their time and should not feel rushed.
Finally, the use of real sinks and fridges should be used whenever possible, and all study supplies should be made apparent to the participant. They should be familiarized with their surroundings as they would at home (or clinic for HCPs). When study supplies were pointed out prior to the simulated use scenario to the participant there was a considerable increase in participants’ washing their hands and utilizing all appropriate materials.
Event Type
Poster Presentation
TimeTuesday, March 264:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives