Presentation
MDD6 - Applying Memory Theories to Determine Participant Injection Experience in Human Factors Usability Testing
DescriptionThis proposal discusses the current ways of determining the experience level of participants in human factors usability studies in the medical field regarding injection products. The discussion is led by research into memory retention and decay and compares these findings with current guidelines placed by the FDA. This then introduces a decision support tool that can establish a standard and rational based way of determining experience in participants.
The importance of determining what makes a participant naïve or experienced shows up in two main ways within human factors studies. The most obvious is accurately designating participants, lest the data incorrectly reflect how experienced and naïve participants perform using the device. Less obvious is the difficulty of recruiting naïve participants, especially with small populations such as rare conditions. This can lead to the use of surrogates reducing the external validity of the data.
Current FDA Guidance (2016) states that "the most important consideration for test participants in human factors validation testing is that they represent the population of intended users." On the surface, this guidance seems intuitive; however, there are questions that need to be addressed when research requires user groups based on experience. Experience for participants can range from practicing injecting once to doing high frequency injections for years. These levels of experience are vastly different but can be sorted into the same category of experience as injecting once does technically make a person injection experienced. By considering participants with minimal experience as experienced users the potential population of naïve users drastically decreases while the experienced user data falsely represents how experience affects use. In fact, research would suggest that following a one-time use, the participant could be considered naïve within three months.
The Ebbinghaus forgetting curve is a model depicting the rate at which people forget learned information when there is no attempt to retain it. The study producing the model was conducted in 1880, but has been replicated several times establishing its relevance and accuracy in modern studies. It has been represented several ways but can be most simply graphed using an exponential decay equation modified by the stability of the memory (Woźniak, Gorzelańczyk, & Murakowski, 1994). The curve suggests that if a person conducts a serial position task (e.g., memorizing a list in order), after one day without rehearsal the average savings were 27% (Murre, Dros, 2015). Savings are the percentage of time saved calculated by dividing the time it took to relearn a list by the amount of time it took to learn the list originally. Simply, the ability to recall knowledge that isn’t reinforced steeply drops and settles towards negligible for regular memory.
’Flashbulb memories’ are argued to highly alter the rate of forgetting though. The term has been used since 1977 to describe memories that do not fade as easily as regular memories (Winograd & Neisser, 2006). The true nature of these memories is unclear, even to the point of questioning if they exist at all, but some researchers argue that emotion reduces the rate at which memory decays (Reisberg, Heuer, 2006) and flashbulb memories are often connected to major emotional events. Flashbulbs are significantly more vivid which can inspire certainty, but that doesn’t mean that the memory is preserved correctly. Instead, it has been observed that comparing recollections taken directly after an event and two years later, there is often a stark difference in the memories with the later recollection incorporating traceable biases. The later account will likely match future recollections of the same event though, showing its resistance to decay (Neisser, Harsch, 2006).
In the context of Human Factors research, if a participant’s injection experience occurs during a high emotion event it is more likely to create a flashbulb memory. These memories may make the participant feel experienced, but their accuracy remembering the event, and thus their actual knowledge regarding the device, can still be deemed negligible within the year. Should it not be a flashbulb memory, it will decay according to the Ebbinghaus forgetting curve and be considered negligible within the same time frame.
The best way for a memory to be preserved accurately remains in reviewing the information. If a person reviews information with certain frequency, it not only increases their ability to recall it but also slows the decay of the knowledge afterward (Chun & Heo, 2018). Each repetition flattens the Ebbinghaus forgetting curve more but never eliminates memory decay. One-time executions of tasks will fade rapidly, but if a participant had to inject once a week for a year their memory of that experience will be more accurate for longer until eventually becoming negligible as well.
Taking this research into account, a decision support tool has been created to make a more standard and rational based way of determining whether participants should be considered experienced or naïve in injection studies. The tool accounts for what experiences mean about the participant’s knowledge and functions through a series of if-then statements to decisively sort experience. As an example, consider an individual that participated in an injection study and is being considered for another one. The first criterion would be the time between studies. If it was less than a year, they may not participate as a naïve participant. If it was greater than a year, the tool indicates they can retain their experience qualification from the last study. However, if a naïve participant began performing injections in the past 12 months, they will now qualify as experienced.
With this type of standard tool to determine experience groups, Human Factors studies can reduce the problems caused by the current over inclusive experience designation. Ensuring participants are accurately represented as naïve or experienced will guarantee more accurate data about how experience affects usability. This is especially powerful when it helps avoid using surrogates in studies by increasing the percentage of the population deemed naïve. Though the decision support tool is not perfect, it provides a representation of experience that does not currently exist in our field.
To present these concepts the full decision support tool will be shown on the presentation/poster along with the applicable forms of the Ebbinghaus forgetting curve. Explanatory information for each curve will give context about how they apply to the decision support tool and relate to the original curve. Several examples of possible participants will be displayed to demonstrate the function of the tool as well as display a small array of the varied experiences people demonstrate. Finally, the presentation/poster will discuss and display the possible implications and benefits of investigating how experience is determined for human factors usability testing of medical devices.
Chun, B. A., & Heo, H. J. (n.d.). ICIET ’18. In The 6th International Conference on Information and Education Technology.
Harsch, N., & Neisser, U. (2006). Phantom flashbulbs: False recollections of hearing the news about Challenger. In Affect and accuracy in recall: Studies of “Flashbulb” memories. essay, Cambridge University Press.
Murre, J. M. J., & Dros, J. (2015). Replication and Analysis of Ebbinghaus’ forgetting curve. PLOS ONE, 10(7).
Reisberg, D., & Heuer, F. (2006). Remembering the details of emotional events. In Affect and accuracy in recall: Studies of “Flashbulb” memories. essay, Cambridge University Press.
Winograd, E., & Neisser, U. (2006a). Affect and accuracy in recall: Studies of “Flashbulb” memories. Cambridge University Press.
Winograd, E., & Neisser, U. (2006b). Preface. In Affect and accuracy in recall: studies of “flashbulb” memories. preface, Cambridge University Press.
Woźniak, P. A., Gorzelańczyk, E. J., & Murakowski, J. A. (1995). Two components of long-term memory. Acta Neurobiologiae Experimentalis, 55.
The importance of determining what makes a participant naïve or experienced shows up in two main ways within human factors studies. The most obvious is accurately designating participants, lest the data incorrectly reflect how experienced and naïve participants perform using the device. Less obvious is the difficulty of recruiting naïve participants, especially with small populations such as rare conditions. This can lead to the use of surrogates reducing the external validity of the data.
Current FDA Guidance (2016) states that "the most important consideration for test participants in human factors validation testing is that they represent the population of intended users." On the surface, this guidance seems intuitive; however, there are questions that need to be addressed when research requires user groups based on experience. Experience for participants can range from practicing injecting once to doing high frequency injections for years. These levels of experience are vastly different but can be sorted into the same category of experience as injecting once does technically make a person injection experienced. By considering participants with minimal experience as experienced users the potential population of naïve users drastically decreases while the experienced user data falsely represents how experience affects use. In fact, research would suggest that following a one-time use, the participant could be considered naïve within three months.
The Ebbinghaus forgetting curve is a model depicting the rate at which people forget learned information when there is no attempt to retain it. The study producing the model was conducted in 1880, but has been replicated several times establishing its relevance and accuracy in modern studies. It has been represented several ways but can be most simply graphed using an exponential decay equation modified by the stability of the memory (Woźniak, Gorzelańczyk, & Murakowski, 1994). The curve suggests that if a person conducts a serial position task (e.g., memorizing a list in order), after one day without rehearsal the average savings were 27% (Murre, Dros, 2015). Savings are the percentage of time saved calculated by dividing the time it took to relearn a list by the amount of time it took to learn the list originally. Simply, the ability to recall knowledge that isn’t reinforced steeply drops and settles towards negligible for regular memory.
’Flashbulb memories’ are argued to highly alter the rate of forgetting though. The term has been used since 1977 to describe memories that do not fade as easily as regular memories (Winograd & Neisser, 2006). The true nature of these memories is unclear, even to the point of questioning if they exist at all, but some researchers argue that emotion reduces the rate at which memory decays (Reisberg, Heuer, 2006) and flashbulb memories are often connected to major emotional events. Flashbulbs are significantly more vivid which can inspire certainty, but that doesn’t mean that the memory is preserved correctly. Instead, it has been observed that comparing recollections taken directly after an event and two years later, there is often a stark difference in the memories with the later recollection incorporating traceable biases. The later account will likely match future recollections of the same event though, showing its resistance to decay (Neisser, Harsch, 2006).
In the context of Human Factors research, if a participant’s injection experience occurs during a high emotion event it is more likely to create a flashbulb memory. These memories may make the participant feel experienced, but their accuracy remembering the event, and thus their actual knowledge regarding the device, can still be deemed negligible within the year. Should it not be a flashbulb memory, it will decay according to the Ebbinghaus forgetting curve and be considered negligible within the same time frame.
The best way for a memory to be preserved accurately remains in reviewing the information. If a person reviews information with certain frequency, it not only increases their ability to recall it but also slows the decay of the knowledge afterward (Chun & Heo, 2018). Each repetition flattens the Ebbinghaus forgetting curve more but never eliminates memory decay. One-time executions of tasks will fade rapidly, but if a participant had to inject once a week for a year their memory of that experience will be more accurate for longer until eventually becoming negligible as well.
Taking this research into account, a decision support tool has been created to make a more standard and rational based way of determining whether participants should be considered experienced or naïve in injection studies. The tool accounts for what experiences mean about the participant’s knowledge and functions through a series of if-then statements to decisively sort experience. As an example, consider an individual that participated in an injection study and is being considered for another one. The first criterion would be the time between studies. If it was less than a year, they may not participate as a naïve participant. If it was greater than a year, the tool indicates they can retain their experience qualification from the last study. However, if a naïve participant began performing injections in the past 12 months, they will now qualify as experienced.
With this type of standard tool to determine experience groups, Human Factors studies can reduce the problems caused by the current over inclusive experience designation. Ensuring participants are accurately represented as naïve or experienced will guarantee more accurate data about how experience affects usability. This is especially powerful when it helps avoid using surrogates in studies by increasing the percentage of the population deemed naïve. Though the decision support tool is not perfect, it provides a representation of experience that does not currently exist in our field.
To present these concepts the full decision support tool will be shown on the presentation/poster along with the applicable forms of the Ebbinghaus forgetting curve. Explanatory information for each curve will give context about how they apply to the decision support tool and relate to the original curve. Several examples of possible participants will be displayed to demonstrate the function of the tool as well as display a small array of the varied experiences people demonstrate. Finally, the presentation/poster will discuss and display the possible implications and benefits of investigating how experience is determined for human factors usability testing of medical devices.
Chun, B. A., & Heo, H. J. (n.d.). ICIET ’18. In The 6th International Conference on Information and Education Technology.
Harsch, N., & Neisser, U. (2006). Phantom flashbulbs: False recollections of hearing the news about Challenger. In Affect and accuracy in recall: Studies of “Flashbulb” memories. essay, Cambridge University Press.
Murre, J. M. J., & Dros, J. (2015). Replication and Analysis of Ebbinghaus’ forgetting curve. PLOS ONE, 10(7).
Reisberg, D., & Heuer, F. (2006). Remembering the details of emotional events. In Affect and accuracy in recall: Studies of “Flashbulb” memories. essay, Cambridge University Press.
Winograd, E., & Neisser, U. (2006a). Affect and accuracy in recall: Studies of “Flashbulb” memories. Cambridge University Press.
Winograd, E., & Neisser, U. (2006b). Preface. In Affect and accuracy in recall: studies of “flashbulb” memories. preface, Cambridge University Press.
Woźniak, P. A., Gorzelańczyk, E. J., & Murakowski, J. A. (1995). Two components of long-term memory. Acta Neurobiologiae Experimentalis, 55.
Event Type
Poster Presentation
TimeTuesday, March 264:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives