Presentation
Setting up for Success: Usability Testing in Older Adults with Mild Cognitive Impairment
DescriptionCreating easy to use, safe, and effective digital health interventions (DHI) for older adults (ages 60 years and older) requires following best practices in human-centered technology design. An emerging area of DHI design and development is for persons with mild cognitive impairment (PwMCI). PwMCI experience mild changes in cognition beyond what is expected for their age and education and are at increased risk of developing dementia. It is estimated that 16-20% of people aged 65 years and older have mild cognitive impairment (Roberts et al., 2012). Conducting formative design research and usability testing with PwMCI can be challenging due to their difficulties remembering and/or focusing attention. We describe the process of planning and conducting a needs assessment and usability testing of a DHI (mobile app) with ten older adults (M = 70.30, SD = 6.31) with memory issues. We describe four key lessons from our experience that will be useful to human factors and research professionals planning to conduct design and usability testing research involving PwMCI.
This work is part of a project to develop and evaluate MEDSReM-M (Medication Education, Decision Support, Reminding and Monitoring-Memory), a mobile app aimed to improve medication adherence to antihypertensive medications in PwMCI with Montreal Cognitive Assessment (MoCA) scores of 20 to 25. Controlling co-morbid health conditions such as hypertension is important to slowing cognitive decline and possibly progression to dementia (Gottesman et al., 2014). PwMCI are at greater risk of medication nonadherence due to impairments in prospective memory (Woods et al., 2014). MEDSReM-M builds on the MEDSReM© system for cognitively normal older adults (Al-Saleh et al., 2022) that uses a theory-based Multifaceted Prospective Memory Intervention (MPMI) developed and tested in a previous study (Insel et al., 2016). Leveraging mobile health technology to foster hypertension medication self-management and remote monitoring would provide an effective and sustainable solutions to minimize the adverse effects of hypertension medication nonadherence while supporting the autonomy of PwMCI.
To optimize the design for PwMCI, we anticipated needing to create a simplified interface while keeping essential MPMI and MEDSReM features. To tailor the needs assessment and usability testing for PwMCI, we first conducted a cognitive walkthrough by considering the cognitive challenges that may be experienced by PwMCI in using MEDSReM.
Cognitive Challenges in PwMCI
Although the hallmark impairment in PwMCI is in episodic memory, these individuals may also present with deficits in other cognitive functions including executive function. Executive function enables one to execute goal-directed actions, adapt to changing environments, and monitor and regulate behavior, and includes cognitive processes such as inhibition, switching, and updating of working memory (Braver, 2012). The difficulties with implementation of executive functions in PwMCI may be apparent in the form of fixation on tasks, inability to have insight on errors, and difficulty keeping up with multistep sequence tasks. Furthermore, PwMCI may experience deficits in prospective memory. Prospective memory is the cognitive ability to remember to perform intended actions or tasks at a specific point in the future, which is particularly important for older adults in the context of medication-taking because they almost always need to manage multiple medications for various health conditions (Insel et al., 2016). To overcome those challenges effective digital health interventions should rely on external cues (e.g., setting alarms) and/or rely on internal strategies, such as associating medication with a routine event-based activity, to help trigger the memory.
Process
First, we held team meetings to educate staff on the characteristics and special needs of PwMCI. This was needed because several new members of the study team lacked knowledge of the cognitive challenges experienced by PwMCI. The meetings provided a shared understanding and sense of empathy for the challenges and difficulties PwMCI might encounter learning to use a mobile app to manage medication and the creation of personas representative of PwMCI. The initial cognitive walkthrough, conducted by the study team and lab-affiliated students trained in the needs of PwMCI, revealed several potential obstacles in using MEDSReM by PwMCI (Azevedo et al., 2023) that informed necessary design modifications.
Second, we finalized details on how to best conduct the needs assessment and usability testing. An insight from team experts was that PwMCI might not be fully aware of their cognitive issues and that assessment instruments and usability testing might foster a sense of frustration and distress and in some cases, a realization of their condition. Therefore, it was decided to start with the needs assessment or usability testing and place cognitive assessments as the last steps in the study visit. Additionally, we wanted to ensure the study visit ends on a positive emotional note to counter potential negative feelings generated by the usability tests or cognitive assessments. We asked participants an open-ended question allowing them to share their recommendations, thus highlighting their positive contribution.
Third, we devoted considerable time to creating materials used during the needs assessment and the instructions for using the prototype mobile app during usability testing. Questions and tasks were organized into logical units related to older adults’ experiences managing their hypertensive medications progressing from relatively simple to more complex questions and tasks. PowerPoint was used to create a slide for each question or step with simple and clearly worded instructions. The prototype app was shown using screenshots of the app and visual guides (e.g., arrows, circled areas) to direct attention to key elements of the user interface. The materials were reviewed by the team and revised to reduce the cognitive burden and possible confusion for PwMCI.
Fourth, considering the challenges PwMCI may experience in following multistep tasks and becoming fixated on a task, we provided additional time for participants to respond to questions and watched for signs of confusion and frustration, at which point we provided a verbal prompt or a “flashcard” consisting of essential instructions needed to complete a task. Thus, we prioritized gaining mastery to complete task rather than strictly time-based criteria and providing PwMCI additional time and access to training materials during the testing sessions (Hale et al., 2023). We were also mindful that PwMCI may not recognize they are having challenges or making errors, and reluctant to ask for help.
Summary and Conclusion
The needs assessment and usability testing confirmed many of the initial issues identified in the cognitive walkthrough had been resolved in the MEDSReM-M app prototype. Additional usability issues were identified during the needs assessment and usability testing and have been addressed in the app design. We are now completing additional rounds of usability testing in preparation for testing MEDSReM-M in a randomized controlled trial in 2024.
Creating an easy-to-use mobile app for older adults with MCI poses many challenges, including issues with how to appropriately identify users’ needs and test prototypes to ensure usability. We identify and describe four key lessons from our experience that will be useful to human factors and research professionals in conducting design and usability testing with older adults with MCI. First, understanding the challenges that PwMCI experience in memory, attention, and learning new tasks. Second, incorporating this understanding into the study design and the testing order in which questionnaires and tasks are delivered. Third, creating materials that provide simple and clear instructions that focus on a single task or question. Fourth, training staff to be mindful that PwMCI may experience greater confusion and frustration completing new tasks and may require more time and support.
This work is part of a project to develop and evaluate MEDSReM-M (Medication Education, Decision Support, Reminding and Monitoring-Memory), a mobile app aimed to improve medication adherence to antihypertensive medications in PwMCI with Montreal Cognitive Assessment (MoCA) scores of 20 to 25. Controlling co-morbid health conditions such as hypertension is important to slowing cognitive decline and possibly progression to dementia (Gottesman et al., 2014). PwMCI are at greater risk of medication nonadherence due to impairments in prospective memory (Woods et al., 2014). MEDSReM-M builds on the MEDSReM© system for cognitively normal older adults (Al-Saleh et al., 2022) that uses a theory-based Multifaceted Prospective Memory Intervention (MPMI) developed and tested in a previous study (Insel et al., 2016). Leveraging mobile health technology to foster hypertension medication self-management and remote monitoring would provide an effective and sustainable solutions to minimize the adverse effects of hypertension medication nonadherence while supporting the autonomy of PwMCI.
To optimize the design for PwMCI, we anticipated needing to create a simplified interface while keeping essential MPMI and MEDSReM features. To tailor the needs assessment and usability testing for PwMCI, we first conducted a cognitive walkthrough by considering the cognitive challenges that may be experienced by PwMCI in using MEDSReM.
Cognitive Challenges in PwMCI
Although the hallmark impairment in PwMCI is in episodic memory, these individuals may also present with deficits in other cognitive functions including executive function. Executive function enables one to execute goal-directed actions, adapt to changing environments, and monitor and regulate behavior, and includes cognitive processes such as inhibition, switching, and updating of working memory (Braver, 2012). The difficulties with implementation of executive functions in PwMCI may be apparent in the form of fixation on tasks, inability to have insight on errors, and difficulty keeping up with multistep sequence tasks. Furthermore, PwMCI may experience deficits in prospective memory. Prospective memory is the cognitive ability to remember to perform intended actions or tasks at a specific point in the future, which is particularly important for older adults in the context of medication-taking because they almost always need to manage multiple medications for various health conditions (Insel et al., 2016). To overcome those challenges effective digital health interventions should rely on external cues (e.g., setting alarms) and/or rely on internal strategies, such as associating medication with a routine event-based activity, to help trigger the memory.
Process
First, we held team meetings to educate staff on the characteristics and special needs of PwMCI. This was needed because several new members of the study team lacked knowledge of the cognitive challenges experienced by PwMCI. The meetings provided a shared understanding and sense of empathy for the challenges and difficulties PwMCI might encounter learning to use a mobile app to manage medication and the creation of personas representative of PwMCI. The initial cognitive walkthrough, conducted by the study team and lab-affiliated students trained in the needs of PwMCI, revealed several potential obstacles in using MEDSReM by PwMCI (Azevedo et al., 2023) that informed necessary design modifications.
Second, we finalized details on how to best conduct the needs assessment and usability testing. An insight from team experts was that PwMCI might not be fully aware of their cognitive issues and that assessment instruments and usability testing might foster a sense of frustration and distress and in some cases, a realization of their condition. Therefore, it was decided to start with the needs assessment or usability testing and place cognitive assessments as the last steps in the study visit. Additionally, we wanted to ensure the study visit ends on a positive emotional note to counter potential negative feelings generated by the usability tests or cognitive assessments. We asked participants an open-ended question allowing them to share their recommendations, thus highlighting their positive contribution.
Third, we devoted considerable time to creating materials used during the needs assessment and the instructions for using the prototype mobile app during usability testing. Questions and tasks were organized into logical units related to older adults’ experiences managing their hypertensive medications progressing from relatively simple to more complex questions and tasks. PowerPoint was used to create a slide for each question or step with simple and clearly worded instructions. The prototype app was shown using screenshots of the app and visual guides (e.g., arrows, circled areas) to direct attention to key elements of the user interface. The materials were reviewed by the team and revised to reduce the cognitive burden and possible confusion for PwMCI.
Fourth, considering the challenges PwMCI may experience in following multistep tasks and becoming fixated on a task, we provided additional time for participants to respond to questions and watched for signs of confusion and frustration, at which point we provided a verbal prompt or a “flashcard” consisting of essential instructions needed to complete a task. Thus, we prioritized gaining mastery to complete task rather than strictly time-based criteria and providing PwMCI additional time and access to training materials during the testing sessions (Hale et al., 2023). We were also mindful that PwMCI may not recognize they are having challenges or making errors, and reluctant to ask for help.
Summary and Conclusion
The needs assessment and usability testing confirmed many of the initial issues identified in the cognitive walkthrough had been resolved in the MEDSReM-M app prototype. Additional usability issues were identified during the needs assessment and usability testing and have been addressed in the app design. We are now completing additional rounds of usability testing in preparation for testing MEDSReM-M in a randomized controlled trial in 2024.
Creating an easy-to-use mobile app for older adults with MCI poses many challenges, including issues with how to appropriately identify users’ needs and test prototypes to ensure usability. We identify and describe four key lessons from our experience that will be useful to human factors and research professionals in conducting design and usability testing with older adults with MCI. First, understanding the challenges that PwMCI experience in memory, attention, and learning new tasks. Second, incorporating this understanding into the study design and the testing order in which questionnaires and tasks are delivered. Third, creating materials that provide simple and clear instructions that focus on a single task or question. Fourth, training staff to be mindful that PwMCI may experience greater confusion and frustration completing new tasks and may require more time and support.
Event Type
Oral Presentations
TimeTuesday, March 263:50pm - 4:10pm CDT
LocationSalon A-2
Digital Health