Presentation
Guidance for implementation of health technology interventions
SessionHealth Technology (DH2)
DescriptionINTRODUCTION
In a Randomized Controlled Trial (RCT), participants are randomly assigned to one of at least two groups: one or more experimental groups receiving the intervention(s) being tested, and a comparison/control condition receiving an alternative, perhaps more conventional, treatment (Kendall, 2003). Intention-to-treat (ITT) is an analysis method used in RCTs where all participants randomly assigned to groups are analyzed even if they did not fully participate in the intervention. ITT ensures that all participants initially assigned are included in the analysis, whether they completed participation for the duration of the study or not. This approach provides a conservative and realistic assessment of the treatment's effectiveness in real-world conditions and is widely adopted in clinical research and other fields where participant nonadherence and dropouts occur.
Particularly in RCTs that test technology interventions (e.g., mHealth applications) a critical aspect of the ITT analysis hinges not only on what the technology is designed to do, but also on how well participants navigate, use, and adopt these systems. From 2012-2019, 21-25% of users reportedly abandoned mobile apps they downloaded after launching the app once (Business 2 Community, 2019). Technology non-adoption and abandonment are significantly related to perceived ease of use and usefulness (Azevedo et al., 2022; Mitzner et al., 2019). Insights regarding why participants may not use the technology intervention is valuable for interpreting results from RCTs using ITT analysis. Participants' non-use can stem from multiple reasons: 1) lack of understanding in how to use the system; 2) trouble remembering to use it; 3) lack of understanding the benefits of using the system; 4) technical difficulties (e.g., hardware problems, internet access); 5) the technology interferes with participants’ routines (e.g., traveling, receiving houseguests, illness); and/or 6) a lack of willingness to engage with the technology for other reasons (Blanson Henkemans, Rogers, & Dumay, 2011). Insights regarding why participants do not use technology interventions can guide the interpretation of ITT analysis findings.
Our objective in this methods-focused presentation is to distinguish instances of non-use resulting from participants' unfamiliarity or lack of interest in utilizing the system from alternative reasons. Identifying causes of technology non-use helps pinpoint features that should be addressed to improve the user experience. For example, limited understanding of how to use or benefits of use can be addressed by training, whereas technical difficulties (e.g., hardware, internet access) can be ameliorated by technical support. Our approach provides guidance for implementing health technology interventions, including training and protocol recommendations.
MEDSReM SYSTEM STUDY
We developed a system named MEDSReM© (Medication Education, Decision Support, Reminding, and Monitoring): a mobile app and a companion website designed to improve medication adherence for older adults taking hypertensive medications (Al-Saleh et al., 2023). Following several iterative rounds of design, heuristic evaluation, usability testing, and summative testing (Hale et al., 2023), we launched an RCT to assess the efficacy of this system to improve medication adherence.
Our study employs a comprehensive approach to address the complexities surrounding potential non-use of the MEDSReM system. Our study protocol consists of five visits. Described here are two of those visits, focused on a teach-back session one week after the intervention training session and a recheck after ten days to assess initial system use.
DESIGN AND GUIDANCE FOR IMPLEMENTATION
To be eligible for the study, participants had to be nonadherent to their antihypertensive medications, during a 4-week monitoring period. Eligible participants were randomized to one of two groups: (a) the MEDSReM system, consisting of a mobile app and website, or (b) the website-only group.
Training Visit: Participants receive in-depth training on how to effectively use the MEDSReM system or the website. Training includes practice to help consolidate understanding and knowledge as well as education about hypertension and medication adherence. They are provided printed instruction guides (Azevedo et al., 2022); guidance to learn and navigate the technology; contact resources; and log sheets to record technical difficulties.
Home Use (1-week): Participants use their assigned technology for one week, which allows for increased familiarity with the mHealth application and/or the website and instruction guides.
Teach-back Visit: A standardized protocol is used during this visit to assess participants' comprehension and mastery of their assigned intervention. Participants are encouraged to demonstrate use and ask questions to address concerns. This step serves as an additional layer of assurance that participants are adequately trained and confident in using the technology on their own. The approach was inspired by the ‘teach-back/teach to goal’ technique used in patient education to ensure patients understand and are empowered to use key concepts to support self-care (Baker et al., 2011). This approach provides an opportunity to resolve concerns that arise during their initial technology engagement (e.g., just in time support).
Use Check: After 10 days, we conduct a check on use of the website by analyzing logs to determine whether participants have accessed the website at least once. This assessment is relevant to both groups and provides a final opportunity to identify participants who are not using the website and to offer timely troubleshooting support. If participants are not using the website, a researcher contacts them to uncover the reason, which may include: a) confusion about specific technology components; b) trouble remembering; c) no perceived need; c) technical problems; d) equipment breakage (e.g., phone); e) item lost (e.g., instruction guides); f) the technology interferes with participants’ routines;, and g) other reasons. For each reason assessors are instructed to provide technical troubleshooting if appropriate, and take detailed notes.
Continued Home Use: Following the Use Check, we do not check on participants' technology use. This approach aligns with the principles of ITT analysis. If participants manifest the need for reinforcing understanding of using the system (i.e., communicating with the research staff requesting help), we point them to the instruction guides they already have. Our approach helps identify and address initial barriers to technology adoption related to ease of use, perceived usefulness, among other reasons for incomplete participation in health interventions, distinguishing non-use stemming from a reluctance or unwillingness to interact with the technology.
DISCUSSION
When presenting the results of an RCT, researchers typically report both the ITT analysis and a per-protocol analysis (only participants who completed the treatment) to obtain a comprehensive view of the treatment effects, acknowledging that real-world adherence to the intervention may vary.
As a future approach, tailoring instruction for participants in a RCT training phase could involve using various measures to assess their readiness and capabilities to customize the amount of instruction provided to each participant, similar to the ‘teach-to-goal’ approach in education (Baker et al., 2011). In health technology intervention studies, these measures could include mobile technology proficiency (e.g., MDPQ-16; Roque & Boot, 2016), technology readiness (Technology Readiness Index 2.0 and Technology Readiness Index-Health; Parasuraman & Colby, 2014; Trinh et al., 2023), and/or domain knowledge (e.g., hypertension; Hypertension Knowledge-Level Scale; HK-LS; Erkoc et al., 2012). By evaluating these factors, we can determine how familiar participants are with mobile devices and technology in general, their readiness to engage with a technology intervention, and existing knowledge related to the subject matter, such as hypertension.
CONCLUSION
Our approach offers guidance for implementation of mHealth technology interventions to facilitate participant understanding and support their engagement with the technology. To address complexities surrounding non-use of mHealth technology, we employed a comprehensive approach, consisting of a structured timeline involving a training phase, a home use period, a teach-back protocol, and a subsequent use check during initial independent use. By incorporating these methodological steps, we aim to comprehensively address challenges associated with technology adoption.
In a Randomized Controlled Trial (RCT), participants are randomly assigned to one of at least two groups: one or more experimental groups receiving the intervention(s) being tested, and a comparison/control condition receiving an alternative, perhaps more conventional, treatment (Kendall, 2003). Intention-to-treat (ITT) is an analysis method used in RCTs where all participants randomly assigned to groups are analyzed even if they did not fully participate in the intervention. ITT ensures that all participants initially assigned are included in the analysis, whether they completed participation for the duration of the study or not. This approach provides a conservative and realistic assessment of the treatment's effectiveness in real-world conditions and is widely adopted in clinical research and other fields where participant nonadherence and dropouts occur.
Particularly in RCTs that test technology interventions (e.g., mHealth applications) a critical aspect of the ITT analysis hinges not only on what the technology is designed to do, but also on how well participants navigate, use, and adopt these systems. From 2012-2019, 21-25% of users reportedly abandoned mobile apps they downloaded after launching the app once (Business 2 Community, 2019). Technology non-adoption and abandonment are significantly related to perceived ease of use and usefulness (Azevedo et al., 2022; Mitzner et al., 2019). Insights regarding why participants may not use the technology intervention is valuable for interpreting results from RCTs using ITT analysis. Participants' non-use can stem from multiple reasons: 1) lack of understanding in how to use the system; 2) trouble remembering to use it; 3) lack of understanding the benefits of using the system; 4) technical difficulties (e.g., hardware problems, internet access); 5) the technology interferes with participants’ routines (e.g., traveling, receiving houseguests, illness); and/or 6) a lack of willingness to engage with the technology for other reasons (Blanson Henkemans, Rogers, & Dumay, 2011). Insights regarding why participants do not use technology interventions can guide the interpretation of ITT analysis findings.
Our objective in this methods-focused presentation is to distinguish instances of non-use resulting from participants' unfamiliarity or lack of interest in utilizing the system from alternative reasons. Identifying causes of technology non-use helps pinpoint features that should be addressed to improve the user experience. For example, limited understanding of how to use or benefits of use can be addressed by training, whereas technical difficulties (e.g., hardware, internet access) can be ameliorated by technical support. Our approach provides guidance for implementing health technology interventions, including training and protocol recommendations.
MEDSReM SYSTEM STUDY
We developed a system named MEDSReM© (Medication Education, Decision Support, Reminding, and Monitoring): a mobile app and a companion website designed to improve medication adherence for older adults taking hypertensive medications (Al-Saleh et al., 2023). Following several iterative rounds of design, heuristic evaluation, usability testing, and summative testing (Hale et al., 2023), we launched an RCT to assess the efficacy of this system to improve medication adherence.
Our study employs a comprehensive approach to address the complexities surrounding potential non-use of the MEDSReM system. Our study protocol consists of five visits. Described here are two of those visits, focused on a teach-back session one week after the intervention training session and a recheck after ten days to assess initial system use.
DESIGN AND GUIDANCE FOR IMPLEMENTATION
To be eligible for the study, participants had to be nonadherent to their antihypertensive medications, during a 4-week monitoring period. Eligible participants were randomized to one of two groups: (a) the MEDSReM system, consisting of a mobile app and website, or (b) the website-only group.
Training Visit: Participants receive in-depth training on how to effectively use the MEDSReM system or the website. Training includes practice to help consolidate understanding and knowledge as well as education about hypertension and medication adherence. They are provided printed instruction guides (Azevedo et al., 2022); guidance to learn and navigate the technology; contact resources; and log sheets to record technical difficulties.
Home Use (1-week): Participants use their assigned technology for one week, which allows for increased familiarity with the mHealth application and/or the website and instruction guides.
Teach-back Visit: A standardized protocol is used during this visit to assess participants' comprehension and mastery of their assigned intervention. Participants are encouraged to demonstrate use and ask questions to address concerns. This step serves as an additional layer of assurance that participants are adequately trained and confident in using the technology on their own. The approach was inspired by the ‘teach-back/teach to goal’ technique used in patient education to ensure patients understand and are empowered to use key concepts to support self-care (Baker et al., 2011). This approach provides an opportunity to resolve concerns that arise during their initial technology engagement (e.g., just in time support).
Use Check: After 10 days, we conduct a check on use of the website by analyzing logs to determine whether participants have accessed the website at least once. This assessment is relevant to both groups and provides a final opportunity to identify participants who are not using the website and to offer timely troubleshooting support. If participants are not using the website, a researcher contacts them to uncover the reason, which may include: a) confusion about specific technology components; b) trouble remembering; c) no perceived need; c) technical problems; d) equipment breakage (e.g., phone); e) item lost (e.g., instruction guides); f) the technology interferes with participants’ routines;, and g) other reasons. For each reason assessors are instructed to provide technical troubleshooting if appropriate, and take detailed notes.
Continued Home Use: Following the Use Check, we do not check on participants' technology use. This approach aligns with the principles of ITT analysis. If participants manifest the need for reinforcing understanding of using the system (i.e., communicating with the research staff requesting help), we point them to the instruction guides they already have. Our approach helps identify and address initial barriers to technology adoption related to ease of use, perceived usefulness, among other reasons for incomplete participation in health interventions, distinguishing non-use stemming from a reluctance or unwillingness to interact with the technology.
DISCUSSION
When presenting the results of an RCT, researchers typically report both the ITT analysis and a per-protocol analysis (only participants who completed the treatment) to obtain a comprehensive view of the treatment effects, acknowledging that real-world adherence to the intervention may vary.
As a future approach, tailoring instruction for participants in a RCT training phase could involve using various measures to assess their readiness and capabilities to customize the amount of instruction provided to each participant, similar to the ‘teach-to-goal’ approach in education (Baker et al., 2011). In health technology intervention studies, these measures could include mobile technology proficiency (e.g., MDPQ-16; Roque & Boot, 2016), technology readiness (Technology Readiness Index 2.0 and Technology Readiness Index-Health; Parasuraman & Colby, 2014; Trinh et al., 2023), and/or domain knowledge (e.g., hypertension; Hypertension Knowledge-Level Scale; HK-LS; Erkoc et al., 2012). By evaluating these factors, we can determine how familiar participants are with mobile devices and technology in general, their readiness to engage with a technology intervention, and existing knowledge related to the subject matter, such as hypertension.
CONCLUSION
Our approach offers guidance for implementation of mHealth technology interventions to facilitate participant understanding and support their engagement with the technology. To address complexities surrounding non-use of mHealth technology, we employed a comprehensive approach, consisting of a structured timeline involving a training phase, a home use period, a teach-back protocol, and a subsequent use check during initial independent use. By incorporating these methodological steps, we aim to comprehensively address challenges associated with technology adoption.
Event Type
Oral Presentations
TimeMonday, March 252:00pm - 2:30pm CDT
LocationSalon A-2
Digital Health