Presentation
The Difference Between Usability Quality and Government Regulations
DescriptionThe Difference Between Usability Quality and Government Regulations
Panelists: Robin Littlejohn (MedStar Heath), Huiyang Li (Food & Drug Administration), Chelsea Riehle (Baxter), Priyanka Shah (ECRI), Tim Arnold (Veteran’s Health Administration)
Moderator: Kathryn Tippey (Philips)
Government regulations of medical devices center on ensuring that these devices are safe and effective for clinical or patient use. For example, in the United States, the Food & Drug Administration (FDA) requires companies to submit rigorous human factors usability testing documentation to show any residual risks for harm when using a device are appropriately mitigated. This does not however ensure the usability quality of the device. The purpose of this panel is to discuss how we define and measure usability quality, how this differs from the guidelines and regulations issued by government agencies, and what methods might be implemented in practice to bridge this gap.
** Note: for all panelists, the views and opinions expressed here are those of the panelist and do not necessarily reflect official policy or position of the agency.
Robin Littlejohn (MedStar Health)
Ms. Littlejohn is a Research and Human Factors Scientist at MedStar Health National Center for Human Factors in Healthcare. Ms. Littlejohn provides usability evaluations of medical devices,
health information technology (health IT), and healthcare processes for both MedStar Health entities and commercial clients. In her role, she applies her expertise to evaluate user-centered systems and designs, seeking to understand the interaction of multifaceted systems and identify system modifications to maximize the safety and efficacy of processes for the benefit of all system users, while considering the needs of all stakeholders.
Thoughts on the topic:
While capturing major risk factors for harm, and creating “broad strokes” for usability evaluation, FDA regulation does not always capture other contextual factors that affect usability on a smaller/more focused scale. Potentially uncaptured contextual factors include socio-economic factors, cultural barriers to technology acceptance and/or use, willingness to engage in technology/process use, available external support affecting use/long-term adoption, system and workflow effects/constraints impacting an implemented solution that may otherwise be deemed “usable, safe, and effective” by regulatory standards. FDA regulatory requirements are often treated as a “final check-list” item where usability is evaluated at the end of the design cycle, whereas usability quality should be continuously assessed throughout the design and development cycle, employ user-centric and co-design methods, and undergo continuous iterations to ensure a solution aligns with the needs and goals of specific users in specific use environments.
Huiyang Li (Food & Drug Administration)
Dr. Huiyang Li is a Lead Reviewer and Human Factors Consultant at the FDA Center for Devices and Radiological Health. She leads and conducts total product life cycle reviews, including both pre-market applications and post-market issues for general surgical devices and cancer diagnosis and treatment devices. Before joining the FDA, Dr. Li worked as a human factors subject matter expert at Pfizer and a design consultancy. Prior to transition to industry, she was an Assistant Professor at Binghamton University teaching and conducting research in human factors with a focus in health care applications.
Thoughts on the topic:
A product having high quality usability means that qualified human factors/usability professionals use the appropriate human-centered design process and applied best usability practices, such as applicable design principles and appropriate human factors standards, to the largest possible extent. The medical device premarket review, however, focuses mainly on the outcomes of a human factors/usability validation study, i.e., use errors in the use of the device or understanding of labeling. This approach could result in a device being able to obtain clearance or approval even when usability quality is not the best. However, aiming at the “best” usability quality is still important to medical devices from the regulatory perspective because regulation goes much beyond the premarket review. I believe only the devices that hold high standard of usability quality will thrive in the long run.
Priyanka Shah (ECRI)
Priyanka Shah is a Principal Project Engineer at ECRI, where she performs comparative medical device evaluations, and investigates medical device-related accidents. Her current subject-matter expertise includes patient monitoring, alarm management, telehealth, human factors analysis, health information technology (HIT) safety and medication reconciliation within electronic health records (EHRs). Priyanka has over 8 years of experience working with various healthcare technologies, and she routinely consults healthcare facilities with technology selection and procurement projects in these areas. In addition, Priyanka is a part of the ECRI human factors engineering team and has been involved with various ECRI human factors research, publication, and business development efforts. Priyanka is a regular contributor to ECRI's annual Top10 Technology Hazards list. Priyanka came to ECRI with a background in research engineering and program management. She holds an MS degree in Biomedical Engineering from Purdue University. Priyanka has presented on various topics related to her subject matter expertise at several international, national, and local conferences including AAMI, HFES, and DVHIMSS. Priyanka is also an active member of the American College of Clinical Engineering (ACCE) Education Committee.
Thoughts on the topic:
By design, pre-market usability and quality focus on whether a device meets its design criteria and can be safely used as the manufacturer intended. Once deployed, however, users will have their own intentions and devices may find themselves in situations that manufacturers and regulators haven’t foreseen. ECRI investigates errors and harm and disseminates known failure modes to close these gaps between intended use and real patient care.
Chelsea Riehle (Baxter)
Chelsea Riehle is a Senior Human Factors Engineer at Baxter in their Care & Connectivity Solutions Division. She performs human factors activities throughout the product development lifecycle, from early formative studies for new products through post market usability analysis of on-market products.
Thoughts on the topic:
Developing products with superior usability requires going beyond the regulatory imperative to develop safe and effective devices. But it can be challenging to quantify the usability of a product and the benefits that brings to customers.
Tim Arnold (Veteran’s Health Administration)
Tim is a Senior Engineering Psychologist in VA’s Human Factors Engineering Division. With a history of providing team-based health care services, he practiced as a clinical pharmacy practitioner in acute and critical care settings. Tim completed a post-doctorate interprofessional patient safety fellowship with work in human computer interaction focused on human-centered design of clinical decision support systems focused on pharmacotherapeutics with an emphasis on error-tolerant systems design. Currently, Tim works within multiprofessional teams supporting human factors and safety engineering efforts and exploring approaches in text analytics for facilitating language awareness and informing human-centered design.
Thoughts on the topic:
Language is dynamic, is diverse, and changes over time. It is possible usability and its synonyms are words that have varied meanings across many user groups. To start understanding the difference between usability quality and government regulations, we should continue to discuss and describe the diverse language and conceptions and meanings of usability.
Moderator: Kathryn Tippey (Philips)
Kathryn (Katie) Tippey, Ph.D., is a Senior Usability Designer at Philips and is part of their Usability Center of Excellence. The purpose of the Center is to support designers and usability engineers at Philips is the development of high-quality products that are safe and effective. Within this group, Katie specializes in formative usability work on special topics, particularly those involving novel measurement techniques, and in the company-wide efforts to increase the understanding and importance of usability.
Panelists: Robin Littlejohn (MedStar Heath), Huiyang Li (Food & Drug Administration), Chelsea Riehle (Baxter), Priyanka Shah (ECRI), Tim Arnold (Veteran’s Health Administration)
Moderator: Kathryn Tippey (Philips)
Government regulations of medical devices center on ensuring that these devices are safe and effective for clinical or patient use. For example, in the United States, the Food & Drug Administration (FDA) requires companies to submit rigorous human factors usability testing documentation to show any residual risks for harm when using a device are appropriately mitigated. This does not however ensure the usability quality of the device. The purpose of this panel is to discuss how we define and measure usability quality, how this differs from the guidelines and regulations issued by government agencies, and what methods might be implemented in practice to bridge this gap.
** Note: for all panelists, the views and opinions expressed here are those of the panelist and do not necessarily reflect official policy or position of the agency.
Robin Littlejohn (MedStar Health)
Ms. Littlejohn is a Research and Human Factors Scientist at MedStar Health National Center for Human Factors in Healthcare. Ms. Littlejohn provides usability evaluations of medical devices,
health information technology (health IT), and healthcare processes for both MedStar Health entities and commercial clients. In her role, she applies her expertise to evaluate user-centered systems and designs, seeking to understand the interaction of multifaceted systems and identify system modifications to maximize the safety and efficacy of processes for the benefit of all system users, while considering the needs of all stakeholders.
Thoughts on the topic:
While capturing major risk factors for harm, and creating “broad strokes” for usability evaluation, FDA regulation does not always capture other contextual factors that affect usability on a smaller/more focused scale. Potentially uncaptured contextual factors include socio-economic factors, cultural barriers to technology acceptance and/or use, willingness to engage in technology/process use, available external support affecting use/long-term adoption, system and workflow effects/constraints impacting an implemented solution that may otherwise be deemed “usable, safe, and effective” by regulatory standards. FDA regulatory requirements are often treated as a “final check-list” item where usability is evaluated at the end of the design cycle, whereas usability quality should be continuously assessed throughout the design and development cycle, employ user-centric and co-design methods, and undergo continuous iterations to ensure a solution aligns with the needs and goals of specific users in specific use environments.
Huiyang Li (Food & Drug Administration)
Dr. Huiyang Li is a Lead Reviewer and Human Factors Consultant at the FDA Center for Devices and Radiological Health. She leads and conducts total product life cycle reviews, including both pre-market applications and post-market issues for general surgical devices and cancer diagnosis and treatment devices. Before joining the FDA, Dr. Li worked as a human factors subject matter expert at Pfizer and a design consultancy. Prior to transition to industry, she was an Assistant Professor at Binghamton University teaching and conducting research in human factors with a focus in health care applications.
Thoughts on the topic:
A product having high quality usability means that qualified human factors/usability professionals use the appropriate human-centered design process and applied best usability practices, such as applicable design principles and appropriate human factors standards, to the largest possible extent. The medical device premarket review, however, focuses mainly on the outcomes of a human factors/usability validation study, i.e., use errors in the use of the device or understanding of labeling. This approach could result in a device being able to obtain clearance or approval even when usability quality is not the best. However, aiming at the “best” usability quality is still important to medical devices from the regulatory perspective because regulation goes much beyond the premarket review. I believe only the devices that hold high standard of usability quality will thrive in the long run.
Priyanka Shah (ECRI)
Priyanka Shah is a Principal Project Engineer at ECRI, where she performs comparative medical device evaluations, and investigates medical device-related accidents. Her current subject-matter expertise includes patient monitoring, alarm management, telehealth, human factors analysis, health information technology (HIT) safety and medication reconciliation within electronic health records (EHRs). Priyanka has over 8 years of experience working with various healthcare technologies, and she routinely consults healthcare facilities with technology selection and procurement projects in these areas. In addition, Priyanka is a part of the ECRI human factors engineering team and has been involved with various ECRI human factors research, publication, and business development efforts. Priyanka is a regular contributor to ECRI's annual Top10 Technology Hazards list. Priyanka came to ECRI with a background in research engineering and program management. She holds an MS degree in Biomedical Engineering from Purdue University. Priyanka has presented on various topics related to her subject matter expertise at several international, national, and local conferences including AAMI, HFES, and DVHIMSS. Priyanka is also an active member of the American College of Clinical Engineering (ACCE) Education Committee.
Thoughts on the topic:
By design, pre-market usability and quality focus on whether a device meets its design criteria and can be safely used as the manufacturer intended. Once deployed, however, users will have their own intentions and devices may find themselves in situations that manufacturers and regulators haven’t foreseen. ECRI investigates errors and harm and disseminates known failure modes to close these gaps between intended use and real patient care.
Chelsea Riehle (Baxter)
Chelsea Riehle is a Senior Human Factors Engineer at Baxter in their Care & Connectivity Solutions Division. She performs human factors activities throughout the product development lifecycle, from early formative studies for new products through post market usability analysis of on-market products.
Thoughts on the topic:
Developing products with superior usability requires going beyond the regulatory imperative to develop safe and effective devices. But it can be challenging to quantify the usability of a product and the benefits that brings to customers.
Tim Arnold (Veteran’s Health Administration)
Tim is a Senior Engineering Psychologist in VA’s Human Factors Engineering Division. With a history of providing team-based health care services, he practiced as a clinical pharmacy practitioner in acute and critical care settings. Tim completed a post-doctorate interprofessional patient safety fellowship with work in human computer interaction focused on human-centered design of clinical decision support systems focused on pharmacotherapeutics with an emphasis on error-tolerant systems design. Currently, Tim works within multiprofessional teams supporting human factors and safety engineering efforts and exploring approaches in text analytics for facilitating language awareness and informing human-centered design.
Thoughts on the topic:
Language is dynamic, is diverse, and changes over time. It is possible usability and its synonyms are words that have varied meanings across many user groups. To start understanding the difference between usability quality and government regulations, we should continue to discuss and describe the diverse language and conceptions and meanings of usability.
Moderator: Kathryn Tippey (Philips)
Kathryn (Katie) Tippey, Ph.D., is a Senior Usability Designer at Philips and is part of their Usability Center of Excellence. The purpose of the Center is to support designers and usability engineers at Philips is the development of high-quality products that are safe and effective. Within this group, Katie specializes in formative usability work on special topics, particularly those involving novel measurement techniques, and in the company-wide efforts to increase the understanding and importance of usability.
Event Type
Discussion Panel
TimeMonday, March 253:30pm - 4:30pm CDT
LocationContinental Ballroom, Lobby Level
Medical and Drug Delivery Devices