Presentation
MDD25 - Validation of a User Interface Design Taxonomy for Categorizing “Minor” vs “Other” Design Differences in Combination Products
DescriptionProposed generic drug-device combination products (DDCPs) submitted for FDA approval through the Abbreviated New Drug Application (ANDA) pathway require a comparison to the reference listed drug (RLD). An important part of this process is the identification of differences in the user interface of the two products, specifically an assessment of whether the design differences qualify as “minor” or “other”. When “other” design differences are present, further investigation/testing is necessary to ensure the proposed generic will be safe and effective for use when it is substituted for the RLD.
Until now, no common method or tool existed to classify UI design features. However, recently, an FDA-funded research project developed a UI design taxonomy providing a repeatable process for ensuring device manufacturers apply common language to describe UI design features. The taxonomy enables users to apply a common language for classifying design features into categories and subcategories and to define what qualifies as a “minor” versus “other” design difference in an FDA submission for a generic drug-device combination product (DDCP). The present IRB-approved research study aimed to evaluate the inter-rater reliability of the proposed UI design taxonomy and to identify any potential improvements to the taxonomy.
The taxonomy is a hierarchical structure classifying UI features of both labeling and interaction points on the device. Taxonomy users employ a custom-designed Microsoft Excel spreadsheet to link design features to specific tasks line by line on the spreadsheet. After creating a detailed task analysis, the user inputs each design feature associated with a specific sub-task using pull-down menus, allowing classification by category and subcategory with descriptors. The user may also leave comments specifying unique aspects of the design. To test the process, an inter-rater reliability study was conducted to ensure repeatability of the process across different taxonomy users.
Study participants were selected representing three potential user groups: human factors professionals, product development professionals, and engineering/design students. Recruitment occurred via email, social networks, and the project staff’s professional contacts. Each participant was provided with a visual map of the taxonomy, a set of instruction slides including taxonomy definitions and an example of taxonomy use, and the customized taxonomy Excel spreadsheet. The participant was also mailed two devices to evaluate with the taxonomy: (1) a blood pressure cuff, and (2) an over-the-counter nasal spray. A detailed task analysis for each device was already entered into the taxonomy spreadsheet to maintain consistency between participants (i.e., identification of tasks would not affect taxonomy use). Each participant was instructed to use the taxonomy to classify the design features of each device. The participant was asked to take notes on the process while completing the spreadsheet. The completed taxonomy spreadsheet was returned to study personnel via email.
Data analysis centered around evaluating the system’s usability and the study participants’ classification of UI design features. After each complete taxonomy was collected, all participants’ answers were recorded in a single comparison spreadsheet. SPSS was used to compute the Kappa statistic indicating the level of inter-rater reliability on a scale ranging from “no agreement” to “almost perfect agreement” between participants. Post-use interviews were also conducted providing the participant the opportunity to provide feedback on the taxonomy tool and its usability. Specifically, interviews were used to identify difficulties with the process and identify how and why design features were consistently or inconsistently classified. Results of this study indicated the taxonomy is a novel, useful tool for identifying and classifying differences in DDCP UI design features.
Until now, no common method or tool existed to classify UI design features. However, recently, an FDA-funded research project developed a UI design taxonomy providing a repeatable process for ensuring device manufacturers apply common language to describe UI design features. The taxonomy enables users to apply a common language for classifying design features into categories and subcategories and to define what qualifies as a “minor” versus “other” design difference in an FDA submission for a generic drug-device combination product (DDCP). The present IRB-approved research study aimed to evaluate the inter-rater reliability of the proposed UI design taxonomy and to identify any potential improvements to the taxonomy.
The taxonomy is a hierarchical structure classifying UI features of both labeling and interaction points on the device. Taxonomy users employ a custom-designed Microsoft Excel spreadsheet to link design features to specific tasks line by line on the spreadsheet. After creating a detailed task analysis, the user inputs each design feature associated with a specific sub-task using pull-down menus, allowing classification by category and subcategory with descriptors. The user may also leave comments specifying unique aspects of the design. To test the process, an inter-rater reliability study was conducted to ensure repeatability of the process across different taxonomy users.
Study participants were selected representing three potential user groups: human factors professionals, product development professionals, and engineering/design students. Recruitment occurred via email, social networks, and the project staff’s professional contacts. Each participant was provided with a visual map of the taxonomy, a set of instruction slides including taxonomy definitions and an example of taxonomy use, and the customized taxonomy Excel spreadsheet. The participant was also mailed two devices to evaluate with the taxonomy: (1) a blood pressure cuff, and (2) an over-the-counter nasal spray. A detailed task analysis for each device was already entered into the taxonomy spreadsheet to maintain consistency between participants (i.e., identification of tasks would not affect taxonomy use). Each participant was instructed to use the taxonomy to classify the design features of each device. The participant was asked to take notes on the process while completing the spreadsheet. The completed taxonomy spreadsheet was returned to study personnel via email.
Data analysis centered around evaluating the system’s usability and the study participants’ classification of UI design features. After each complete taxonomy was collected, all participants’ answers were recorded in a single comparison spreadsheet. SPSS was used to compute the Kappa statistic indicating the level of inter-rater reliability on a scale ranging from “no agreement” to “almost perfect agreement” between participants. Post-use interviews were also conducted providing the participant the opportunity to provide feedback on the taxonomy tool and its usability. Specifically, interviews were used to identify difficulties with the process and identify how and why design features were consistently or inconsistently classified. Results of this study indicated the taxonomy is a novel, useful tool for identifying and classifying differences in DDCP UI design features.
Event Type
Poster Presentation
TimeMonday, March 254:45pm - 6:15pm CDT
LocationSalon C
Digital Health
Simulation and Education
Hospital Environments
Medical and Drug Delivery Devices
Patient Safety Research and Initiatives