Presentation
Smarter UX Evaluations? Comparing AI and Human Experts in Usability Analysis
SessionPoster Session 1
DescriptionThis study investigates the potential of custom generative AI models for evaluating product usability, comparing their assessments to those of human experts. A custom GPT model, trained on social robot usability feedback and Nielsen's heuristics, demonstrated moderate to high agreement with human experts in quantitative and qualitative evaluations across three product evaluations. The AI model exhibited good test-retest reliability and provided clear, logical explanations for its scores, correctly identifying usability issues and offering suitable recommendations in many cases. While results indicate AI's promise as a tool for efficient usability assessments, some limitations were noted in handling ambiguous feedback, suggesting the continued need for human oversight in complex scenarios. The study highlights the potential of AI to reduce resource demands in product development while maintaining adequate assessment quality. Further research should explore the generalizability of these findings across different product domains and the impact of larger training datasets, focusing on enhancing AI evaluation transparency and integration within existing usability frameworks.
Event Type
Poster
TimeTuesday, October 14th5:30pm - 6:30pm CDT
LocationRiverside East