Presentation

· Contributors · Organizations · Search Program · Flagged · Happening Now

AI-Driven Heuristic Analysis: Enhancing Work Instructions with Generative Models

DescriptionThe present study investigated the feasibility and reliability of using generative AI to conduct heuristic evaluations of workplace instructions, comparing its performance to experienced human evaluators. A custom GPT model, fine-tuned with examples and heuristic criteria, evaluated nine sets of aerospace-based work instructions. The AI's output included identifying weaknesses, suggesting improvements, scoring heuristics (1-10), and providing rationales for its input. Results showed moderate-to-high agreement between the AI and human experts, with consistent and reproducible AI scoring. Qualitative analysis confirmed the AI's ability to identify common weaknesses and offer relevant feedback, sometimes even identifying issues missed by humans. While the AI provided adequate transparency, some explanations lacked detail, and minor discrepancies in judgment necessitate continued human oversight. The research demonstrates the potential of AI-driven heuristic evaluations to streamline assessment processes and augment human analysis in high-risk industries, while acknowledging the need for ongoing model refinement and improved transparency.

Contributors