Mistake Detection in Egocentric Procedural Videos Via CoT-Based Action Anticipation

Mistake detection aims to identify deviations from expected action sequences during task execution, a process particularly crucial in egocentric procedural videos to enable reliable task monitoring and the development of intelligent assistive systems. However, existing approaches often depend on preconceived mistake scenarios and require extensive annotations, limiting their scalability and generalizability. To address these challenges, we propose an anticipation-based detection approach that identifies procedural action mistakes by contrasting anticipated actions with actual observed future actions, enabling flexible and robust detection without reliance on large predefined priors. Specifically, our approach integrates three key components: an action recognition module for precise action identification, an action captioning module for complementary semantic description, and an action anticipation module for predicting subsequent actions. Through the interplay of these components, it facilitates open-ended detection of diverse and unforeseen mistakes, effectively circumventing the reliance on preconceived mistake scenarios and extensive annotations. Notably, the final one employs a large language model to capture sequential and contextual dependencies across actions and narrations, leveraging instruction tuning and few-shot chain-of-thought prompts to enhance stepwise action prediction. Comprehensive experiments on widely used benchmarks not only demonstrate the effectiveness and generalizability of our approach but also offer in-depth ablation analysis that validates the role of each module.