Enhancing an Emotion-Labelled Video Dataset with Pose Estimation Key-points for Face, Body and Hands

Research in automated emotion recognition from video has traditionally focused on facial expression as the primary indicator of emotional state. More recently, increasing attention has been given to incorporating features from body posture and gesture alongside facial expression to improve the performance of emotion recognition models. Although many emotion-labelled video datasets exist, few incorporate pose estimation data, and many of those that do are collected in lab-based environments.
To address this limitation, we introduce a new emotion-labelled pose estimation dataset derived from the Multimodal EmotionLines Dataset (MELD). This dataset combines MELD’s emotion annotations with speaker-specific, detailed pose estimation key-points from face, body and hands, extracted from the corresponding video recordings. The MELD-Keypoints v0.1 release contains pose estimation data extracted from 4,059 MELD video clips and 283,093 frame-level keypoint records. To date, 5,601 video clips have been processed through the complete extraction pipeline, of which 72.5% successfully produced speaker-specific pose data.