Spontaneous facial micro expression recognition and analysis using varying resolutions

Abstract

During the early years of facial expression research, works have mostly employed macro expressions which are easily identifiable. In contrast, in recent years utilizing facial micro expression has gained more acknowledgement in facial analysis due to stronger genuineness of its attributes. Subsequently, emotion analysis through facial micro expression has higher acceptability especially in psychology, autism, pain assessment, security, criminal investigations, and similar circumstances that demand critical decision making. Owing to its cross-discipline application, today micro expression analysis using facial images remains an active research field. Due to extreme minuteness of these expressions, they are often missed during observations however, studies show with the introduction of computer vison and machine/deep learning algorithms they have a higher chance of being identified. Therefore, the focus of this thesis is to conduct thorough investigations and design novel approaches for micro expression analysis employing suitable methods.

Most of the existing literature has overlooked the phase information while describing image patterns specifically to achieve micro expression recognition. Consequently, this thesis investigates the effectiveness of employing phase information for micro expression analysis. Furthermore, interpolation and video magnification are also introduced in later experiments to aid the extraction method. Additionally, the literature also highlighted the absence of adequate work examining the impact of resolutions and image quality for micro expression analysis. Therefore, the aim of this thesis is to explore micro expression to design a pipeline capable of boosting the expression recognition performance. Moreover, this thesis establishes threefold contributions to address the research gap: firstly, a pipeline that exploits interpolation, phase and temporal information in a non-cross database environment is utilized. Secondly, influence of video magnification is examined to improve expression recognition within this pipeline. Third, a novel pipeline to employ low quality micro expression images is developed by reconstructing such images using deep learning and generative adversarial networks.

In this thesis, to verify the suitability of combining phase, temporal, and magnification methods for micro expression, experiments are conducted on seven spontaneous micro expression databases. Results obtained clearly indicate the approach is as competitive as any other existing traditional methods. Furthermore, the experimental results obtained after introducing deep learning and generative adversarial networks into the second novel pipeline clearly highlight the significance of image reconstruction in achieving recognition boost even when the quality of input is compromised. Therefore, this thesis establishes significant progress towards the development of techniques for micro expression recognition that can be collaborated with medical/security and similar fields to assist in identifying vital cues.

Date of Award	Sept 2022
Original language	English
Supervisor	Pratheepan Yogarajah (Supervisor), Laurence Taggart (Supervisor) & Sonya Coleman (Supervisor)