Affective and Cognitive State Modelling within Human-Computer Interaction

Abstract

There is an opportunity and necessity to enhance computer systems with automated intelligence in order to permit natural and reliable interaction similar to human-human interaction. The recent availability of unobtrusive input modalities, such as eye trackers and web cameras, has enabled the viable real-time detection of user’s emotions and mental states. However, a
key challenge is utilising such modalities to enable a computer to actively interact with users based on their emotions and mental states. Consequently, the aim of this PhD is to develop a user model that will be utilised to infer a user’s emotional and cognitive state, which may be subsequently exploited to adapt the user experience in order to maximise the performance of the system and guarantee task completion. An underlying framework for adaptive Human-Computer Interaction has been developed that comprises a Perception Component responsible for capturing and modelling affective and cognitive states, and an Adaptation Component, which drives the appropriate adaptation
to the User Interface or User Experience. The research presented within this thesis primarily contributes to the Perception Component by probing the use of facial expressions as an input modality for computer systems. Additionally, eye-gaze tracking data has been investigated for assessing and modelling cognitive workload. This work presented herein involves data collection from unobtrusive input modalities, as well as the development of machine learning algorithms to build user models from the collected data. Subsequently, user affective states and cognitive load have been modelled and investigated within various computer-based tasks. Moreover, the credibility of using facial expressions for modelling affective states and pupil size for modelling cognitive load has been explored and discussed. Intrinsically, pupil size variation can be used to model
cognitive load during interaction with user interfaces, while facial expressions do not reflect the actual feelings of the user in that context.

Date of Award	Dec 2018
Original language	English
Sponsors	Ulster University
Supervisor	Raymond Bond (Supervisor), L Galway (Supervisor) & H. Wang (Supervisor)