NeuroSense: Short-Term Emotion Recognition and Understanding Based on Spiking Neural Network Modelling of Spatio-Temporal EEG Patterns

Clarence Tan, Marko Sarlija, Nikola Kasabov

Research output: Contribution to journalArticlepeer-review

56 Citations (Scopus)
474 Downloads (Pure)


Emotion recognition still poses a challenge lying at the core of the rapidly growing area of affective computing and is crucial for establishing a successful human-computer interaction. Identification and understanding of emotions are achieved through various measures, such as subjective self-reports, face-tracking, voice analysis, gaze-tracking, as well as the analysis of autonomic and
central neurophysiological measurements. Current approaches to emotion recognition based on electroencephalography (EEG) mostly rely on various handcrafted features extracted over relatively long time windows of EEG during participants exposure to appropriate affective stimuli. In this paper, we present a short-term emotion recognition framework based on spiking neural network
(SNN) modelling of Spatio-temporal EEG patterns. Our method relies on EEG signal segmentation based on detection of short-term changes in facial landmarks, and as such includes no computation of handcrafted EEG features.
Differences between participants’ EEG properties are taken into account via subject-dependent spike encoding in the formulated subject-independent
emotion recognition task. We test our methods on the publicly available DEAP and MAHNOB-HCI databases due to the availability of both EEG and frontal face video data. Through an exhaustive hyperparameter optimisation strategy, we show that the proposed SNN-based representation of EEG spiking patterns provides valuable information for short- term emotion recognition.
The obtained accuracies are 78.97% and 79.39% in arousal classification, and 67.76% and 72.12% in valence classification, on the DEAP and MAHNOB-HCI datasets, respectively. Furthermore, through the application of a brain-inspired SNN model, this study provides novel insight and helps in the understanding of the neural mechanisms involved in emotional processing in the context of audiovisual stimuli, such as affective videos. The presented results encourage the use of the proposed EEG processing methodology as a complement to existing features and methods commonly used for EEG-based emotion recognition, especially for short-term arousal recognition.
Original languageEnglish
Article number23238
Pages (from-to)137-148
Number of pages12
Early online date7 Jan 2021
Publication statusPublished (in print/issue) - 28 Apr 2021

Bibliographical note

Publisher Copyright:
© 2021 Elsevier B.V.

Copyright 2021 Elsevier B.V., All rights reserved.


  • Affective computing
  • EEG
  • Emotion recognition
  • Event detection
  • Spiking neural networks


Dive into the research topics of 'NeuroSense: Short-Term Emotion Recognition and Understanding Based on Spiking Neural Network Modelling of Spatio-Temporal EEG Patterns'. Together they form a unique fingerprint.

Cite this