TY - JOUR
T1 - Emotion recognition with convolutional neural network and EEG-based EFDMs
AU - Wang, Fei
AU - Wu, Shichao
AU - Zhang, Weiwei
AU - Xu, Zongfeng
AU - Zhang, Yahui
AU - Wu, Chengdong
AU - Coleman, Sonya
PY - 2020/9/30
Y1 - 2020/9/30
N2 - Electroencephalogram (EEG), as a direct response to brain activity, can be used to detect mental states and physical conditions. Among various EEG-based emotion recognition studies, due to the non-linear, non-stationary and the individual difference of EEG signals, traditional recognition methods still have the disadvantages of complicated feature extraction and low recognition rates. Thus, this paper first proposes a novel concept of electrode-frequency distribution maps (EFDMs) with short-time Fourier transform (STFT). Residual block based deep convolutional neural network (CNN) is proposed for automatic feature extraction and emotion classification with EFDMs. Aim at the shortcomings of the small amount of EEG samples and the challenge of differences in individual emotions, which makes it difficult to construct a universal model, this paper proposes a cross-datasets emotion recognition method of deep model transfer learning. Experiments carried out on two publicly available datasets. The proposed method achieved an average classification score of 90.59% based on a short length of EEG data on SEED, which is 4.51% higher than the baseline method. Then, the pre-trained model was applied to DEAP through deep model transfer learning with a few samples, resulted an average accuracy of 82.84%. Finally, this paper adopts the gradient weighted class activation mapping (Grad-CAM) to get a glimpse of what features the CNN has learned during training from EFDMs and concludes that the high frequency bands are more favorable for emotion recognition.
AB - Electroencephalogram (EEG), as a direct response to brain activity, can be used to detect mental states and physical conditions. Among various EEG-based emotion recognition studies, due to the non-linear, non-stationary and the individual difference of EEG signals, traditional recognition methods still have the disadvantages of complicated feature extraction and low recognition rates. Thus, this paper first proposes a novel concept of electrode-frequency distribution maps (EFDMs) with short-time Fourier transform (STFT). Residual block based deep convolutional neural network (CNN) is proposed for automatic feature extraction and emotion classification with EFDMs. Aim at the shortcomings of the small amount of EEG samples and the challenge of differences in individual emotions, which makes it difficult to construct a universal model, this paper proposes a cross-datasets emotion recognition method of deep model transfer learning. Experiments carried out on two publicly available datasets. The proposed method achieved an average classification score of 90.59% based on a short length of EEG data on SEED, which is 4.51% higher than the baseline method. Then, the pre-trained model was applied to DEAP through deep model transfer learning with a few samples, resulted an average accuracy of 82.84%. Finally, this paper adopts the gradient weighted class activation mapping (Grad-CAM) to get a glimpse of what features the CNN has learned during training from EFDMs and concludes that the high frequency bands are more favorable for emotion recognition.
KW - Convolutional neural network
KW - Electrode-frequency distribution maps
KW - Electroencephalogram
KW - Emotion recognition
KW - Gradient-weighted class activation mapping
UR - https://pure.ulster.ac.uk/en/publications/emotion-recognition-with-convolutional-neural-network-and-eeg-bas
UR - http://www.scopus.com/inward/record.url?scp=85085879045&partnerID=8YFLogxK
U2 - 10.1016/j.neuropsychologia.2020.107506
DO - 10.1016/j.neuropsychologia.2020.107506
M3 - Article
C2 - 32497532
SN - 0028-3932
VL - 146
JO - Neuropsychologia
JF - Neuropsychologia
M1 - 107506
ER -