A Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech

Research output: Contribution to journalArticlepeer-review

11 Citations (Scopus)
93 Downloads (Pure)


OBJECTIVE: Brain-computer interfaces (BCI) studies are increasingly leveraging different attributes of multiple signal modalities simultaneously. Bimodal data acquisition protocols combining the temporal resolution of electroencephalography (EEG) with the spatial resolution of functional near-infrared spectroscopy (fNIRS) require novel approaches to decoding.

METHODS: We present an EEG-fNIRS Hybrid BCI that employs a new bimodal deep neural network architecture consisting of two convolutional sub-networks (subnets) to decode overt and imagined speech. Features from each subnet are fused before further feature extraction and classification. Nineteen participants performed overt and imagined speech in a novel cue-based paradigm enabling investigation of stimulus and linguistic effects on decoding.

RESULTS: Using the hybrid approach, classification accuracies (46.31% and 34.29% for overt and imagined speech, respectively (chance: 25%)) indicated a significant improvement on EEG used independently for imagined speech (p = 0.020) while tending towards significance for overt speech (p = 0.098). In comparison with fNIRS, significant improvements for both speech-types were achieved with bimodal decoding (p<0.001). There was a mean difference of ∼12.02% between overt and imagined speech with accuracies as high as 87.18% and 53%. Deeper subnets enhanced performance while stimulus effected overt and imagined speech in significantly different ways.

CONCLUSION: The bimodal approach was a significant improvement on unimodal results for several tasks. Results indicate the potential of multi-modal deep learning for enhancing neural signal decoding.

SIGNIFICANCE: This novel architecture can be used to enhance speech decoding from bimodal neural signals.

Original languageEnglish
Pages (from-to)1983-1994
Number of pages12
JournalIEEE Transactions on Biomedical Engineering
Issue number6
Early online date7 Dec 2021
Publication statusPublished (in print/issue) - 1 Jun 2022

Bibliographical note

Publisher Copyright:
© 1964-2012 IEEE.


  • electroencephalography
  • EEG
  • functional near-infrared spectroscopy
  • fNIRS
  • brain-computer interfaces
  • imagined speech
  • deep learning
  • bimodal deep learning
  • Electroencephalography
  • Decoding
  • Functional near-infrared spectroscopy
  • Task analysis
  • Deep learning
  • Production
  • Linguistics


Dive into the research topics of 'A Bimodal Deep Learning Architecture for EEG-fNIRS Decoding of Overt and Imagined Speech'. Together they form a unique fingerprint.

Cite this