Paradigm and AI design choices for decoding imagined speech from electroencephalography

Abstract

Brain-computer interfaces (BCI) enabling imagined speech communication would be transformative for sufferers of neuro-pathologies such as Amyotrophic Lateral Sclerosis (ALS) resulting in Locked-In Syndrome. However, several critical obstacles impede development of a BCI predicated on imagined speech. Pressing concerns include limitations in current understanding of imagined speech and its relationship to overt speech, difficulties in designing robust protocols to study imagined speech, and development of sufficiently accurate algorithms to make speech decoding practical. This thesis tackles each of these obstacles with the aim of advancing the field of speech decoding for non-invasive BCIs. The research presented is contextualized by a synthesis of the literature in two key areas. The first focuses on neurolinguistics research on imagined speech while the second critically evaluates the various paradigms used in speech decoding experiments. Relative decoding potential of machine- and deep-learning (ML; DL) techniques for imagined speech Electroencephalography (EEG) is then investigated. Results show superior decoding performance for DL over traditional ML methods. However, several avenues of analysis, including the effect of different stimuli and linguistic features, were limited by the experimental design. These limitations informed a novel experimental procedure consisting of three distinct methods for presenting cues to participants: text, audio and image stimuli, and words selected based on two different linguistic criteria: semantic relationship to parts of the body and the presence or absence of a syntactic modification. In addition, the design enabled comparative analysis between overt and imagined speech. Results demonstrated significant differences in decoding dependent on the type of stimulus. For imagined speech, the use of images to cue tasks resulted in significantly superior decoding in comparison with other stimuli. In contrast, linguistic categories did not affect decoding. All analyses pointed to differences between overt and imagined speech, indicating that caution must be applied when using one as a proxy for the other. Finally, a novel bimodal DL architecture is presented to facilitate decoding of imagined speech from simultaneously recorded EEG and functional Near-Infrared Spectroscopy (fNIRS). The design consists of two sub-networks trained to extract features from the data before being fused within a main network for classification. Operation of the network was validated on overt and imagined speech and suggested future potential for non-invasive bimodal decoding of speech. While demonstrating potential for decoding imagined speech, this thesis highlights the impact of different aspects of experimental paradigms on speech decoding and shows that robust and novel experimental procedures are critical for future efforts to understand and progress imagined speech decoding.

Date of Award	Jun 2021
Original language	English
Sponsors	Department for the Economy
Supervisor	Raffaella Folli (Supervisor) & Damien Coyle (Supervisor)