Integrating Space, Time, and Orientation in Spiking Neural Networks: A Case Study on Multimodal Brain Data Modelling 

Neelava Sengupta, Carolyn McNabb, Nikola Kasabov, Bruce Russel

Research output: Contribution to journalArticlepeer-review

19 Citations (Scopus)


Recent progress in a noninvasive brain data sampling technology has facilitated simultaneous sampling of multiple modalities of brain data, such as functional magnetic resonance imaging, electroencephalography, diffusion tensor
imaging, and so on. In spite of the potential benefits from integrating predictive modeling of multiple modality brain data, this area of research remains mostly unexplored due to a lack of methodological advancements. The difficulty in fusing multiple modalities of brain data within a single model lies in the heterogeneous temporal and spatial characteristics of the data sources. Recent advances in spiking neural network systems, however, provide the flexibility to incorporate multidimensional information within the model. This paper proposes a novel, unsupervised learning algorithm for fusing temporal, spatial, and orientation information in a spiking neural network architecture
that could potentially be used to understand and perform predictive modeling using multimodal data. The proposed algorithm is evaluated both qualitatively and quantitatively using synthetically generated data to characterize its behavior and its ability to utilize spatial, temporal, and orientation information within the
model. This leads to improved pattern recognition capabilities and performance along with robust interpretability of the brain data. Furthermore, a case study is presented, which aims to build a computational model that discriminates between people with schizophrenia who respond or do not respond to monotherapy with the antipsychotic clozapine.
Original languageEnglish
Pages (from-to)5249-5263
Number of pages15
JournalIEEE Transactions on Neural Networks and Learning Systems
Issue number11
Early online date13 Feb 2018
Publication statusPublished (in print/issue) - 16 Oct 2018


  • diffusion tensor imaging (DTI)
  • fMRI
  • multimodal brain data
  • NeuCube
  • response to treatment
  • schitzophrenia
  • spike-time dependent plasticity
  • spiking neural networks


Dive into the research topics of 'Integrating Space, Time, and Orientation in Spiking Neural Networks: A Case Study on Multimodal Brain Data Modelling '. Together they form a unique fingerprint.

Cite this