Optimality and limitations of audio-visual integration for cognitive systems

William Boyce, Anthony Lindsay, Arkady Zgonnikov, Inaki Rano, KongFatt Wong-Lin

Research output: Contribution to journalReview articlepeer-review

3 Citations (Scopus)
58 Downloads (Pure)


Multimodal integration is an important process in perceptual decision-making. In humans, this process has often been shown to be statistically optimal, or near optimal: sensory information is combined in a fashion that minimizes the average error in perceptual representation of stimuli. However, sometimes there are costs that come with the optimization, manifesting as illusory percepts. We review audio-visual facilitations and illusions that are products of multisensory integration, and the computational models that account for these phenomena. In particular, the same optimal computational model can lead to illusory percepts, and we suggest that more studies should be needed to detect and mitigate these illusions, as artifacts in artificial cognitive systems. We provide cautionary considerations when designing artificial cognitive systems with the view of avoiding such artifacts. Finally, we suggest avenues of research toward solutions to potential pitfalls in system design. We conclude that detailed understanding of multisensory integration and the mechanisms behind audio-visual illusions can benefit the design of artificial cognitive systems. [Abstract copyright: Copyright © 2020 Boyce, Lindsay, Zgonnikov, Rañó and Wong-Lin.]
Original languageEnglish
Article number94
Number of pages17
JournalFrontiers in Robotics and AI
Publication statusPublished (in print/issue) - 17 Jul 2020

Bibliographical note

Funding Information:
The authors are grateful to Shufan Yang for helpful discussions on the computational modeling. Funding. This research was supported by the Northern Ireland Functional Brain Mapping Project (1303/101154803), funded by Invest NI and the University of Ulster (KW-L), The Royal Society (KW-L and IR), and ASUR (1014-C4-Ph1-071) (KW-L, IR, and AL).

Publisher Copyright:
© Copyright © 2020 Boyce, Lindsay, Zgonnikov, Rañó and Wong-Lin.

Copyright 2020 Elsevier B.V., All rights reserved.


  • Bayesian integration
  • audio-visual illusions
  • cognitive systems
  • multi-modal processing
  • multisensory integration
  • optimality


Dive into the research topics of 'Optimality and limitations of audio-visual integration for cognitive systems'. Together they form a unique fingerprint.

Cite this