LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

, ,

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

26 Downloads (Pure)

Abstract

This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network
algorithms.
Original languageEnglish
Title of host publicationProceedings IEEE International Symposium on Circuits & Systems
PublisherIEEE
Pages1
Number of pages5
ISBN (Electronic)9781665451093
DOIs
Publication statusPublished (in print/issue) - 20 Feb 2023
EventIEEE International Symposium on Circuits & Systems - Monterey, United States
Duration: 21 May 202325 May 2023
https://iscas2023.org/

Publication series

NameIEEE International Symposium on Circuits and Systems

Conference

ConferenceIEEE International Symposium on Circuits & Systems
Country/TerritoryUnited States
CityMonterey
Period21/05/2325/05/23
Internet address

Bibliographical note

Funding Information:
This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a ”Formación de Personal Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program.

Funding Information:
This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a Formacion de Personal Universitario Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program

Publisher Copyright:
© 2023 IEEE.

Keywords

  • Neuromorphic dataset
  • sensory fusion
  • dynamic vision sensor
  • neuromorphic auditory sensor

Fingerprint

Dive into the research topics of 'LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading'. Together they form a unique fingerprint.

Cite this