LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

doi:10.1109/ISCAS46773.2023.10181685

LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

, ,

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

85 Downloads (Pure)

Abstract

This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network
algorithms.

Original language	English
Title of host publication	Proceedings IEEE International Symposium on Circuits & Systems
Publisher	IEEE
Pages	1
Number of pages	5
ISBN (Electronic)	9781665451093
DOIs	https://doi.org/10.1109/ISCAS46773.2023.10181685
Publication status	Published (in print/issue) - 20 Feb 2023
Event	IEEE International Symposium on Circuits & Systems - Monterey, United States Duration: 21 May 2023 → 25 May 2023 https://iscas2023.org/

Publication series

Name	IEEE International Symposium on Circuits and Systems

Conference

Conference	IEEE International Symposium on Circuits & Systems
Country/Territory	United States
City	Monterey
Period	21/05/23 → 25/05/23
Internet address	https://iscas2023.org/

Bibliographical note

Funding Information:
This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a ”Formación de Personal Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program.

Funding Information:
This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a Formacion de Personal Universitario Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program

Publisher Copyright:
© 2023 IEEE.

Funding

Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a ”Formación de Personal Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program. Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a Formacion de Personal Universitario Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program Publisher Copyright: © 2023 IEEE.

Keywords

Neuromorphic dataset
sensory fusion
dynamic vision sensor
neuromorphic auditory sensor

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/ISCAS46773.2023.10181685

ISCAS_2023___LIPSFUS_CNN_based_Audio_and_Vision (1)Author Accepted Manuscript -, 2.12 MB

Cite this

@inproceedings{6e56f37f8bda49b4b2c60d667a4f0bcc,

title = "LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading",

abstract = "This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms. ",

keywords = "Neuromorphic dataset, sensory fusion, dynamic vision sensor, neuromorphic auditory sensor",

author = "Aqib Javed and Jim Harkin",

note = "Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a ”Formaci{\'o}n de Personal Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program. Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a Formacion de Personal Universitario Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program Publisher Copyright: {\textcopyright} 2023 IEEE.; IEEE International Symposium on Circuits \& Systems ; Conference date: 21-05-2023 Through 25-05-2023",

year = "2023",

month = feb,

day = "20",

doi = "10.1109/ISCAS46773.2023.10181685",

language = "English",

series = "IEEE International Symposium on Circuits and Systems",

publisher = "IEEE",

pages = "1",

booktitle = "Proceedings IEEE International Symposium on Circuits \& Systems",

address = "United States",

url = "https://iscas2023.org/",

}

TY - GEN

T1 - LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

AU - Javed, Aqib

AU - Harkin, Jim

N1 - Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a ”Formación de Personal Universitario” Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program. Funding Information: This research was partially supported by the Spanish grants MINDROB (PID2019-105556GB-C33/AEI/10.13039/501100011033) and SMALL (PCI2019-111841-2/AEI/10.1309/501100011033) projects. E. P.-F. was supported by a Formacion de Personal Universitario Scholarship from the Spanish Ministry of Education, Culture and Sport. Authors thank the Ulster University people and their HPC. A. L-B thanks the Salvador de Madariaga mobility program Publisher Copyright: © 2023 IEEE.

PY - 2023/2/20

Y1 - 2023/2/20

N2 - This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

AB - This paper presents a sensory fusion neuromorphic dataset collected with precise temporal synchronization using a set of Address-Event-Representation sensors and tools. The target application is the lip reading of several keywords for different machine learning applications, such as digits, robotic commands, and auxiliary rich phonetic short words. The dataset is enlarged with a spiking version of an audio-visual lip reading dataset collected with frame-based cameras. LIPSFUS is publicly available and it has been validated with a deep learning architecture for audio and visual classification. It is intended for sensory fusion architectures based on both artificial and spiking neural network algorithms.

KW - Neuromorphic dataset

KW - sensory fusion

KW - dynamic vision sensor

KW - neuromorphic auditory sensor

UR - https://www.scopus.com/pages/publications/85167695520

U2 - 10.1109/ISCAS46773.2023.10181685

DO - 10.1109/ISCAS46773.2023.10181685

M3 - Conference contribution

T3 - IEEE International Symposium on Circuits and Systems

SP - 1

BT - Proceedings IEEE International Symposium on Circuits & Systems

PB - IEEE

T2 - IEEE International Symposium on Circuits & Systems

Y2 - 21 May 2023 through 25 May 2023

ER -

LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Intelligent hotspot prediction in networks-on-chip using spiking neural networks

Cite this

LIPSFUS: A neuromorphic dataset for audio-visual sensory fusion of lip reading

Abstract

Publication series

Conference

Bibliographical note

Funding

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Student theses

Intelligent hotspot prediction in networks-on-chip using spiking neural networks

Cite this