Integration of text and image analysis for flood event image recognition

Min Jing, Scotney Bryan, Sonya Coleman, T.Martin McGinnity, Xiubo Zhang, Stephen Kelly, Khurshid Ahmad, Antje Schlaf, Sabine Gr¨under-Fahrer, Gerhard Heyer

Research output: Chapter in Book/Report/Conference proceedingConference contribution

5 Citations (Scopus)

Abstract

Flood event monitoring plays an important role for emergency management. With the fast growth of social media, a large number of images and videos are uploaded and searched on the internet during disasters, which can be used as “sensors” for improving efficiency of emergency management. This work proposes a novel framework in which the rich information available from social media is incorporated with image analysis to enhance image retrieval for disaster management. The text associated with images of flooding events was used to extract prominent words associated with flooding. The image features are represented by a histogram of visual words obtained using the Bag-of-Words (BoW) model. The text and image analysis are integrated at the feature level, in which the text features are conjoined directly with image features. The proposed approach was evaluated based on two flood event corpuses obtained from the US Federal Emergency Management Agency media library and public Facebook pages and groups related to flood and flood aid (in German). The experimental results demonstrate the improved performance of image recognition after incorporating the text features, which suggests the potential to enhance the efficiency of emergency management.
LanguageEnglish
Title of host publicationUnknown Host Publication
Number of pages6
DOIs
Publication statusAccepted/In press - 21 Jun 2016
EventSignals and Systems Conference (ISSC), 2016 27th Irish -
Duration: 21 Jun 2016 → …

Conference

ConferenceSignals and Systems Conference (ISSC), 2016 27th Irish
Period21/06/16 → …

Fingerprint

image analysis
flooding
disaster management
histogram
aid
disaster
sensor
monitoring
social media

Keywords

  • flood event image recognition
  • social media analysis
  • multimodal data fusion
  • emergency management

Cite this

Jing, Min ; Bryan, Scotney ; Coleman, Sonya ; McGinnity, T.Martin ; Zhang, Xiubo ; Kelly, Stephen ; Ahmad, Khurshid ; Schlaf, Antje ; Gr¨under-Fahrer, Sabine ; Heyer, Gerhard. / Integration of text and image analysis for flood event image recognition. Unknown Host Publication. 2016.
@inproceedings{7045b77e0f49400a9a45165fc7d64058,
title = "Integration of text and image analysis for flood event image recognition",
abstract = "Flood event monitoring plays an important role for emergency management. With the fast growth of social media, a large number of images and videos are uploaded and searched on the internet during disasters, which can be used as “sensors” for improving efficiency of emergency management. This work proposes a novel framework in which the rich information available from social media is incorporated with image analysis to enhance image retrieval for disaster management. The text associated with images of flooding events was used to extract prominent words associated with flooding. The image features are represented by a histogram of visual words obtained using the Bag-of-Words (BoW) model. The text and image analysis are integrated at the feature level, in which the text features are conjoined directly with image features. The proposed approach was evaluated based on two flood event corpuses obtained from the US Federal Emergency Management Agency media library and public Facebook pages and groups related to flood and flood aid (in German). The experimental results demonstrate the improved performance of image recognition after incorporating the text features, which suggests the potential to enhance the efficiency of emergency management.",
keywords = "flood event image recognition, social media analysis, multimodal data fusion, emergency management",
author = "Min Jing and Scotney Bryan and Sonya Coleman and T.Martin McGinnity and Xiubo Zhang and Stephen Kelly and Khurshid Ahmad and Antje Schlaf and Sabine Gr¨under-Fahrer and Gerhard Heyer",
year = "2016",
month = "6",
day = "21",
doi = "10.1109/ISSC.2016.7528454",
language = "English",
isbn = "978-1-5090-3409-3",
booktitle = "Unknown Host Publication",

}

Jing, M, Bryan, S, Coleman, S, McGinnity, TM, Zhang, X, Kelly, S, Ahmad, K, Schlaf, A, Gr¨under-Fahrer, S & Heyer, G 2016, Integration of text and image analysis for flood event image recognition. in Unknown Host Publication. Signals and Systems Conference (ISSC), 2016 27th Irish, 21/06/16. https://doi.org/10.1109/ISSC.2016.7528454

Integration of text and image analysis for flood event image recognition. / Jing, Min; Bryan, Scotney; Coleman, Sonya; McGinnity, T.Martin; Zhang, Xiubo; Kelly, Stephen; Ahmad, Khurshid; Schlaf, Antje; Gr¨under-Fahrer, Sabine; Heyer, Gerhard.

Unknown Host Publication. 2016.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

TY - GEN

T1 - Integration of text and image analysis for flood event image recognition

AU - Jing, Min

AU - Bryan, Scotney

AU - Coleman, Sonya

AU - McGinnity, T.Martin

AU - Zhang, Xiubo

AU - Kelly, Stephen

AU - Ahmad, Khurshid

AU - Schlaf, Antje

AU - Gr¨under-Fahrer, Sabine

AU - Heyer, Gerhard

PY - 2016/6/21

Y1 - 2016/6/21

N2 - Flood event monitoring plays an important role for emergency management. With the fast growth of social media, a large number of images and videos are uploaded and searched on the internet during disasters, which can be used as “sensors” for improving efficiency of emergency management. This work proposes a novel framework in which the rich information available from social media is incorporated with image analysis to enhance image retrieval for disaster management. The text associated with images of flooding events was used to extract prominent words associated with flooding. The image features are represented by a histogram of visual words obtained using the Bag-of-Words (BoW) model. The text and image analysis are integrated at the feature level, in which the text features are conjoined directly with image features. The proposed approach was evaluated based on two flood event corpuses obtained from the US Federal Emergency Management Agency media library and public Facebook pages and groups related to flood and flood aid (in German). The experimental results demonstrate the improved performance of image recognition after incorporating the text features, which suggests the potential to enhance the efficiency of emergency management.

AB - Flood event monitoring plays an important role for emergency management. With the fast growth of social media, a large number of images and videos are uploaded and searched on the internet during disasters, which can be used as “sensors” for improving efficiency of emergency management. This work proposes a novel framework in which the rich information available from social media is incorporated with image analysis to enhance image retrieval for disaster management. The text associated with images of flooding events was used to extract prominent words associated with flooding. The image features are represented by a histogram of visual words obtained using the Bag-of-Words (BoW) model. The text and image analysis are integrated at the feature level, in which the text features are conjoined directly with image features. The proposed approach was evaluated based on two flood event corpuses obtained from the US Federal Emergency Management Agency media library and public Facebook pages and groups related to flood and flood aid (in German). The experimental results demonstrate the improved performance of image recognition after incorporating the text features, which suggests the potential to enhance the efficiency of emergency management.

KW - flood event image recognition

KW - social media analysis

KW - multimodal data fusion

KW - emergency management

U2 - 10.1109/ISSC.2016.7528454

DO - 10.1109/ISSC.2016.7528454

M3 - Conference contribution

SN - 978-1-5090-3409-3

BT - Unknown Host Publication

ER -