Multi-Layer-Mesh: A Novel Topology and SDN-based Path Switching for Big Data Cluster Networks

Leandro Batista de Almeida, Damien Magoni, Philip Perry, Eduardo Cunha de Almeida, John Murphy, Anthony Ventresque

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Big Data technologies and tools have being used for the past decade to solve several scientific and industry problems, with Hadoop/YARN becoming the "de facto" standard for these applications, although other technologies run on top of it. As any other distributed application, those big data technologies rely heavily on the network infrastructure to read and move data from hundreds or thousands of cluster nodes. Although these technologies are based on reliable and efficient distributed algorithms, there are scenarios and conditions that can generate bottlenecks and inefficiencies, i.e., when a high number of concurrent users creates data access contention. In this paper, we propose a novel network topology called Multi-Layer-Mesh and a path switching algorithm based on SDN, that can increase the performance of a big data cluster while reducing the amount of utilized resources (network equipment), in turn reducing the energy and cooling consumption. A thorough simulation-based evaluation of our algorithms shows an average improvement in performance of 31.77% and an average decrease in resource utilization of 36.03% compared to a traditional Spine-Leaf topology, in the selected test scenarios.

Original languageEnglish
Title of host publicationICC 2019 - 2019 IEEE International Conference on Communications
Place of PublicationShanghai, China
PublisherIEEE Xplore
Pages1-7
Number of pages7
ISBN (Electronic)978-1-5386-8088-9
ISBN (Print)978-1-5386-8089-6
DOIs
Publication statusPublished - 15 Jul 2019
EventICC 2019 - 2019 IEEE International Conference on Communications - Shanghai, China
Duration: 20 May 201924 May 2019
https://ieeexplore.ieee.org/document/8761785/authors#authors

Publication series

Name ICC 2019 - 2019 IEEE International Conference on Communications (ICC)
PublisherIEEE
ISSN (Print)1550-3607
ISSN (Electronic)1938-1883

Conference

ConferenceICC 2019 - 2019 IEEE International Conference on Communications
Abbreviated titleICC 2019
CountryChina
CityShanghai
Period20/05/1924/05/19
Internet address

Keywords

  • Big data
  • Hadoop
  • network topology
  • SDN

Cite this