Parameter Optimization on Spark for Particulate Matter Estimation

Zhenyu Yu; Zhibao Wang; Lu Bai; Liangfu Chen; Jinhua Tao

doi:10.1145/3456389.3456406

Parameter Optimization on Spark for Particulate Matter Estimation

Zhenyu Yu, Zhibao Wang, Lu Bai, Liangfu Chen, Jinhua Tao

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Citations (Scopus)

23 Downloads (Pure)

Abstract

With the rapid growth of remote sensing satellites, the volume of remote sensing data has been continuously increasing, which makes it necessary to utilize the big data platform for the rapid practical application of remote sensing inversion algorithms. This paper proposes an atmospheric remote sensing inversion processing method based on Spark. As a popular large-scale data processing framework, the memory-based iterable calculation model of Spark makes it suitable for the application of atmospheric remote sensing inversion. In this paper, we use the Spark computing framework to calculate the average value of the particulate matter in China over the past 10 years and the running time is much faster than the traditional single-node method. Furthermore, how Spark configuration parameters affect the performance of the task is explored. Different regression models in XGBoost are used to evaluate the performance of the parameters obtained by the parameter optimization algorithm in order to find the Spark optimal configuration parameters that meet the requirements.

Original language	English
Title of host publication	2021 Workshop on Algorithm and Big Data
Publisher	Association for Computing Machinery
Pages	9-13
Number of pages	5
ISBN (Electronic)	9781450389945
DOIs	https://doi.org/10.1145/3456389.3456406
Publication status	Published (in print/issue) - 12 Mar 2021

Bibliographical note

Funding Information:
This work was supported in part by TUOHAI special project 2020 from Bohai Rim Energy Research Institute of Northeast Petroleum University under Grant HBHZX202002 and project of Excellent and Middle-aged Scientific Research Innovation Team of Northeast Petroleum University under Grant KYCXTD201903.

Publisher Copyright:
© 2021 ACM.

Keywords

Particulate matter estimation
Spark
Parameter optimization
Performance prediction

Access to Document

10.1145/3456389.3456406

WABD174 - acceptedAuthor Accepted Manuscript, 117 KB

Cite this

@inproceedings{cfa7ae69fa47476aa0dded6343fbf633,

title = "Parameter Optimization on Spark for Particulate Matter Estimation",

abstract = "With the rapid growth of remote sensing satellites, the volume of remote sensing data has been continuously increasing, which makes it necessary to utilize the big data platform for the rapid practical application of remote sensing inversion algorithms. This paper proposes an atmospheric remote sensing inversion processing method based on Spark. As a popular large-scale data processing framework, the memory-based iterable calculation model of Spark makes it suitable for the application of atmospheric remote sensing inversion. In this paper, we use the Spark computing framework to calculate the average value of the particulate matter in China over the past 10 years and the running time is much faster than the traditional single-node method. Furthermore, how Spark configuration parameters affect the performance of the task is explored. Different regression models in XGBoost are used to evaluate the performance of the parameters obtained by the parameter optimization algorithm in order to find the Spark optimal configuration parameters that meet the requirements.",

keywords = "Particulate matter estimation, Spark, Parameter optimization, Performance prediction",

author = "Zhenyu Yu and Zhibao Wang and Lu Bai and Liangfu Chen and Jinhua Tao",

note = "Funding Information: This work was supported in part by TUOHAI special project 2020 from Bohai Rim Energy Research Institute of Northeast Petroleum University under Grant HBHZX202002 and project of Excellent and Middle-aged Scientific Research Innovation Team of Northeast Petroleum University under Grant KYCXTD201903. Publisher Copyright: {\textcopyright} 2021 ACM.",

year = "2021",

month = mar,

day = "12",

doi = "10.1145/3456389.3456406",

language = "English",

pages = "9--13",

booktitle = "2021 Workshop on Algorithm and Big Data",

publisher = "Association for Computing Machinery",

address = "United States",

}

TY - GEN

T1 - Parameter Optimization on Spark for Particulate Matter Estimation

AU - Yu, Zhenyu

AU - Wang, Zhibao

AU - Bai, Lu

AU - Chen, Liangfu

AU - Tao, Jinhua

N1 - Funding Information: This work was supported in part by TUOHAI special project 2020 from Bohai Rim Energy Research Institute of Northeast Petroleum University under Grant HBHZX202002 and project of Excellent and Middle-aged Scientific Research Innovation Team of Northeast Petroleum University under Grant KYCXTD201903. Publisher Copyright: © 2021 ACM.

PY - 2021/3/12

Y1 - 2021/3/12

N2 - With the rapid growth of remote sensing satellites, the volume of remote sensing data has been continuously increasing, which makes it necessary to utilize the big data platform for the rapid practical application of remote sensing inversion algorithms. This paper proposes an atmospheric remote sensing inversion processing method based on Spark. As a popular large-scale data processing framework, the memory-based iterable calculation model of Spark makes it suitable for the application of atmospheric remote sensing inversion. In this paper, we use the Spark computing framework to calculate the average value of the particulate matter in China over the past 10 years and the running time is much faster than the traditional single-node method. Furthermore, how Spark configuration parameters affect the performance of the task is explored. Different regression models in XGBoost are used to evaluate the performance of the parameters obtained by the parameter optimization algorithm in order to find the Spark optimal configuration parameters that meet the requirements.

AB - With the rapid growth of remote sensing satellites, the volume of remote sensing data has been continuously increasing, which makes it necessary to utilize the big data platform for the rapid practical application of remote sensing inversion algorithms. This paper proposes an atmospheric remote sensing inversion processing method based on Spark. As a popular large-scale data processing framework, the memory-based iterable calculation model of Spark makes it suitable for the application of atmospheric remote sensing inversion. In this paper, we use the Spark computing framework to calculate the average value of the particulate matter in China over the past 10 years and the running time is much faster than the traditional single-node method. Furthermore, how Spark configuration parameters affect the performance of the task is explored. Different regression models in XGBoost are used to evaluate the performance of the parameters obtained by the parameter optimization algorithm in order to find the Spark optimal configuration parameters that meet the requirements.

KW - Particulate matter estimation

KW - Spark

KW - Parameter optimization

KW - Performance prediction

UR - http://www.scopus.com/inward/record.url?scp=85108208493&partnerID=8YFLogxK

U2 - 10.1145/3456389.3456406

DO - 10.1145/3456389.3456406

M3 - Conference contribution

SP - 9

EP - 13

BT - 2021 Workshop on Algorithm and Big Data

PB - Association for Computing Machinery

ER -

Parameter Optimization on Spark for Particulate Matter Estimation

Abstract

Bibliographical note

Keywords

Access to Document

Other files and links

Fingerprint

Cite this