With the rapid growth of remote sensing satellites, the volume of remote sensing data has been continuously increasing, which makes it necessary to utilize the big data platform for the rapid practical application of remote sensing inversion algorithms. This paper proposes an atmospheric remote sensing inversion processing method based on Spark. As a popular large-scale data processing framework, the memory-based iterable calculation model of Spark makes it suitable for the application of atmospheric remote sensing inversion. In this paper, we use the Spark computing framework to calculate the average value of the particulate matter in China over the past 10 years and the running time is much faster than the traditional single-node method. Furthermore, how Spark configuration parameters affect the performance of the task is explored. Different regression models in XGBoost are used to evaluate the performance of the parameters obtained by the parameter optimization algorithm in order to find the Spark optimal configuration parameters that meet the requirements.
|Title of host publication||2021 Workshop on Algorithm and Big Data|
|Publisher||Association for Computing Machinery|
|Number of pages||5|
|Publication status||Published (in print/issue) - 12 Mar 2021|
Bibliographical noteFunding Information:
This work was supported in part by TUOHAI special project 2020 from Bohai Rim Energy Research Institute of Northeast Petroleum University under Grant HBHZX202002 and project of Excellent and Middle-aged Scientific Research Innovation Team of Northeast Petroleum University under Grant KYCXTD201903.
© 2021 ACM.
- Particulate matter estimation
- Parameter optimization
- Performance prediction