Revisiting the Encoding of Satellite Image Time Series

Satellite Image Time Series (SITS) representation learning is complex due to high
spatiotemporal resolutions, irregular acquisition times, and intricate spatiotemporal interactions.
These challenges result in specialized neural network architectures tailored
for SITS analysis. The field has witnessed promising results achieved by pioneering researchers,
but transferring the latest advances or established paradigms from Computer
Vision (CV) to SITS is still highly challenging due to the existing suboptimal representation
learning framework. In this paper, we develop a novel perspective of SITS
processing as a direct set prediction problem, inspired by the recent trend in adopting
query-based transformer decoders to streamline the object detection or image segmentation
pipeline. We further propose to decompose the representation learning process of
SITS into three explicit steps: collect–update–distribute, which is computationally efficient
and suits for irregularly-sampled and asynchronous temporal satellite observations.
Facilitated by the unique reformulation, our proposed temporal learning backbone of
SITS, initially pre-trained on the resource efficient pixel-set format and then fine-tuned
on the downstream dense prediction tasks, has attained new state-of-the-art (SOTA) results
on the PASTIS benchmark dataset. Specifically, the clear separation between temporal
and spatial components in the semantic/panoptic segmentation pipeline of SITS
makes us leverage the latest advances in CV, such as the universal image segmentation
architecture, resulting in a noticeable 2.5 points increase in mIoU and 8.8 points increase
in PQ, respectively, compared to the best scores reported so far.