Video Snow Removal Based on Self-adaptation Snow Detection and Patch-based Gaussian Mixture Model

Video desnowing has become a challenging research topic in computer vision in recent years. Existing methods cannot remove most of the snow in heavy snow scenes and will cause the deformation of moving objects when used for snowy videos that include moving objects. These methods have poor generalizability, exhibiting poor performance when removing snow from videos with different resolutions. In this paper, we propose a new video snow removal method based on self-adaptation snow detection and a patch-based Gaussian mixture model (VSRSG). First, an optical flow estimation method and a support vector machine (SVM) are used to detect snowflakes, and a self-adaptation threshold process is used to remove dense snowflakes in the snowflake detection map to obtain a sparse snowflake detection map. Then, a patch-based Gaussian mixture model (PBGMM), which can remove moving objects and both sparse and dense snowflakes from videos and restore a clear video background, is applied for background modeling. A Markov random field (MRF) and self-adaptation threshold processing are used to extract sparse snowflakes and moving objects and combine them with the background to form an input video without dense snowflakes. Finally, a similar block matching method is employed to fill in the detected snowflake pixels with the information from adjacent frames to remove the sparse snowflakes in the near range. This method can also remove snowflakes in front of moving objects. Experiments show that the proposed method can simultaneously remove sparse snowflakes, dense snowflakes and snowflakes in front of moving objects and outperforms other state-of-the-art methods.