Video Stabilization via Prediction with Time‐Series Network and Image Inpainting with Pyramid Fusion

• Published in: Chinese Journal of Electronics Vol. 30; no. 6; pp. 1103 - 1110
• Main Authors: Keyang, CHENG, Shichao, LI, Lan, RONG, Wenshan, WANG, Wenxi, SHI, Yongzhao, ZHAN
• Format: Journal Article
• Language: English
• Published: Published by the IET on behalf of the CIE 01-11-2021
• Subjects: Multi‐resolution fusion; Optimal seam; Time series network; Video inpainting; Video stabilization
• ISSN: 1022-4653; 2075-5597
• DOI: 10.1049/cje.2021.08.006

Due to the poor filling effect of the video image defect commonly used in the video stabilization field, the video is seemed still unstable after the image stabilization process, which seriously affects the visual effect. To solve this problem, we improve a video stabilization method based on time‐series network prediction and pyramid fusion restoration is proposed to optimize the visual effect after image stabilization. The flow of the proposed method is as follows: First, it is adaptive to determine whether the defect of the corresponding frame at the current time needs padding inpainting. Then, for the frame that needs to be inpainting, the frames generated before the current moment are sent to the model combining the convolutional neural networks and the gate recurrent unit to predict the part to be filled. Next the current defect image and the complete image to be filled are brought into the Laplacian pyramid reconstruction, and the improved weighted optimal suture is introduced for splicing during the fusion. Finally, the video frame is cut after reconstruction. The method is tested on a data set composed of videos commonly used in the field of video stabilization. The experimental results show that the average peak signal to noise ratio of the method is 2 to 5dB higher than that of the comparison algorithm, and the average structural similarity index is improved by about 2% to 7% compared with the contrast algorithm.