Spatio-Temporal Transformer Network for Video Restoration外文学习材料.pdfVIP

Spatio-Temporal Transformer Network for Video Restoration Tae Hyun Kim1,2, Mehdi S. M. Sajjadi1,3, Michael Hirsch1,4† , Bernhard Sch¨olkopf1 1 Max Planck Institute for Intelligent Systems, T¨ubingen, Germany {tkim,msajjadi,bs}@tue.mpg.de 2 Hanyang University, Seoul, Republic of Korea 3 Max Planck ETH Center for Learning Systems 4 Amazon Research, T¨ubingen, Germany hirsch@ Abstract. State-of-the-art video restoration methods integrate optical flow estimation networks to utilize temporal information. However, these networks typically consider only a pair of consecutive frames and hence are not capable of capturing long-range temporal dependencies and fall short of establishing correspondences across several timesteps. To alle- viate these problems, we propose a novel Spatio-temporal Transformer Network (STTN) which handles multiple frames at once and thereby manages to mitigate the common nuisance of occlusions in optical flow estimation. Our proposed STTN comprises a module that estimates op- tical flow in both space and time and a resampling layer that selectively warps target frames using the estimated flow. In our experiments, we demonstrate the efficiency of the proposed network and show state-of- the-art restoration results in video super-resolution and video deblurring. Keywords: Spatio-temporal transformer network, Spatio-temporal flow, Spatio-temporal sampler, Video super-resolution, Video deblurring 1 Introduction Motion estimation via dense optical flow is a crucial component in video pro- cessing including restoration tasks su