Artificial intelligence

Abstract: Video content has become integral to our daily lives, but poor video quality can significantly reduce viewers' experience and engagement. Various super-resolution methods are used to correct this, thereby reconstructing high-resolution videos from low-resolution ones. Two main categories of super-resolution methods exist traditional image processing and deep learning-based techniques. Deep learning-based techniques, such as Convolutional Neural Networks (CNNs), Generative Adversarial Networks (GANs), and Recurrent Neural Networks (RNNs), have shown great promise in enhancing video quality. The article discusses multiple adaptations of contemporary deep learning models to enhance video resolution. It also briefly explains the framework's design and implementation aspects. Lastly, the paper presents an overview and comparative analysis of the VSR techniques' efficiency on various benchmark datasets. At the same time, the paper describes potential challenges when choosing training sets; performance metrics, which can be used to compare different algorithms quantitatively. This work does not describe absolutely all existing VSR methods, but it is expected to contribute to the development of recent research in this field and potentially deepen our understanding of deep learning-based VSR methods, as well as stimulate further research in this area. In this work, new solutions for improving the performance of the methods are proposed, in particular, new quality metrics and datasets for model training. Overall, AI-based methods for VSR are becoming increasingly crucial with the rising demand for high-quality video content.

Keywords: Video quality, super-resolution, deep learning, single-image super-resolution, multi-image super-resolution.

References:

1. Harris J L 1964 Diffraction and resolving power Journal of the Optical Society of America 54 (7) 931-933.
2. Suresh S, Babu R V and Kim H J 2008 No-reference image quality assessment using modified extreme learning machine classifier Applied Soft Computing Journal 9 (2) 541-552.
3. Criminisi A, Perez P and Toyama K 2004 Region filling and object removal by exemplar-based image inpainting IEEE Transactions on Image Processing 13 (9) 1200-1212.
4. Freeman W T, Pasztor E C and Carmichael O T 2000 Learning low-level vision International Journal of Computer Vision 40 (1) 25-47.
5. Upscaling: R. Keys, "Cubic convolution interpolation for digital image processing," IEEE Transactions on Acoustics, Speech, and Signal Processing, vol. 29, no. 6, pp. 1153-1160, Dec. 1981.
6. Super Resolution: W. T. Freeman, T. R. Jones, and E. C. Pasztor, "Example-based super-resolution," IEEE Computer Graphics and Applications, vol. 22, no. 2, pp. 56-65, Mar./Apr. 2002.doi: 10.1109/38.988747
7. Deblurring: L. Yuan, J. Sun, L. Quan, and H.-Y. Shum, "Image deblurring with blurred/noisy image pairs," ACM Transactions on Graphics, vol. 26, no. 3, pp. 1-10, Aug. 2007. doi: 10.1145/1276377.1276390
8. Denoising: D. L. Donoho and I. M. Johnstone, "Ideal spatial adaptation by wavelet shrinkage," Biometrika, vol. 81, no. 3, pp. 425-455, Sep. 1994. doi: 10.1093/biomet/81.3.425
9. Yang, J., Wright, J., Huang, T., & Ma, Y. (2010). Image super-resolution via sparse representation. IEEE Transactions on Image Processing, 19(11), 2861-2873. doi: 10.1109/TIP.2010.2050625
10. Dong, C., Loy, C. C., He, K., & Tang, X. (2016). Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, 38(2), 295-307. doi: 10.1109/TPAMI.2015.2439281
11. Timofte, R., Agustsson, E., Van Gool, L., Yang, M. H., Zhang, L., Lim, B., ... & Lee, K. M. (2018). NTIRE 2017 challenge on single image super-resolution: Methods and results. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops (pp. 1110-1121). doi: .1109/CVPRW.2017.146
12. Wang, Y., Fan, Y., Yang, J., & Liu, Y. (2019). Deep recursive residual network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3158-3166). doi: 1109/CVPR.2019.00329
13. Huang, J. B., Singh, A., Ahuja, N., & Yang, M. H. (2015). Single image super-resolution from transformed self-exemplars. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5197-5206). doi: 10.1109/CVPR.2015.7299187
14. Hu, H., Wang, R., Xu, J., Sun, Z., & Zhang X. (2014). A survey on multi-image super-resolution algorithms. Signal Processing, 93(10), 2876-2894.
15. Daithankar, Mrunmayee & Ruikar, Dr. Sachin. (2020). Video Super Resolution by Neural Network: A Theoretical Aspect. Journal of Computational and Theoretical Nanoscience. 17. 4202-4206. doi: 10.1166/jctn.2020.9045.
16. The interpolation algorithm: Lanczos Interpolation, 2023. URL: https://ww2.lacan.upc.edu/doc/intel/ipp/ipp_manual/IPPI/ippi_appendices/ippi_appB_LanczosInterpolation.htm
17. Dr. Steve Arar. An Introduction to the Discrete Fourier Transform, July 20, 2017 . URL: https://www.allaboutcircuits.com/technical-articles/an-introduction-to-the-discrete-fourier-transform/
18. Runyuan Cai, Yue Ding, Hongtao Lu (2021). FreqNet: A Frequency-domain Image Super-Resolution Network with Dicrete Cosine Transform. URL: https://arxiv.org/abs/2111.10800v1
19. Bartlomiej Wronski, Ignacio Garcia-Dorado, Manfred Ernst, Damien Kelly, Michael Krainin, Chia-kai Liang, Marc Levoy, and Peyman Milanfar (2021). Handheld Multi-Frame Super-Resolution. URL: https://arxiv.org/pdf/1905.03277v2.pdf
20. Zhao, H., Wang, Y., & Cai, J. (2014). Recursive filtering based super-resolution for medical images. Journal of Medical Imaging and Health Informatics, 4(2), 254-261. doi: 10.1166/jmihi.2014.1222
21. Udupa, J. K., & Herman, G. T. (1986). A recursive algorithm for nonlinear digital filtering. IEEE Transactions on Acoustics, Speech, and Signal Processing, 34(2), 310-321. doi: 10.1109/TASSP.1986.1164786
22. Dong, C., Loy, C. C., He, K., & Tang, X. (2014). Learning a deep convolutional network for image super-resolution. In European Conference on Computer Vision (pp. 184-199). Springer, Cham.
23. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., & Fu, Y. (2018). Residual dense network for image super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 806-814). doi: 10.1109/CVPRW.2017.29
24. Kim, J., Kwon Lee, J., & Mu Lee, K. (2016). Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1646-1654).
25. Ledig, C., Theis, L., Huszr, F., Caballero, J., Cunningham, A., Acosta, A., ... & Shi, W. (2017). Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 4681-4690). doi: 10.1109/CVPR.2017.19
26. Chan KC, Wang X, Yu K, Dong C, Loy CC (2021b) BasicVSR: The search for essential components in video super-resolution and beyond. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp 4947–4956
27. Wang, X., Jiang, Y., Ma, S., Yang, Y., Sun, X., & Zhang, Q. (2021). IconVSR: A Generic Framework for Video Super-Resolution with Incomplete Supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(7), 2405-2420. doi: 10.1109/TPAMI.2020.3042388
28. Zhang, R., Isola, P., & Efros, A. A. (2018). "Learning a perception-based distance metric for image restoration." In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7606-7615. URL: https://arxiv.org/abs/1801.03924
29. Wang, Z., Bovik, A. C., Sheikh, H. R., & Simoncelli, E. P. (2004). "Image quality assessment: From error visibility to structural similarity." IEEE Transactions on Image Processing, 13(4), 600-612. doi: 10.1109/TIP.2003.819861
30. Wang X, Chan KCK, Yu K, Dong C, Loy CC (2019). EDVR: Video restoration with enhanced deformable convolutional networks. In: Proc. IEEE Conf. Comput. Vis. Pattern Recognit. Workshops, pp 1954–1963
31. Lucas, B. D., & Kanade, T. (1981). An iterative image registration technique with an application to stereo vision. In Proceedings of the 7th International Joint Conference on Artificial Intelligence-Volume 2 (pp. 674-679).
32. Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, Yichen Wei (2017). Deformable Convolutional Networks. Computer Vision and Pattern Recognition. URL: https://doi.org/10.48550/arXiv.1703.06211
33. Wang L, Guo Y, Lin Z, Deng X, An W (2019) Learning for video super-resolution through HR optical flow estimation. In: Proc. Asian Conf. Comput. Vis., pp 514–529
34. Xue T, Chen B, Wu J, Wei D, Freeman WT (2019) Video enhancement with task-oriented flow. Int J Comput Vis 127(8):1106–1125
35. Isobe T, Jia X, Gu S, Li S, Wang S, Tian Q (2020). Video super-resolution with recurrent structure dedetail network. In: Eur. Conf. Comput. Vis., pp 645–660
36. Li W, Tao X, Guo T, Qi L, Lu J, Jia J (2020) MuCAN: Multi-correspondence aggregation network for video super-resolution. In: Eur. Conf. Comput. Vis., pp 335–351
37. Renjie Liao, Xin Tao, Ruiyu Li, Ziyang Ma, Jiaya Jia (2015). Video Super-Resolution via Deep Draft-Ensemble Learning IEEE International Conference on Computer Vision. doi: 10.1109/ICCV.2015.68
38. MMEditing, C. (2022). MMEditing: OpenMMLab Image and Video Editing Toolbox (Version 0.13.0) [Computer software]. URL: https://github.com/open-mmlab/mmediting
39. Chan, Kelvin C.K. and Zhou, Shangchen and Xu, Xiangyu and Loy, Chen Change (2021). BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment. URL: https://doi.org/10.48550/arXiv.2104.13371
40. Wang, Jialu & Teng, Guowei & An, Ping. (2021). Video Super-Resolution Based on Generative Adversarial Network and Edge Enhancement. Electronics. 10. 459. doi: 10.3390/electronics10040459.
41. YouTube-8M Segments training dataset (2023). URL: https://research.google.com/youtube8m