Russian Federation
Russian Federation
GRNTI 50.07 Теоретические основы вычислительной техники
BBK 3297 Вычислительная техника
The field of automatic image inpainting has progressed rapidly in recent years, but no one has yet proposed a standard method of evaluating algorithms. This absence is due to the problem’s challenging nature: image-inpainting algorithms strive for realism in the resulting images, but realism is a subjective concept intrinsic to human perception. Existing objective image-quality metrics provide a poor approximation of what humans consider more or less realistic. To improve the situation and to better organize both prior and future research in this field, we conducted a subjective comparison of nine state-of-the-art inpainting algorithms and propose objective quality metrics that exhibit high correlation with the results of our comparison.
image inpainting, objective quality metric, quality perception, subjective evaluation, deep learning
1. https://research.adobe.com/project/contentawarefill/.
2. J. H. Bappy, A. K. RoyChowdhury, J. Bunk,L. Nataraj, and B. S. Manjunath. Exploiting spatialstructure for localizing manipulated image regions. InThe IEEE International Conference on Computer Vision (ICCV), Oct 2017.
3. R. A. Bradley and M. E. Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 39(3/4):324-345, 1952.
4. F. Chollet. Xception: Deep learning with depthwiseseparable convolutions. In The IEEE Conference onComputer Vision and Pattern Recognition (CVPR),July 2017.
5. A. Criminisi, P. Pérez, and K. Toyama. Region filling and object removal by exemplarbased image inpainting. IEEE Transactions on Image Processing,13(9):1200-1212, 2004.
6. J. Deng, W. Dong, R. Socher, L.J. Li, K. Li, andL. FeiFei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computervision and pattern recognition, pages 248-255, 2009.
7. K. He and J. Sun. Statistics of patch offsets for imagecompletion. In European Conference on ComputerVision, pages 16-29. Springer, 2012.
8. K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In The IEEE Conference on Computer Vision and Pattern Recognition(CVPR), June 2016.
9. K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. In European conference on computer vision, pages 630-645, 2016.
10. S. Iizuka, E. SimoSerra, and H. Ishikawa. Globallyand locally consistent image completion. ACM Transactions on Graphics (ToG), 36(4):107, 2017.
11. J. Johnson, A. Alahi, and L. FeiFei. Perceptual lossesfor realtime style transfer and superresolution. InEuropean conference on computer vision, pages 694-711. Springer, 2016.
12. H. Li, G. Li, L. Lin, H. Yu, and Y. Yu. Contextawaresemantic inpainting. IEEE Transactions on Cybernetics, 2018.
13. H. Li, W. Luo, X. Qiu, and J. Huang. Imageforgery localization via integrating tampering possibility maps. IEEE Transactions on InformationForensics and Security, 12(5):1240-1252, 2017.
14. T.Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona,D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoftcoco: Common objects in context. In European conference on computer vision, pages 740-755, 2014.
15. C. Liu, B. Zoph, M. Neumann, J. Shlens, W. Hua,L.J. Li, L. FeiFei, A. Yuille, J. Huang, and K. Murphy. Progressive neural architecture search. In TheEuropean Conference on Computer Vision (ECCV),September 2018.
16. G. Liu, F. A. Reda, K. J. Shih, T.C. Wang, A. Tao,and B. Catanzaro. Image inpainting for irregularholes using partial convolutions. In The EuropeanConference on Computer Vision (ECCV), 2018.
17. P. Liu, X. Qi, P. He, Y. Li, M. R. Lyu, and I. King.Semantically consistent image completion with finegrained details. arXiv preprint arXiv:1711.09345,2017.
18. T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida.Spectral normalization for generative adversarial networks. arXiv preprint arXiv:1802.05957, 2018.
19. D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, andA. A. Efros. Context encoders: Feature learning byinpainting. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
20. C.M. Pun, X.C. Yuan, and X.L. Bi. Image forgerydetection using adaptive oversegmentation and feature point matching. IEEE Transactions on Information Forensics and Security, 10(8):1705-1716, 2015.
21. R. Salloum, Y. Ren, and C.C. J. Kuo. Image splicinglocalization using a multitask fully convolutional network (mfcn). Journal of Visual Communication andImage Representation, 51:201-209, 2018.
22. K. Simonyan and A. Zisserman. Very deep convolutional networks for largescale image recognition.arXiv preprint arXiv:1409.1556, 2014.
23. Y. Song, C. Yang, Z. Lin, H. Li, Q. Huang, and C. J.Kuo. Image inpainting using multiscale feature image translation. arXiv preprint arXiv:1711.08590, 2,2017.
24. C. Szegedy, S. Ioffe, V. Vanhoucke, and A. A.Alemi. Inceptionv4, inceptionresnet and the impactof residual connections on learning. In ThirtyFirstAAAI Conference on Artificial Intelligence, 2017.
25. C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, andZ. Wojna. Rethinking the inception architecture forcomputer vision. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June2016.
26. A. Telea. An image inpainting technique based onthe fast marching method. Journal of Graphics Tools,9(1):23-34, 2004.
27. D. Ulyanov, A. Vedaldi, and V. Lempitsky. Deepimage prior. In The IEEE Conference on ComputerVision and Pattern Recognition (CVPR), June 2018.
28. Z. Wang, A. C. Bovik, H. R. Sheikh, E. P. Simoncelli,et al. Image quality assessment: from error visibilityto structural similarity. IEEE transactions on imageprocessing, 13(4):600-612, 2004.
29. Z. Yan, X. Li, M. Li, W. Zuo, and S. Shan. Shiftnet: Image inpainting via deep feature rearrangement.In The European Conference on Computer Vision(ECCV), September 2018.
30. C. Yang, X. Lu, Z. Lin, E. Shechtman, O. Wang, andH. Li. Highresolution image inpainting using multiscale neural patch synthesis. In The IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR),July 2017.
31. J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S.Huang. Freeform image inpainting with gated convolution. arXiv preprint arXiv:1806.03589, 2018.
32. J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S.Huang. Generative image inpainting with contextualattention. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
33. R. Zhang, P. Isola, A. A. Efros, E. Shechtman, andO. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In The IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR),June 2018.
34. P. Zhou, X. Han, V. I. Morariu, and L. S. Davis.Learning rich features for image manipulation detection. In The IEEE Conference on Computer Visionand Pattern Recognition (CVPR), June 2018.
35. X. Zhu, Y. Qian, X. Zhao, B. Sun, and Y. Sun. Adeep learning approach to patchbased image inpainting forensics. Signal Processing: Image Communication, 67:90-99, 2018.
