基於區域標準化及感知色彩距離的兩階段影像修補方法

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：43

、訪客IP：18.116.87.189

姓名

李政瑩(Cheng-Ying Li) 查詢紙本館藏

畢業系所

軟體工程研究所

論文名稱

基於區域標準化及感知色彩距離的兩階段影像修補方法
(Two-Stage Image Inpainting based on Region Normalization and Perceptual Color Distance)

相關論文

★ 影片指定對象臉部置換系統	★ 以單一攝影機實現單指虛擬鍵盤之功能
★ 基於視覺的手寫軌跡注音符號組合辨識系統	★ 利用動態貝氏網路在空照影像中進行車輛偵測
★ 以視訊為基礎之手寫簽名認證	★ 使用膚色與陰影機率高斯混合模型之移動膚色區域偵測
★ 影像中賦予信任等級的群眾切割	★ 航空監控影像之區域切割與分類
★ 在群體人數估計應用中使用不同特徵與回歸方法之分析比較	★ 以視覺為基礎之強韌多指尖偵測與人機介面應用
★ 在夜間受雨滴汙染鏡頭所拍攝的影片下之車流量估計	★ 影像特徵點匹配應用於景點影像檢索
★ 自動感興趣區域切割及遠距交通影像中的軌跡分析	★ 基於回歸模型與利用全天空影像特徵和歷史資訊之短期日射量預測
★ Analysis of the Performance of Different Classifiers for Cloud Detection Application	★ 全天空影像之雲追蹤與太陽遮蔽預測

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2026-7-6以後開放)

摘要(中)

影像修補(Image Inpainting)在電腦視覺領域中是一項具有挑戰性的任務，過去的研究大多是基於樣例的方法(Exemplar-based methods)。但隨著人工智慧領域的蓬勃發展，最近的研究發現基於深度學習的方法(Deep-learning-based methods)可以在影像修補上獲得更好的效果。本篇論文提出一個兩階段的生成對抗網路(Generative Adversarial Networks)，藉由使用者輸入的影像及遮罩，來執行由粗糙到精細的影像修補。
在網路的第一階段，我們使用區域標準化(Region Normalization)來產生具有正確結構的粗糙模糊結果；在第二階段，我們使用上下文注意機制(Contextual Attention)來利用周圍區域的紋理資訊來產生最終結果。
儘管使用區域標準化可以改善模型的效能和輸出結果的品質，但是可能會出現明顯的色彩偏移問題。為了解決此問題，我們在損失函數使用了感知色彩距離(Perceptual Color Distance)。
最後根據定量實驗結果，本論文提出的方法在Inception Score、Fréchet Inception Distance及感知色彩距離上，皆優於現有的類似方法。

摘要(英)

Image inpainting is a challenging task in computer vision, and most of the previous studies are exemplar-based methods. However, with the vigorous development of artificial intelligence, recent studies have found that deep-learning-based methods can achieve better results on image inpainting. In this thesis, we proposed a two-stage architecture to perform image inpainting from coarse to fine, which uses images and masks input by the user.
In the first stage, we apply Region Normalization (RN) to generate coarse blur results with the correct structure. In the second stage, we use Contextual Attention to utilize the texture information of surrounding regions to generate the final results. Although using RN can improve the network′s performance and quality, there may be visible color shifts. To solve this problem, we introduced Perceptual Color Distance into loss function.
In quantitative comparison with other similar methods, the method proposed in this thesis is superior to existing similar methods in Inception Score, Fréchet Inception Distance, and Perceptual Color Distance.

關鍵字(中)

★ 生成對抗網路
★ 影像修補

關鍵字(英)

★ Generative Adversarial Network
★ Image Inpainting

論文目次

摘要 VI
Abstract VII
致謝 VIII
目錄 IX
圖目錄 XI
表目錄 XI
第一章緒論 1
1.1 研究背景與動機 1
1.2 論文架構 2
第二章文獻回顧 4
2.1 Places2資料集 [17] 4
2.2 生成對抗網路 [18] 5
2.3 基於樣例的影像修補方法 6
2.4 基於深度學習的影像修補方法 8
2.4.1 Contextual Attention [10] 8
2.4.2 Gated Convolution [12] 9
2.4.3 Region Normalization [13] 11
2.5 PatchGAN Discriminator [19] 14
2.6 CIEDE2000 [20] 15
第三章研究方法 18
3.1 資料集 18
3.1.1 影像資料集 18
3.1.2 遮罩資料集 18
3.2 二階段的生成模型 19
3.2.1 第一階段 20
3.2.2 第二階段 23
3.3 鑑別模型 25
3.4 損失函數 25
3.4.1 感知色彩損失 25
3.4.2 對抗損失 27
第四章實驗結果 28
4.1 設備環境與參數設置 28
4.2 資料集 28
4.3 驗證指標 29
4.3.1 L1距離 29
4.3.2 峰值信噪比(PSNR) 29
4.3.3 結構相似性指標(SSIM) [23] 30
4.3.4 Inception Score [24] 31
4.3.5 FID (Fréchet Inception Distance) [25] 32
4.3.6 感知色彩距離 33
4.3.7 驗證指標在影像修補任務中的問題 34
4.4 完整模型之實驗比較結果 35
4.5 消融實驗(Ablation Experiments) 38
4.5.1 不同的第一階段架構之影響 38
4.5.2 加入第二階段架構與感知色彩損失的影響 42
4.6 速度評測 44
第五章結論與未來研究方向 46
參考文獻 47

參考文獻

[1] M. Bertalmio, G. Sapiro, V. Caselles, and C. Ballester, “Image inpainting,” in Proc. SIGGRAPH, 2000, pp. 417–424.
[2] C. Ballester, M. Bertalmio, V. Caselles, G. Sapiro, and J. Verdera, “Filling-in by joint interpolation of vector fields and gray levels,” IEEE Trans. Image Process., vol. 10, no. 8, pp. 1200–1211, Aug. 2001.
[3] M. Bertalmio, L. Vese, G. Sapiro, and S. Osher, “Simultaneous structure and texture image inpainting,” IEEE Trans. Image Process., vol. 12, no. 8, pp. 882–889, Aug. 2003.
[4] I. Drori, D. Cohen-Or, and H. Yeshurun, “Fragment-based image completion,” in Proc. ACM SIGGRAPH Papers SIGGRAPH, 2003, pp. 303–312.
[5] A. Criminisi, P. Perez and K. Toyama, “Region filling and object removal by exemplar-based image inpainting,” in IEEE Trans. on Image Process., vol. 13, no. 9, pp. 1200-1212, Sept. 2004.
[6] C. Barnes, E. Shechtman, A. Finkelstein, and D. Goldman, “PatchMatch: A randomized correspondence algorithm for structural image editing,” ACM Trans. Graph., vol. 28, no. 3, p. 24, 2009.
[7] C. Barnes, E. Shechtman, D. B. Goldman, and A. Finkelstein, “The generalized PatchMatch correspondence algorithm,” in Proc. European Conf. Computer Vision (ECCV), Sept. 2010, vol. 6313, pp. 29–43.
[8] D. Pathak, P. Krahenbuhl, J. Donahue, T. Darrell, and A. A. Efros, “Context encoders: Feature learning by inpainting,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), Jun. 2016, pp. 2536–2544.
[9] S. Iizuka, E. Simo-Serra, and H. Ishikawa, “Globally and locally consistent image completion,” ACM Trans. Graph., vol. 36, no. 4, p. 107, 2017
[10] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang, “Generative image inpainting with contextual attention,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., Jun. 2018, pp. 5505–5514.
[11] G. Liu, F. A. Reda, K. J. Shih, T.-C. Wang, A. Tao, and B. Catanzaro, “Image inpainting for irregular holes using partial convolutions,” in Proc. Eur. Conf. Comput. Vis. (ECCV), Sep. 2018, pp. 85–100.
[12] J. Yu, Z. Lin, J. Yang, X. Shen, X. Lu, and T. Huang, “Free-form image inpainting with gated convolution,” in Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), Oct. 2019, pp. 4471–4480.
[13] T. Yu, Z. Guo, X. Jin, S. Wu, Z. Chen, W. Li, Z. Zhang, and S. Liu, “Region normalization for image inpainting,” in Proc. Assoc. Advan. Artif. Intell. (AAAI), 2020, pp. 12733–12740.
[14] S. Ioffe and C. Szegedy, “Batch normalization: Accelerating deep network training by reducing internal covariate shift,” in Proc. Int. Conf. Mach. Learn. (ICML), 2015, pp. 448–456.
[15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., Jun. 2016, pp. 770–778.
[16] D. Ulyanov, A. Vedaldi, and V. Lempitsky, “Instance normalization: The missing ingredient for fast stylization,” 2016, arXiv:1607.08022. [Online]. Available: http://arxiv.org/abs/1607.08022
[17] B. Zhou, A. Lapedriza, A. Khosla, A. Oliva, and A. Torralba, “Places: A 10 million image database for scene recognition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 40, no. 6, pp. 1452–1464, Jun. 2018.
[18] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,” in Proc. Adv. Neural Inf. Process. Syst., 2014, pp. 2672–2680.
[19] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” 2016, arXiv:1611.07004. [Online]. Available: http://arxiv.org/abs/1611.07004
[20] G. Sharma, W. Wu, and E. N. Dalal, “The CIEDE2000 color-difference formula: Implementation notes, supplementary test data, and mathematical observations,” Color Res. Appl., vol. 30, no. 1, pp. 21–30, Feb. 2005
[21] X. Glorot, A. Bordes, and Y. Bengio, “Deep Sparse Rectifier Neural Networks,” in Proc. Int. Conf. Artificial Intelligence and Statistics, 2011, pp. 315–323.
[22] Z. Zhao, Z. Liu and M. Larson, “Towards Large Yet Imperceptible Adversarial Image Perturbations With Perceptual Color Distance,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2020, pp. 1036-1045.
[23] Z. Wang, A. C. Bovik, H. R. Sheikh and E. P. Simoncelli, “Image quality assessment: from error visibility to structural similarity,” in IEEE Trans. on Image Process., vol. 13, no. 4, pp. 600-612, April 2004.
[24] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, X. Chen, and X. Chen, “Improved techniques for training GANs,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2016, pp. 2234– 2242.
[25] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter, “GANs trained by a two time-scale update rule converge to a local nash equilibrium,” in Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), 2017, pp. 6626–6637.
[26] A. I. Oncu, F. Deger, and J. Y. Hardeberg, “Evaluation of digital inpainting quality in the context of artwork restoration,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 561–570.

指導教授

鄭旭詠(Hsu-Yung Cheng)

審核日期

2021-7-13

推文