運用正負影像進行監督式訓練以實現紅外光與可見光之畫面亮度自適應融合

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：71

、訪客IP：3.144.41.200

姓名

葉凌瑋(Ling-Wei Yeh) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

運用正負影像進行監督式訓練以實現紅外光與可見光之畫面亮度自適應融合
(Using Positive and Negative Images for Supervised Training to Achieve Luminance-Adaptive Fusion of Infrared and Visible Light Images)

相關論文

★ 基於QT之跨平台無線心率分析系統實現	★ 網路電話之額外訊息傳輸機制
★ 針對與運動比賽精彩畫面相關串場效果之偵測	★ 植基於向量量化之視訊/影像內容驗證技術
★ 植基於串場效果偵測與內容分析之棒球比賽精華擷取系統	★ 以視覺特徵擷取為基礎之影像視訊內容認證技術
★ 使用動態背景補償以偵測與追蹤移動監控畫面之前景物	★ 應用於H.264/AVC視訊內容認證之適應式數位浮水印
★ 棒球比賽精華片段擷取分類系統	★ 利用H.264/AVC特徵之多攝影機即時追蹤系統
★ 利用隱式型態模式之高速公路前車偵測機制	★ 基於時間域與空間域特徵擷取之影片複製偵測機制
★ 結合數位浮水印與興趣區域位元率控制之車行視訊編碼	★ 應用於數位智權管理之H.264/AVC視訊加解密暨數位浮水印機制
★ 基於文字與主播偵測之新聞視訊分析系統	★ 植基於數位浮水印之H.264/AVC視訊內容驗證機制

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

至系統瀏覽論文 (2025-7-25以後開放)

摘要(中)

紅外光與可見光影像融合的目的是將同一場景之不同光譜影像資訊保留於單一畫面中。然而，兩輸入圖的較大亮度差異可能導致彼此內容干擾，影響融合影像的互補資訊呈現。現存的影像融合方法通常對於較低亮度影像有較佳的效果，但是當其中一張影像具有較高亮度內容時，我們發現融合影像的紋理對比度經常出現降低的情況。為了避免極高亮度和極低亮度影像所引起的融合效果不理想，我們提出新的深度學習模型訓練方法，運用正負影像進行自監督式訓練。考量畫面內容的邊緣是融合影像的呈現重點，我們計算影像梯度提取紋理以保留原圖細節供監督式學習參考。畫面中不同區域的紋理協助產生訓練融合影像的引導圖，我們另也運用邊緣增強做為融合影像梯度的參考以降低極端亮度對於畫面細節的影響。我們引入通道注意力模塊來針對特徵圖中不同通道進行強化或弱化，並加速模型訓練。監督式訓練計算正負影像與引導圖間的相似度、負融合影像反轉後與正融合影像間的相似度，以及融合影像梯度的相似度，盡可能保留畫面細節，並實現亮度自適應的影像融合目標。實驗結果證明我們所提出的方法能夠取得良好的成效。

摘要(英)

The main task of fusing infrared and visible light images is to preserve the spectral information of the same scene in a single frame. However, the extreme brightness differences between the two input images can lead to content interference and affect the presentation of complementary information in the fused image. Existing image fusion methods often perform well for lower-brightness images, but when one of the images contains high-brightness content, we observed a decrease in texture contrast in the fused image. To overcome the issue of poor fusion results caused by extremely high and low brightness images, we propose a new training method that utilizes self-supervised learning with positive and negative images in a deep learning neural network. We extract image gradients to generate ground truth that preserves the details from the original images for reference in supervised learning. Additionally, we use edge enhancement as the ground truth for the gradients of the fused image to mitigate the adverse effects of brightness on preserving fused details. We also introduce a channel attention module to enhance or weaken different channels in the feature maps. The training process measures the similarity between the positive and negative images and the designed ground truth, as well as the similarity between the inverted negative fused image and the positive fused image. This encourages the deep learning network to preserve detailed features and achieve Luminance-adaptive image fusion. Experimental results demonstrate the effectiveness of our proposed method, confirming that the generation of ground truth can guide the preservation of information in the fusion of infrared and visible light images.

關鍵字(中)

★ 影像融合
★ 深度學習
★ 卷積神經網路
★ 監督式學習
★ 亮度自適應

關鍵字(英)

★ Image Fusion
★ Deep Learning
★ Convolution Neural Network
★ Supervised Learning
★ Luminance Self-Adaptive

論文目次

論文摘要 ii
Abstract iii
致謝 v
目錄 vi
附圖目錄 ix
表格目錄 xi
一、緒論 1
1-1 研究背景與動機 1
1-2 研究貢獻 3
1-3 論文架構 3
二、相關研究 4
2-1 基於自動編碼器的融合方法 4
2-2 基於卷積神經網路的融合方法 6
2-3 基於生成對抗網路的融合方法 9
三、提出方法 11
3-1 模型架構 13
3-1-1 特徵萃取器 14
3-1-2 SEBlock 15
3-1-3 特徵還原器 16
3-2 訓練流程 17
3-3 損失函數設計 18
3-3-1 強度損失 19
3-3-2 梯度損失 21
3-3-3 共同特徵損失 23
3-4 訓練細節 24
四、實驗結果 26
4-1 實驗開發環境 26
4-2 資料集 27
4-3 評估方法 28
4-3-1 Structural similarity index measure（SSIM） 28
4-3-2 Edge-based similarity measure（QAB/F） 29
4-3-3 Visual information fidelity（VIF） 29
4-3-4 Sum of the correlations of differences（SCD） 29
4-3-5 Mutual information（MI） 29
4-3-6 Entropy（EN） 30
4-3-7 Average gradient（AG） 30
4-3-8 Standard deviation（SD） 30
4-4 實驗效果 30
4-4-1 LLVIP實驗效果 31
4-4-2 TNO實驗效果 34
4-4-3 融合畫面比較 34
4-5 消融實驗 36
4-5-1 正負影像訓練效果評估 36
4-5-2 SEBlock效果評估 37
4-5-3 邊緣梯度增強效果評估 38
4-5-4 模糊梯度與原始梯度兩階段訓練效果評估 39
4-5-5 不同引導圖的訓練效果比較 40
五、結論與未來展望 42
六、參考文獻 43

參考文獻

[1] L. Tang, J. Yuan, and J. Ma, "Image fusion in the loop of high-level vision tasks: A semantic-aware real-time infrared and visible image fusion network," Information Fusion, vol. 82, pp. 28-42, 2022/06/01/ 2022, doi: https://doi.org/10.1016/j.inffus.2021.12.004.
[2] H. Li and X. J. Wu, "DenseFuse: A Fusion Approach to Infrared and Visible Images," IEEE Transactions on Image Processing, vol. 28, no. 5, pp. 2614-2623, 2019, doi: 10.1109/TIP.2018.2887342.
[3] J. Hu, L. Shen, and G. Sun, "Squeeze-and-Excitation Networks," in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18-23 June 2018 2018, pp. 7132-7141, doi: 10.1109/CVPR.2018.00745.
[4] Y. Zhang, Y. Liu, P. Sun, H. Yan, X. Zhao, and L. Zhang, "IFCNN: A general image fusion framework based on convolutional neural network," Information Fusion, vol. 54, pp. 99-118, 2020/02/01/ 2020, doi: https://doi.org/10.1016/j.inffus.2019.07.011.
[5] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[6] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, "Densely connected convolutional networks," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 4700-4708.
[7] R. Hou et al., "VIF-Net: An Unsupervised Framework for Infrared and Visible Image Fusion," IEEE Transactions on Computational Imaging, vol. 6, pp. 640-651, 2020, doi: 10.1109/TCI.2020.2965304.
[8] H. Li, X.-J. Wu, and J. Kittler, "Infrared and visible image fusion using a deep learning framework," in 2018 24th international conference on pattern recognition (ICPR), 2018: IEEE, pp. 2705-2710.
[9] J. Ma, W. Yu, P. Liang, C. Li, and J. Jiang, "FusionGAN: A generative adversarial network for infrared and visible image fusion," Information Fusion, vol. 48, pp. 11-26, 2019/08/01/ 2019, doi: https://doi.org/10.1016/j.inffus.2018.09.004.
[10] H. Xu, P. Liang, W. Yu, J. Jiang, and J. Ma, "Learning a Generative Model for Fusing Infrared and Visible Images via Conditional Generative Adversarial Network with Dual Discriminators," presented at the Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, 2019, 2019. [Online]. Available: https://doi.org/10.24963/ijcai.2019/549.
[11] D. Han, L. Li, X. Guo, and J. Ma, "Multi-exposure image fusion via deep perceptual enhancement," Information Fusion, vol. 79, pp. 248-262, 2022/03/01/ 2022, doi: https://doi.org/10.1016/j.inffus.2021.10.006.
[12] H. Xu, J. Ma, J. Jiang, X. Guo, and H. Ling, "U2Fusion: A Unified Unsupervised Image Fusion Network," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 1, pp. 502-518, 2022, doi: 10.1109/TPAMI.2020.3012548.
[13] X. Jia, C. Zhu, M. Li, W. Tang, and W. Zhou, "LLVIP: A visible-infrared paired dataset for low-light vision," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 3496-3504.
[14] A. Toet, "TNO Image Fusion Dataset," ed: figshare, 2022.
[15] Q. Ha, K. Watanabe, T. Karasawa, Y. Ushiku, and T. Harada, "MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes," in 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 24-28 Sept. 2017 2017, pp. 5108-5115, doi: 10.1109/IROS.2017.8206396.
[16] J. Liu, X. Fan, J. Jiang, R. Liu, and Z. Luo, "Learning a Deep Multi-Scale Feature Ensemble and an Edge-Attention Guidance for Image Fusion," IEEE Transactions on Circuits and Systems for Video Technology, vol. 32, no. 1, pp. 105-119, 2022, doi: 10.1109/TCSVT.2021.3056725.
[17] J. Liu et al., "Target-aware dual adversarial learning and a multi-scenario multi-modality benchmark to fuse infrared and visible for object detection," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5802-5811.
[18] J. Ma, L. Tang, F. Fan, J. Huang, X. Mei, and Y. Ma, "SwinFusion: Cross-domain Long-range Learning for General Image Fusion via Swin Transformer," IEEE/CAA Journal of Automatica Sinica, vol. 9, no. 7, pp. 1200-1217, 2022, doi: 10.1109/JAS.2022.105686.

指導教授

蘇柏齊(Po-Chyi Su)

審核日期

2023-7-27

推文