整合錯誤更正碼技術之自動化編碼簿學習

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：151

、訪客IP：18.219.242.150

姓名

周哲宇(Che-Yu Chou) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

整合錯誤更正碼技術之自動化編碼簿學習
(Automated Codebook Learning with Error Correcting Output Code Technique)

相關論文

★ 透過網頁瀏覽紀錄預測使用者之個人資訊與性格特質	★ 透過矩陣分解之多目標預測方法預測使用者於特殊節日前之瀏覽行為變化
★ 動態多模型融合分析研究	★ 擴展點擊流：分析點擊流中缺少的使用者行為
★ 關聯式學習：利用自動編碼器與目標傳遞法分解端到端倒傳遞演算法	★ 融合多模型排序之點擊預測模型
★ 分析網路日誌中有意圖、無意圖及缺失之使用者行為	★ 基於自注意力機制產生的無方向性序列編碼器使用同義詞與反義詞資訊調整詞向量
★ 探索深度學習或簡易學習模型在點擊率預測任務中的使用時機	★ 空氣品質感測器之故障偵測--基於深度時空圖模型的異常偵測框架
★ 以同反義詞典調整的詞向量對下游自然語言任務影響之實證研究	★ 結合時空資料的半監督模型並應用於PM2.5空污感測器的異常偵測
★ 藉由權重之梯度大小調整DropConnect的捨棄機率來訓練神經網路	★ 使用圖神經網路偵測 PTT 的低活躍異常帳號
★ 針對個別使用者從其少量趨勢線樣本生成個人化趨勢線	★ 基於雙變量及多變量貝他分布的兩個新型機率分群模型

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

錯誤更正碼（Error Correcting Output Codes, ECOC）是一種用於解決多元分類問題的技術，其核心概念是設計編碼簿（Codebook），將每個類別映射到唯一的碼字（Codeword），並將編碼簿作為標籤讓模型學習。在基於錯誤更正碼技術的模型中，編碼簿的設計至關重要。過去的研究中，編碼簿多為人為設計、使用已知的編碼技術或隨機生成。然而，這些方法不僅需在模型訓練前額外產生，其產生的編碼簿也不一定能適用於任意資料集。本論文基於對比學習的模型框架，提出了三種自動化編碼簿學習的錯誤更正碼模型。這些模型無需在訓練前生成編碼簿，且編碼簿的生成由模型根據資料集的特性自動學習，從而解決了上述提及的編碼簿問題。我們在四種資料集中與兩種基礎模型進行比較，並評估三種錯誤更正碼模型的優劣與限制。此外，我們還實驗了自動化編碼簿學習的錯誤更正碼模型是否具有抵禦對抗攻擊的能力，並討論了未來改進的方向。

摘要(英)

Error Correcting Output Codes (ECOC) is a technique for solving multi-class classification problems. Its core concept involves designing a codebook: each class maps to a unique codeword; these codewords are treated as labels for model training. Thus, the design of the codebook is crucial. In past research, codebooks were often manually designed based on known encoding techniques or generated randomly. However, these methods require manual codebook design before model training, and there may be better choices of codebooks for the given datasets. This paper proposes three automated codebook learning models for ECOC based on the framework of contrastive learning. These models do not require manual codebook design before training, and the model automatically learns the codebook based on the dataset′s characteristics. We compare these models with two baseline models on four open datasets and evaluate the strengths, weaknesses, and limitations of the three ECOC models. Additionally, we experiment with whether the ECOC models with automated codebook learning can resist adversarial attacks and discuss directions for future improvements.

關鍵字(中)

★ 對比學習
★ 自監督式學習
★ 錯誤更正碼
★ 對抗攻擊

關鍵字(英)

★ Contrastive Learning
★ Self-Supervised Learning
★ Error Correcting Output Codes
★ Adversarial Attacks

論文目次

摘要 v
Abstract vi
致謝 viii
目錄 ix
一、緒論 1
二、相關研究 4
2.1 對抗攻擊 (Adversarial Attack)....................................... 4
2.2 對抗攻擊的防禦方式 ................................................... 5
2.3 錯誤更正碼 (Error Correcting Output Code)..................... 6
2.4 SimCLR ................................................................... 8
三、研究模型及方法 9
3.1 ACL: 基於預訓練的自動化編碼簿學習 ............................ 9
3.1.1 預訓練模型 ...................................................... 10
3.1.2 微調模型 ......................................................... 12
3.2 模型之損失函數設計 ................................................... 15
3.2.1 對比學習之損失函數 .......................................... 15
3.2.2 分類問題之損失函數 .......................................... 16
3.2.3 預測碼字與正確類別碼字間之損失函數 .................. 16
3.2.4 錯誤更正碼之損失函數 ....................................... 17
3.3 ACL-CFPC: 微調模型的編碼簿再訓練 ............................ 20
3.4 ACL-TFC: 基於 ACL-CFPC 模型編碼簿的模型訓練 .......... 22
3.5 三種自動化學習編碼簿模型的比較 ................................. 23
四、實驗設計與結果分析 25
4.1 資料集介紹與對抗例生成 ............................................. 25
4.2 實驗環境與模型參數設定 ............................................. 26
4.3 實驗模型介紹與評估方式 ............................................. 27
4.4 實驗結果與分析 ......................................................... 29
4.4.1 ACL 模型與基礎模型間的比較 ............................. 29
4.4.2 隨機初始權重學習的模型比較 .............................. 32
4.5 ACL 模型與其他 ECOC 模型的結果比較......................... 33
4.6 基於不同資料集特性的自動化學習編碼簿效果分析 ............ 34
4.7 消融實驗 .................................................................. 35
4.7.1 不同碼字長度及損失函數之 ACL 模型於 CLEAN
下的表現 .................................................................. 36
4.7.2 不同碼字長度及損失函數之 ACL 模型於 FGSM 下
的表現 ..................................................................... 38
4.7.3 不同碼字長度及損失函數之 ACL 模型於 PGD 下
的表現 ..................................................................... 39
4.7.4 ϵ 與碼字長度之相關實驗 ..................................... 41
4.7.5 ϵ 與損失函數之相關實驗 ..................................... 43
五、總結 46
5.1 結論 ........................................................................ 46
5.2 未來展望 .................................................................. 47
參考文獻 48
附錄 A 實驗程式碼 52
附錄 B ACL 編碼簿的詳細生成流程 53

參考文獻

[1] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proceedings of the IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
[2] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classification with deep convolutional neural networks,” Advances in neural information processing systems, vol. 25, 2012.
[3] G. Hinton, L. Deng, D. Yu, et al., “Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups,” IEEE Signal processing magazine, vol. 29, no. 6, pp. 82–97, 2012.
[4] D. Andor, C. Alberti, D. Weiss, et al., “Globally normalized transition-based neural networks,” arXiv preprint arXiv:1603.06042, 2016.
[5] S. Hochreiter and J. Schmidhuber, “Long short-term memory,” Neural computation, vol. 9, no. 8, pp. 1735–1780, 1997.
[6] C. Szegedy, W. Zaremba, I. Sutskever, et al., “Intriguing properties of neural networks,” arXiv preprint arXiv:1312.6199, 2013.
[7] I. J. Goodfellow, J. Shlens, and C. Szegedy, “Explaining and harnessing adversarial examples,” arXiv preprint arXiv:1412.6572, 2014.
[8] S.-M. Moosavi-Dezfooli, A. Fawzi, and P. Frossard, “Deepfool: A simple and accurate method to fool deep neural networks,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2574–2582.
[9] A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu, “Towards deep learning models resistant to adversarial attacks,” arXiv preprint arXiv:1706.06083, 2017.
[10] N. Carlini and D. Wagner, “Towards evaluating the robustness of neural networks,” in 2017 ieee symposium on security and privacy (sp), Ieee, 2017, pp. 39–57.
[11] N. Papernot, P. McDaniel, X. Wu, S. Jha, and A. Swami, “Distillation as a defense to adversarial perturbations against deep neural networks,” in 2016 IEEE symposium on security and privacy (SP), IEEE, 2016, pp. 582–597.
[12] D. Hendrycks and K. Gimpel, “A baseline for detecting misclassified and out-ofdistribution examples in neural networks,” arXiv preprint arXiv:1610.02136, 2016.
[13] D. Meng and H. Chen, “Magnet: A two-pronged defense against adversarial examples,” in Proceedings of the 2017 ACM SIGSAC conference on computer and communications security, 2017, pp. 135–147.
[14] G. Verma and A. Swami, “Error correcting output codes improve probability estimation and adversarial robustness of deep neural networks,” Advances in Neural Information Processing Systems, vol. 32, 2019.
[15] Y. Song, Q. Kang, and W. P. Tay, “Error-correcting output codes with ensemble diversity for robust learning in neural networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 35, 2021, pp. 9722–9729.
[16] L. Wan, T. Alpcan, E. Viterbo, and M. Kuijper, “Efficient error-correcting output codes for adversarial learning robustness,” in ICC 2022-IEEE International Conference on Communications, IEEE, 2022, pp. 2345–2350.
[17] T. Philippon and C. Gagné, “Improved robustness against adaptive attacks with ensembles and error-correcting output codes,” arXiv preprint arXiv:2303.02322, 2023.
[18] T. G. Dietterich and G. Bakiri, “Solving multiclass learning problems via errorcorrecting output codes,” Journal of artificial intelligence research, vol. 2, pp. 263–286, 1994.
[19] I. Evron, O. Onn, T. Weiss, H. Azeroual, and D. Soudry, “The role of codeword-toclass assignments in error-correcting codes: An empirical study,” in International Conference on Artificial Intelligence and Statistics, PMLR, 2023, pp. 8053–8077.
[20] A. Zhang, Z.-L. Wu, C.-H. Li, and K.-T. Fang, “On hadamard-type output coding in multiclass learning,” in Intelligent Data Engineering and Automated Learning: 4th International Conference, IDEAL 2003, Hong Kong, China, March 21-23, 2003. Revised Papers 4, Springer, 2003, pp. 397–404.
[21] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, PMLR, 2020, pp. 1597–1607.
[22] N. Papernot, P. McDaniel, and I. Goodfellow, “Transferability in machine learning: From phenomena to black-box attacks using adversarial samples,” arXiv preprint arXiv:1605.07277, 2016.
[23] A. Ilyas, L. Engstrom, A. Athalye, and J. Lin, “Query-efficient black-box adversarial examples (superceded),” arXiv preprint arXiv:1712.07113, 2017.
[24] W. Xu, D. Evans, and Y. Qi, “Feature squeezing: Detecting adversarial examples in deep neural networks,” arXiv preprint arXiv:1704.01155, 2017.
[25] C. Xie, J. Wang, Z. Zhang, Z. Ren, and A. Yuille, “Mitigating adversarial effects through randomization,” arXiv preprint arXiv:1711.01991, 2017.
[26] S. Yang, P. Luo, C. C. Loy, K. W. Shum, and X. Tang, “Deep representation learning with target coding,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 29, 2015.
[27] P. Rodríguez, M. A. Bautista, J. Gonzalez, and S. Escalera, “Beyond one-hot encoding: Lower dimensional target embedding,” Image and Vision Computing, vol. 75, pp. 21–31, 2018.
[28] A. Kusupati, M. Wallingford, V. Ramanujan, et al., “Llc: Accurate, multi-purpose learnt low-dimensional binary codes,” Advances in neural information processing systems, vol. 34, pp. 23 900–23 913, 2021.
[29] S. Gupta and S. Amin, “Integer programming-based error-correcting output code design for robust classification,” in Uncertainty in Artificial Intelligence, PMLR, 2021, pp. 1724–1734.
[30] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
[31] D. Shah and T. M. Aamodt, “Learning label encodings for deep regression,” arXiv preprint arXiv:2303.02273, 2023. [32] K. Sohn, “Improved deep metric learning with multi-class n-pair loss objective,” Advances in neural information processing systems, vol. 29, 2016.
[33] Z. Wu, Y. Xiong, S. X. Yu, and D. Lin, “Unsupervised feature learning via nonparametric instance discrimination,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 3733–3742.
[34] A. v. d. Oord, Y. Li, and O. Vinyals, “Representation learning with contrastive predictive coding,” arXiv preprint arXiv:1807.03748, 2018.
[35] P. Khosla, P. Teterwak, C. Wang, et al., “Supervised contrastive learning,” Advances in neural information processing systems, vol. 33, pp. 18 661–18 673, 2020.
[36] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel, “Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition,” Neural networks, vol. 32, pp. 323–332, 2012.
[37] Y. You, I. Gitman, and B. Ginsburg, “Large batch training of convolutional networks,” arXiv preprint arXiv:1708.03888, 2017.
[38] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick, “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9729–9738.
[39] J.-B. Grill, F. Strub, F. Altché, et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems, vol. 33, pp. 21 271–21 284, 2020.
[40] M. Caron, I. Misra, J. Mairal, P. Goyal, P. Bojanowski, and A. Joulin, “Unsupervised learning of visual features by contrasting cluster assignments,” Advances in neural information processing systems, vol. 33, pp. 9912–924, 2020.

指導教授

陳弘軒(Hung-Hsuan Chen)

審核日期

2024-7-16

推文