| 摘要: | 隨著深度學習（Deep Learning, DL）模型於自然語言處理（Natural Language Processing, NLP）與電腦視覺（Computer Vision, CV）等領域的廣泛應用，愈來愈多含有敏感性資訊（Sensitive Data）被用於訓練模型，導致深度學習中的隱私保護議題日益受到重視。已有研究指出，在深度學習流程中，若模型訓練過程缺乏隱私保護，可能導致訓練資料的洩漏，甚至被用於重建原始資料。此類隱私攻擊所揭露的資訊安全風險，不僅是學術關注焦點，亦已對實際社會帶來影響。
差分隱私（Differential Privacy, DP）作為一種具備理論保證的隱私保護機制，目前已廣泛應用於深度學習模型的訓練流程中。傳統差分隱私訓練方法雖可有效防止敏感資訊洩漏，但普遍面臨訓練效能下降與硬體資源消耗高等問題。為兼顧隱私保護與模型效能，本論文提出多尺度梯度裁剪（Multi-Scale Gradient Clipping, MSGC）方法，結合逐樣本裁剪（Per-sample Clipping）與逐層自適應裁剪（Per-layer Adaptive Clipping）機制，並採用雙次反向傳播以提升記憶體使用效率。該方法可根據各層梯度特性動態調整裁剪門檻，有效降低過度裁剪對模型學習能力的影響，同時注入高斯雜訊以滿足差分隱私要求。實驗結果顯示，在文本生成任務中，MSGC相較於無隱私保護下仍可保有90.48%的生成品質；在文本分類任務中同樣能達到80.51%之準確率；在影像辨識任務上僅存15.90%之分類錯誤率。在硬體資源支援方面，MSGC相較於傳統差分隱私方法在GPT-Neo與BERT模型上分別提升200.00%與215.38%之記憶體效能，並提升23.41%與7.57%的訓練吞吐量。
;With the rapid adoption of deep learning (DL) in areas such as natural language processing (NLP) and computer vision (CV), the use of sensitive data for model training has become increasingly common, intensifying concerns about privacy protection. It has been demonstrated that, without appropriate privacy-preserving mechanisms, deep learning models are vulnerable to privacy attacks that can leak or reconstruct sensitive training data. These privacy risks have not only attracted significant academic attention but have also begun to impact broader society.
Differential Privacy (DP), a privacy protection mechanism with rigorous theoretical guarantees, has now been widely applied to the training process of deep learning models. Although traditional differentially private training methods can effectively prevent sensitive information leakage, they often suffer from significant drops in model performance and high consumption of hardware resources. To balance privacy protection and model utility, this thesis proposes a Multi-Scale Gradient Clipping (MSGC) method, which integrates per-sample clipping and per-layer adaptive clipping mechanisms and adopts double backward propagation to improve memory efficiency. This method dynamically adjusts the clipping threshold for each layer based on gradient characteristics, effectively mitigating the negative impact of over-clipping on learning ability, while injecting Gaussian noise to ensure differential privacy.
Experimental results show that, in text generation tasks, MSGC retains 90.48% of generation quality compared to non-private baselines; in text classification, it achieves an accuracy of 80.51%; and in image recognition, the classification error rate remains as low as 15.90%. In terms of hardware efficiency, MSGC outperforms traditional differential privacy methods, achieving 200.00% and 215.38% improvement in memory utilization on GPT-Neo and BERT, respectively, as well as a 23.41% and 7.57% increase in training throughput. |
