聯邦學習中以知識蒸餾與量化來降低通訊及運算成本之架構;An Architecture for Reducing Communication and Computation Costs in Federated Learning via Knowledge Distillation and Quantization

NCU Institutional Repository > 資訊電機學院 > 資訊工程研究所 > 博碩士論文 > Item 987654321/98587

jsp.display-item.identifier=請使用永久網址來引用或連結此文件: https://ir.lib.ncu.edu.tw/handle/987654321/98587

题名:	聯邦學習中以知識蒸餾與量化來降低通訊及運算成本之架構;An Architecture for Reducing Communication and Computation Costs in Federated Learning via Knowledge Distillation and Quantization
作者:	陳力嘉;Chen, Li-Jia
贡献者:	資訊工程學系
关键词:	聯邦學習;知識蒸餾;量化;通訊成本;運算成本;Federated Learning;Knowledge Distillation;Quantization;Communication Cost;Computation Cost
日期:	2025-08-18
上传时间:	2025-10-17 12:57:28 (UTC+8)
出版者:	國立中央大學
摘要:	傳統機器學習（Machine Learning, ML）方式要求必須取得所有的資料，但對於高隱私性的資料取得是相當困難的，因此，一種分散式機器學習方法「聯邦學習」（Federated Learning, FL）由此誕生，這樣一來能夠用戶就無需傳輸敏感資料給外部模型訓練，取而代之，用戶就必須在本地進行訓練並且上傳模型參數給伺服器聚合，達到多個用戶協同訓練（Collaborative Training）的效果。然而，在模型參數量較大的情況下，就會需要傳輸相當多的資料給伺服器，通訊成本也因此遽增。另外，用戶端的本地訓練也會消耗大量的運算資源，加上用戶端的運算資源相當有限，非常不適合進行大模型的訓練以及部署。因此，如何有效壓縮模型以降低通訊負擔，並減輕本地端的運算壓力，成為本研究的重要目標。本論文提出Federated Learning Combining Knowledge Distillation and Quantization（FedKDQ）架構，使用知識蒸餾來壓縮模型，原先的大模型稱之為教師模型，壓縮後的小模型稱為學生模型，並且在部署學生模型的情況下提升模型準確率，另外使用量化來進一步減少通訊成本。透過將模型參數量化成Float16與Int8分別減少50%與75%的儲存空間。另外在訓練完後，透過將訓練完的模型進行Int8整數量化來減少推論時間。實驗結果顯示，與Baseline-Teacher相比，在Cinic-10資料集中，FedKDQ減少94.91%模型大小與65.88%推論時間，準確率僅減少4.34%。在IoTID20資料集中，FedKDQ與Baseline-Teacher相較則減少98.44%模型大小與57.14%推論時間，準確率僅減少4.53%。這顯示FedKDQ在圖像及網路資料集上皆能顯著降低通訊與運算資源需求，而模型準確率僅分別犧牲4.34% 與4.53%。 ;Traditional machine learning (ML) approaches require access to all data for training. However, acquiring highly sensitive data is often difficult due to privacy concerns. To address this issue, a distributed learning paradigm called Federated Learning (FL) has emerged. In FL, users no longer need to share raw data with external servers. Instead, training is performed locally, and only model parameters are uploaded to a central server for aggregation, enabling collaborative training across multiple users. However, when the model is large, a significant amount of parameter data must be transmitted, resulting in high communication costs. In addition, local training on client devices consumes considerable computational resources, which is problematic since these devices are often resource-constrained and unsuitable for training or deploying large-scale models. Therefore, how to effectively compress the model to reduce communication overhead and alleviate local computational burden has become a key objective of this study. This paper proposes a architecture called Federated Learning Combining Knowledge Distillation and Quantization (FedKDQ). It utilizes knowledge distillation to compress the model: the original large model is referred to as the teacher model, while the compressed version is the student model. The student model is deployed on the client side to reduce resource demands while maintaining accuracy. In addition, quantization is applied to further reduce communication cost. By quantizing model parameters to Float16 and Int8, storage requirements are reduced by approximately 50% and 75%, respectively. Furthermore, post-training integer quantization (Int8) is used to accelerate inference after training. The experimental results show that, compared with the Baseline-Teacher, FedKDQ reduces model size by 94.91% and inference time by 65.88% on the Cinic-10 dataset, with only a 4.34% drop in accuracy. On the IoTID20 dataset, FedKDQ achieves a reduction of 98.44% in model size and 57.14% in inference time compared to the Baseline-Teacher, while accuracy decreases by only 4.53%. These results demonstrate that FedKDQ can significantly reduce communication and computation resource demands on both image and network datasets, while sacrificing only 4.34% and 4.53% accuracy, respectively.
显示于类别:	[資訊工程研究所] 博碩士論文

文件中的档案:

档案	描述	大小	格式	浏览次数
index.html		0Kb	HTML	4	检视/开启

在NCUIR中所有的数据项都受到原著作权保护.

社群 sharing

数据加载中.....