一種可同時更新神經網路各層網路參數的新技術— 採用關聯式學習及管路化機制;Enabling simultaneous parameter updates in different layers for a neural network —using associated learning and pipeline

NCUIR > College of Electrical Engineering & Computer Science > Graduate Institute of Computer Science and Information Engineering > Electronic Thesis & Dissertation > Item 987654321/93152

Please use this identifier to cite or link to this item: http://ir.lib.ncu.edu.tw/handle/987654321/93152

Title:	一種可同時更新神經網路各層網路參數的新技術— 採用關聯式學習及管路化機制;Enabling simultaneous parameter updates in different layers for a neural network —using associated learning and pipeline
Authors:	林廷翰;Lin, Ting-Han
Contributors:	資訊工程學系
Keywords:	倒傳遞;反向鎖定;關聯式學習;平行化訓練;模型平行化;back-propagation;backward locking;associated learning;parallel training;model parallelism
Date:	2023-07-20
Issue Date:	2024-09-19 16:44:40 (UTC+8)
Publisher:	國立中央大學
Abstract:	倒傳遞 (Back-propagation, BP) 廣泛運用於今日的深度學習演算法，然而它仍存在反向鎖定的問題導致模型訓練效率不佳。許多研究嘗試解決反向鎖定問題，而關聯式學習 (Associated Learning, AL) 便是其中一種模型架構。雖然關聯式學習理論上可透過管線化來增加訓練的效率，但原論文並未實現管線化，本論文補足這個部份，並透過大量實驗及效能分析工具(profiler) 觀察關聯式學習管線化後的實際行為。本論文亦和過去使用倒傳遞訓練之模型做比較，探討各自的優勢與限制，並討論關聯式學習未來的研究方向。;Back-propagation (BP) is widely utilized in deep learning algorithms, but it suffers from the issue of backward locking, resulting in inefficient model training. Various research efforts have been made to address this problem, and one promising solution is Associated Learning (AL). In theory, AL has the potential to enhance training efficiency through pipelining. However, the original proposal lacks the implementation of the pipeline. In this thesis, we bridge this gap by implementing the pipeline mechanism and conducting experiments on multiple GPUs. By leveraging profiling tools, we analyze the behavior of AL after pipelining. We compare models trained using back-propagation and pipelined AL to examine their respective advantages and limitations. Moreover, we discuss potential future research directions for Associated Learning.
Appears in Collections:	[Graduate Institute of Computer Science and Information Engineering] Electronic Thesis & Dissertation

Files in This Item:

File	Description	Size	Format
index.html		0Kb	HTML	5	View/Open

社群 sharing

Loading...