Transformer-based Gloss-free Sign Language Translation

NCUIR > College of Electrical Engineering & Computer Science > Graduate Institute of Computer Science and Information Engineering > Electronic Thesis & Dissertation > Item 987654321/98287

Please use this identifier to cite or link to this item: https://ir.lib.ncu.edu.tw/handle/987654321/98287

Title:	Transformer-based Gloss-free Sign Language Translation
Authors:	江子青;Chiang, Zi-Cing
Contributors:	資訊工程學系
Keywords:	手語翻譯
Date:	2025-07-21
Issue Date:	2025-10-17 12:35:07 (UTC+8)
Publisher:	國立中央大學
Abstract:	全球有超過四億三千萬人受致殘性聽力損失影響，溝通障礙常在教育、就業和社會融合方面造成重大挑戰，這使得手語翻譯（SLT）成為促進更無障礙與共融社會的關鍵領域。為應對這些障礙，本論文提出一個基於Transformer 的框架，旨在將手語影片直接翻譯為文本，以克服時空複雜性與高昂標註成本等關鍵挑戰。我們的架構整合了自適應遮蔽與融合模組、上下文位置編碼以及詞元級對比損失，以提升翻譯準確性及多模態對齊效果。該模型表現優於現有方法，在公開基準數據集上達到了頂尖的精確度與流暢度，同時在更具挑戰性的對話型數據上展現出強大的泛化能力。此整合式架構證實其有效，為生成準確、流暢的翻譯提供了一種卓越的方法，並推動了免詞彙庫模型的技術前沿。;With over 430 million people worldwide affected by disabling hearing loss, communication barriers often create significant challenges in education, employment, and social integration, making Sign Language Translation (SLT) a crucial field for fostering a more accessible and inclusive society. To address these barriers, we introduce a Transformer-based framework designed to translate sign language videos directly into text, confronting key challenges like spatio-temporal complexity and costly annotation. Our architecture integrates an Adaptive Masking and Fusion module, Contextual Position Encoding, and a Token-Level Contrastive Loss to enhance accuracy and multimodal alignment. The model outperforms existing approaches, demonstrating a state-of-the-art level of precision and fluency on public benchmarks and a strong generalization on more challenging conversational data. Ultimately, this integrated architecture proves highly effective, offering a superior method to generate accurate and fluent translations compared to existing gloss-free models.
Appears in Collections:	[Graduate Institute of Computer Science and Information Engineering] Electronic Thesis & Dissertation

Files in This Item:

File	Description	Size	Format
index.html		0Kb	HTML	26	View/Open

社群 sharing

Loading...