利用 Attentive 來改善端對端中文語篇剖析遞迴類 神經網路系統

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：37

、訪客IP：3.139.70.131

姓名

王育任(Yu-Jen Wang) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

利用 Attentive 來改善端對端中文語篇剖析遞迴類神經網路系統
(Using Attentive to improve Recursive LSTM End-to- End Chinese Discourse Parsing)

相關論文

★ 行程邀約郵件的辨識與不規則時間擷取之研究	★ NCUFree校園無線網路平台設計及應用服務開發
★ 網際網路半結構性資料擷取系統之設計與實作	★ 非簡單瀏覽路徑之探勘與應用
★ 遞增資料關聯式規則探勘之改進	★ 應用卡方獨立性檢定於關連式分類問題
★ 中文資料擷取系統之設計與研究	★ 非數值型資料視覺化與兼具主客觀的分群
★ 關聯性字組在文件摘要上的探討	★ 淨化網頁：網頁區塊化以及資料區域擷取
★ 問題答覆系統使用語句分類排序方式之設計與研究	★ 時序資料庫中緊密頻繁連續事件型樣之有效探勘
★ 星狀座標之軸排列於群聚視覺化之應用	★ 由瀏覽歷程自動產生網頁抓取程式之研究
★ 動態網頁之樣版與資料分析研究	★ 同性質網頁資料整合之自動化研究

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

篇章剖析，可以幫助我們以不同角度來理解文句之間的關係與連結，但篇章剖析的資料結構目前仰賴人工標記，使得這項技術無法直接被利用在任意篇章中。因此至目前為止，有許多研究著手於讓電腦能夠自動對篇章進行剖析，並建立出一個完整的剖析樹。以中文語料庫 CDTB 來說，欲建立完整的篇章剖析程式，其問題主要可以被分成四個，分別是子句分割、剖析樹建立、子句關係辨識、中心關係辨識。
由於深度學習近幾年發展快速，因此針對篇章剖析的建構方法也從傳統的 SVM, CRF 等，進展到目前已遞迴類神經方式來建構剖析篇章。而在本篇論文中，我們也加入了許多目前最新的深度學習技術，例如 Attentive RvNN、self-attentive、BERT 等方法，來提高模型的準確度。
最後，我們成功將每一項任務的 F1 都提高了近 10% 左右，達到目前我們所知研究中最好的效能。

摘要(英)

Discourse parser can help us to understand the relationship and connection between sentences from different angles, but the tree structure data still need to rely on manual marking, which makes this technology cannot be directly used in life. So far, there have been many research studies on automatically construct the complete tree structure on the computer. Since deep learning has progressed rapidly in recent years, the construction method for discourse parser has also changed from the traditional SVM, CRF method to the current recursive neural.
In the Chinese corpus tree library CDTB, the parsing analysis problem can be divided into four main problems, including elementary discourse unit (EDU) segmentation, tree structure construction, center labeling, and sense labeling. In this paper, we use many state-of-the-art deep learning techniques, such as attentive recursive neural networks, self-attentive, and BERT to improve the performance.
In the end, we succeed to increase the accuracy by more than 10% of F1 of each task, reaching the best performance we know so far.

關鍵字(中)

★ 深度學習
★ 篇章剖析
★ 注意力機制
★ 遞迴類神經網路

關鍵字(英)

★ Deep Learning
★ Discourse Parsing
★ Attention
★ Recursive neural network

論文目次

摘要 i
Abstract ii
誌謝 iii
目錄 iv
圖目錄 vi
表目錄 vii
壹、緒論 1
1.1. 問題定義 2
1.2. 研究動機與目標 4
貳、相關研究 6
2.1. 篇章剖析語料庫 6
2.1.1. 英文語料庫 6
2.1.2. 中文語料庫 7
2.2. 篇章剖析程式 (Discourse Parser) 8
2.2.1. SVM Base Parser 8
2.2.2. DCRF Base Parser 8
2.2.3. Recursive Deep Models Parser 8
2.2.4. RvNN Chinese Discourse Parser 9
2.2.5. Transition-Based Dependency Parser 12
2.3. 深度學習相關技術 13
2.3.1. Recursive Neural Network (RvNN) 13
2.3.2. Attentive RvNN 15
2.3.3. Self-Attentive Sentence Embedding 16
2.3.4. FastText 16
2.3.5. BERT 16
參、 CDTB語料庫 18
3.1. 子句關係 18
3.2. 隱性關係與顯性關係 20
肆、模型設計 22
4.1. 子句分割 23
4.2. 文字嵌入與注意力機制 25
4.3. 注意力遞迴類神經網路 26
伍、實驗 28
5.1. 標準子句實驗 29
5.2. 端對端剖析模型實驗 30
5.3. 二元樹與多元樹比較分析實驗 30
5.4. 二元樹分析實驗 32
5.5. 子句關係分析 33
5.6. 個別實驗數據分析 34
5.6.1. 文字嵌入實驗 34
5.6.2. Self-Attentive 子句效能實驗 34
5.6.3. 多層Attentive RvNN 實驗 35
5.6.4. 子句分割實驗 35
5.7. 學習曲線 36
陸、結論 37
6.1. 未來展望 37
參考 39

參考文獻

[1]. William C Mann and Sandra A Thompson. Rhetorical structure theory: Toward a functional theory of text organization. Text-Interdisciplinary Journal for the Study of Discourse, 8(3):243–281, 1988.
[2]. Lynn Carlson, Daniel Marcu, and Mary Ellen Oku- rowski. 2003. Building a Discourse- tagged Corpus in the Framework of Rhetorical Structure Theory. Springer Netherlands.
[3]. Rashmi Prasad, Bonnie Webber, and Aravind Joshi. Reflections on the penn discourse treebank, comparable corpora, and complementary annotation. Computational Linguistics,
40(4):921–950, 2014.
[4]. Li Yancui, Feng Wenhe, Sung Jing, Kong Fang, *Zhou Guodong. Building Chinese
Discourse Corpus with Connective-driven Dependency Tree Structure[C]. In Proceedings of the 2014 conference on Emporical Methods in Natural Language Processing, pages 2105– 2114,October 25-29, 2014, Doha, Qatar.
[5]. Annie Louis, Aravind Joshi, and Ani Nenkova. Discourse indicators for content selection in summarization. In Proceedings of the SIGDIAL 2010 Conference, pages 147–156, Tokyo, Japan, September 2010. Association for Computational Linguistics. URL http: //www.aclweb.org/anthology/W/W10/W10-4327.
[6]. Christina Lioma, Birger Larsen, and Wei Lu. Rhetorical relations for information retrieval. In Proceedings of the 35th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR’12), pages 931–940, 2012. ISBN 978-1-4503- 1472-5. . URL http://doi.acm.org/10.1145/2348283.2348407.
[7]. Richard Socher, Alex Perelygin, Jean Y. Wu, Jason Chuang, Christopher D. Manning, Andrew Y. Ng and Christopher Potts. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631–1642, Seattle, Washington, USA, 18-21 October 2013. 2013 Association for Computational Linguistics
[8]. Kai Sheng Tai, Richard Socher*, Christopher D. Manning. Improved Semantic Representations From Tree-Structured Long Short-Term Memory Networks. ACL 2015
[9]. Yangfeng Ji and Noah A. Smith. Neural Discourse Structure for Text Categorization. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, pages 996–1005 Vancouver, Canada, July 30 - August 4, 2017. 2017 Association for Computational Linguistics
[10]. Jean Maillard, Stephen Clark, Dani Yogatama. Jointly Learning Sentence Embeddings and Syntax with Unsupervised Tree-LSTMs. Submitted to 31st Conference on Neural Information Processing Systems (NIPS 2017).
[11]. Yanyan Jia, Yuan Ye, Yansong Feng, Yuxuan Lai, Rui Yan and Dongyan Zhao. Modeling Discourse Cohesion for Discourse Parsing via Memory Network. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Short Papers), pages
39
438–443 Melbourne, Australia, July 15 - 20, 2018. ©2018 Association for Computational Linguistics.
[12]. Yancui Li, Wenhe Feng, Jing Sun, Fang Kong, and Guodong Zhou. Building chinese dis- course corpus with connective-driven dependency tree structure. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP’14), pages 2105–2114, Doha, Qatar, October 2014b. Association for Computational Linguistics. URL http://www.aclweb.org/anthology/D14-1224.
[13]. Tomas Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean .Efficient Estimation of Word Representations in Vector Space. Computation and Language (cs.CL)
[14]. Piotr Bojanowski∗ and Edouard Grave∗ and Armand Joulin and Tomas Mikolov. Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics, vol. 5, pp. 135–146, 2017. Action Editor: Hinrich Schu ̈tze. Submission batch: 9/2016; Revision batch: 12/2016; Published 6/2017. 2017 Association for Computational Linguistics. Distributed under a CC-BY 4.0 license
[15]. Jeffrey Pennington, Richard Socher, Christopher D. Manning. GloVe: Global Vectors for Word Representation. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, October 25-29, 2014, Doha, Qatar. 2014 Association for Computational Linguistics
[16]. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.
[17]. Pitler, M. Raghupathy, H. Mehta, A. Nenkova, A. Lee, and A. Joshi. 2008. Easily identifiable discourse relations. In Proceedings of the 22nd International Conference on Computational Linguistics (COLING08), short paper.
[18]. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.
[19]. C. Goller and A. Kuchler. Learning task-dependent distributed representations by back- propagation through structure. In Proceedings of the 1996 IEEE International Conference on Neural Networks, volume 1, pages 347–352 vol.1, Jun 1996.
[20]. Chuan-An Lin. A Unified RvNN Framework for End-to-End Chinese Discourse Parsing. Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, pages 73–77 Santa Fe, New Mexico, USA, August 20-26, 2018.
[21]. David A. duVerle , Helmut Prendinger .A Novel Discourse Parser Based on Support Vector Machine Classification. Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 665–673, Suntec, Singapore, 2-7 August 2009. 2009 ACL and AFNLP.
40

[22]. Shafiq Joty, Giuseppe Carenini, Raymond Ng, and Yashar Mehdad. 2013. Combining intra- and multi-sentential rhetorical parsing for document-level discourse analysis. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL 2013), pages 486–496, Sofia, Bulgaria, August.
[23]. Jiwei Li, Rumeng Li and Eduard Hovy. Recursive Deep Models for Discourse Parsing. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2061–2069, October 25-29, 2014, Doha, Qatar. c©2014 Association for Computational Linguistics
[24]. Daniel H Younger. Recognition and parsing of context-free languages in time n3. Infor- mation and control, 10(2):189–208, 1967.
[25]. Samuel R. Bowman, Jon Gauthier, Abhinav Rastogi, Raghav Gupta, Christopher D. Man- ning, and Christopher Potts. A fast unified model for parsing and sentence understanding. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL’16), pages 1466–1477, August 2016. URL http://www.aclweb.org/ anthology/P16- 1139.
[26]. Miguel Ballesteros, Chris Dyer, and Noah A. Smith. 2015. Improved transition-based parsing by modeling characters instead of words with LSTMs. In EMNLP’15, Lisbon, Portugal. 349–359.
[27]. YANYAN JIA, YANSONG FENG, YUAN YE, CHAO LV. Improved Discourse Parsing with Two-Step Neural Transition-Based Model. ACM Trans. Asian Low-Resour. Lang. Inf. Process., Vol. 17, No. 2, Article 11. Publication date: January 2018.
[28]. Emily Pitler, Annie Louis, Ani Nenkova. Automatic sense prediction for implicit discourse relations in text, Proceedings of the 47th Annual Meeting of the ACL and the 4th IJCNLP of the AFNLP, pages 683ñ691, Suntec, Singapore, 2-7 August 2009. c©2009 ACL and AFNLP
[29]. D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” arXiv preprint arXiv:1409.0473, 2014.
[30]. Yao Zhou, Cong Liu, Yan Pan. Modelling Sentence Pairs with Tree-structured Attentive Encoder. 10 pages, 3 figures, COLING2016.
[31]. Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer. Improving Tree-LSTM with Tree Attention. 8 Pages, 3 figures, Accepted in The 13th IEEE International Conference on Semantic Computing (ICSC 2019)
[32]. Zhouhan Lin , Minwei Feng, Cicero Nogueira dos Santos, Mo Yu,Bing Xiang, Bowen Zhou& Yoshua Bengio. A S TRUCTURED S ELF-ATTENTIVE S ENTENCE EMBEDDING. Published as a conference paper at ICLR 2017.
41
[33]. Mahtab Ahmed, Muhammad Rifayat Samee, Robert E. Mercer . Improving Tree-LSTM with Tree Attention. 8 Pages, 3 figures, Accepted in The 13th IEEE International Conference on Semantic Computing (ICSC 2019)
[34]. Xiaomian Kang, Haoran Li, Long Zhou, Jiajun Zhang, and Chengqing Zong. An end-to-end chinese discourse parser with adaptation to explicit and non-explicit relation recognition. In Proceedings of the CoNLL-16 shared task, pages 27–32, August 2016. URL http://anthology.aclweb.org/K16-2003.
[35]. Lynn Carlson, Daniel Marcu, and Mary Ellen Okurowski. Building a discourse-tagged corpus in the framework of rhetorical structure theory. In Proceedings of the Second SIGdial Workshop on Discourse and Dialogue (SIGDIAL’01), pages 1–10, 2001 . URL https://doi.org/10.3115/1118078.1118083.
[36]. Jihun Choi, Kang Min Yoo, Sang-goo Lee. Learning to Compose Task-Specific Tree Structures. Copyright 2018, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved.
[37]. Maillard, J.; Clark, S.; and Yogatama, D. 2017. Jointly learning sentence embeddings and syntax with unsupervised Tree-LSTMs. arXiv preprint arXiv:1705.09189.

指導教授

張嘉惠(Chia-Hui Chang)

審核日期

2019-8-8

推文