基於注意力機制的開放式對話系統

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：11

、訪客IP：18.218.172.249

姓名

黃俊杰(Jun-Jie Huang) 查詢紙本館藏

畢業系所

資訊管理學系

論文名稱

基於注意力機制的開放式對話系統

相關論文

★ 多重標籤文本分類之實證研究 : word embedding 與傳統技術之比較	★ 基於圖神經網路之網路協定關聯分析
★ 學習模態間及模態內之共用表示式	★ Hierarchical Classification and Regression with Feature Selection
★ 病徵應用於病患自撰日誌之情緒分析	★ 針對特定領域任務—基於常識的BERT模型之應用
★ 基於社群媒體使用者之硬體設備差異分析文本情緒強烈程度	★ 機器學習與特徵工程用於虛擬貨幣異常交易監控之成效討論
★ 捷運轉轍器應用長短期記憶網路與機器學習實現最佳維保時間提醒	★ 基於半監督式學習的網路流量分類
★ ERP日誌分析-以A公司為例	★ 企業資訊安全防護：網路封包蒐集分析與網路行為之探索性研究
★ 資料探勘技術在顧客關係管理之應用─以C銀行數位存款為例	★ 人臉圖片生成與增益之可用性與效率探討分析
★ 人工合成文本之資料增益於不平衡文字分類問題	★ 探討使用多面向方法在文字不平衡資料集之分類問題影響

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

近年許多研究提出各種基於神經網路的對話系統，但模擬對話仍然是對話生成領域中最棘手的挑戰之一。而大多對話系統與其相關研究仍採用基於RNN架構Seq2Seq模型，此外Transformer在Neural Machine Translation （NMT）領域上的表現遠超於基於RNN架構的Seq2Seq模型，但鮮少研究將基於RNN的Seq2Seq模型和Transformer模型在對話生成領域上進行評估和比較，且對話生成模型的評估方式仍然無法使用單一的評估基準來對模型的生成回應進行評估。
因此本研究會採用基於RNN的Seq2Seq模型和Transformer模型，並使用二種電影字幕及對話相關資料集Cornell Movie-Dialog Corpus和OpenSubtitles Corpus進行對模型進行訓練。因資料集的特性，本篇研究也將著重在於open-domain對話模型之上進行探討，並且使用多種量化分析指標和質性分析來證實二者模型架構對於open-domain對話生成領域中的合適性，並且探討各個對話評估方式的相依性和可靠性。
從量化分析和質性分析結果顯示，基於RNN的Seq2Seq模型合適回答較短且保守的回應，Transformer模型在回應的整體質量和預測能力比基於RNN的Seq2Seq模型較高，且擅長回答推論簡單的問題，以及比後者較能夠生成較長的回應。並在本研究中找出各項評估指標的相依關係。而期望在未來研究中，將Transformer模型導入和取代基於RNN的Seq2Seq模型在不同的模型的架構和任務當中，並且將本研究評估的流程導入未來的研究當中。

摘要(英)

In recent years, many studies have proposed many kinds of neural network-based dialogue systems, but analog dialogue is still one of the most difficult challenges in the field of dialogue generation. Most of the dialogue systems and related research still use the Seq2Seq model based on RNN architecture. In addition, Transformer performs much better in the field of Neural Machine Translation (NMT) than RNN-based Seq2Seq models, but few studies have evaluated and compared RNN-based Seq2Seq models and Transformer models in the field of dialog generation, and the way in which the dialog generation model is evaluated is still not able to use a single evaluation benchmark to evaluate the model′s generated response.
Therefore, this study will use RNN-based Seq2Seq model and Transformer model, and models were trained using two movie subtitles and conversation-related data sets, Cornell Movie-Dialog Corpus and OpenSubtitles Corpus. Due to the nature of the dataset, this study will also focus on the open-domain dialogue model and use a variety of quantitative analysis indicators and qualitative analysis to verify the suitability of the two model architectures in the open-domain dialog generation domain. And explore the interdependence and reliability of the various methods of dialogue evaluation.
From the results of quantitative analysis and qualitative analysis, the RNN-based Seq2Seq model is suitable for short answers and conservative responses. The overall quality and predictive power of the Transformer model is higher than that of the RNN-based Seq2Seq model, and it is good at answering simple inference questions and generating longer responses than the latter. In this study, we find the dependence of various evaluation indicators. It is expected that in the future research, the Transformer model will be introduced and replaced with the RNN-based Seq2Seq model in the architecture and tasks of different models, and the process of this research evaluation will be introduced into future research.

關鍵字(中)

★ 對話生成
★ Seq2Seq
★ 基於注意力機制模型
★ Transformer

關鍵字(英)

★ Dialogue generation
★ Seq2Seq
★ Attention based Models
★ Transformer

論文目次

中文摘要 I
Abstract II
目錄 IV
圖目錄 VII
表目錄 VIII
1. 緒論 1
1.1. 研究背景 1
1.2. 研究動機 1
1.3. 研究目的 2
1.4. 論文架構 2
2. 對話系統模型與相關研究 4
2.1. Encoder-Decoder模型 4
2.1.1. 遞歸神經網路（Recurrent Neural Networks） 4
2.1.2. Sequence-to-sequence（Seq2seq）模型 5
2.1.3. Seq2Seq模型與相關研究 7
2.2. Attention based Models 11
2.2.1. Attention機制 11
2.2.2. Transformer模型 14
2.2.3. Transformer模型之相關研究 17
2.3. 評估方法 20
2.3.1. Word overlap-based metrics – BLEU 20
2.3.2. Perplexity 21
2.3.3. Embedding-based metrics 21
2.3.4. 基於神經網路的自動評估 22
2.3.5. Human Evaluation 25
2.3.6. 對話生成之評估方法綜合比較 25
2.4. 綜合討論 29
3. 實驗方法 30
3.1. Datasets 30
3.1.1. Cornell Movie-Dialog Corpus 30
3.1.2. OpenSubtitles Corpus 31
3.2. 方法及流程 32
3.2.1. Preprocessing 32
3.2.2. 基於RNN的Seq2Seq模型建模 33
3.2.3. Transformer模型建模 33
3.2.4. 模型評估之指標 33
3.3. 實驗：模型超參數設置對於生成對話品質之影響 34
3.3.1. 基於RNN的Seq2Seq模型設置 34
3.3.2. Transformer 模型設置 35
4. 結果與分析 37
4.1. 量化分析 37
4.1.1. BLEU 37
4.1.2. Perplexity 42
4.1.3. Embedding-based metrics 42
4.1.4. RUBER 43
4.1.5. 人工問卷評估 43
4.1.6. 量化之綜合分析 44
4.2. 質性分析 47
4.2.1. 對話或問題的類別對模型生成回應的品質影響 47
4.2.2. 質性之綜合分析 49
4.3. 綜合分析 49
4.3.1. 單雙向LSTM在Seq2Seq模型 50
4.3.2. 基於RNN的Seq2Seq模型和Transformer模型 50
4.3.3. 各個評估方式之間的相依性和可靠性 51
5. 總結 52
5.1. 結論 52
5.2. 實驗貢獻 52
5.3. 未來展望 53
6. Reference 54
7. 附錄 58

參考文獻

Bahdanau, D., Cho, K., Bengio, Y., 2014. Neural Machine Translation by Jointly Learning to Align and Translate. ArXiv14090473 Cs Stat.
Bottou, L., 2010. Large-Scale Machine Learning with Stochastic Gradient Descent. Proc. COMPSTAT p177-186.
Cheng, J., Dong, L., Lapata, M., 2016. Long Short-Term Memory-Networks for Machine Reading. ArXiv160106733 Cs.
Cho, K., van Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., Bengio, Y., 2014. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation. ArXiv14061078 Cs Stat.
Cornell Movie-Dialogs Corpus [WWW Document], 2019. URL https://www.cs.cornell.edu/~cristian/Cornell_Movie-Dialogs_Corpus.html (accessed 3.29.19).
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. V., & Salakhutdinov, R. 2019. Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context. ArXiv:1901.02860 Cs Stat.
Danescu-Niculescu-Mizil, C., Lee, L., 2010. Chameleons in Imagined Conversations: A New Approach to Understanding Coordination of Linguistic Style in Dialogs. Association for Computational Linguistics.
Devlin, J., Chang, M.-W., Lee, K., & Toutanova, K. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. ArXiv:1810.04805 Cs.
Dong, J., Huang, J., 2018. Enhance word representation for out-of-vocabulary on Ubuntu dialogue corpus. ArXiv180202614 Cs.
Edunov, S., Ott, M., Auli, M., Grangier, D., 2018. Understanding Back-Translation at Scale. ArXiv180809381 Cs.
Gehring, J., Auli, M., Grangier, D., Yarats, D., Dauphin, Y.N., 2017. Convolutional Sequence to Sequence Learning, in: International Conference on Machine Learning.
Ghazvininejad, M., Brockett, C., Chang, M.-W., Dolan, B., Gao, J., Yih, W., Galley, M., 2018. A Knowledge-Grounded Neural Conversation Model, in: Thirty-Second AAAI Conference on Artificial Intelligenc.
Graves, A., Wayne, G., Danihelka, I., 2014. Neural Turing Machines. ArXiv14105401 Cs.
Hahnloser, R.H.R., Sarpeshkar, R., Mahowald, M.A., Douglas, R.J., Seung, H.S., 2000. Digital selection and analogue amplification coexist in a cortex-inspired silicon circuit. Nature 405, 947–951.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep Residual Learning for Image Recognition. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778.
Hochreiter, S., 1998. The Vanishing Gradient Problem During Learning Recurrent Neural Nets and Problem Solutions. Int. J. Uncertain. Fuzziness Knowl.-Based Syst. 06, 107–116.
Hochreiter, S., Schmidhuber, J., 1997. Long Short-Term Memory [WWW Document]. URL https://www.mitpressjournals.org/doi/abs/10.1162/neco.1997.9.8.1735 (accessed 3.18.19).
Kai Lempinen, 2017. What are Chatbots and how they impact Service Management. URL http://www.lempinenpartners.com/what-are-chatbots-and-how-they-impact-service-management/ (accessed 3.28.19).
Konstas, I., Iyer, S., Yatskar, M., Choi, Y., Zettlemoyer, L., 2017. Neural AMR: Sequence-to-Sequence Models for Parsing and Generation. ArXiv170408381 Cs.
Li, J.; Galley, M.; Brockett, C.; Spithourakis, G.; Gao, J.; and Dolan, W. B. 2016a. A persona-based neural conversation model. In: ACL, pp. 994–1003.
Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., Gao, J., 2016b. Deep Reinforcement Learning for Dialogue Generation, in: Proceedings of the Conference on Empirical Methods in Natural Language Processing. pp. 1192–1202.
Lison, P., Tiedemann, J., 2016. OpenSubtitles2016: Extracting Large Parallel Corpora from Movie and TV Subtitles 7.
Liu, C.-W., Lowe, R., Serban, I., Noseworthy, M., Charlin, L., Pineau, J., 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. Presented at the Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing.
Liu, Y., Bi, W., Liu, X., Shi, S., Zhang, H., 2018. Rethinking Neural Dialogue Generation: A Practical Guide. ResearchGate.
Lowe, R., Noseworthy, M., Serban, I.V., Angelard-Gontier, N., Bengio, Y., Pineau, J., 2017. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses, in: Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. pp. 1116–1126.
Lowe, R., Pow, N., Serban, I., Pineau, J., 2015. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems, in: Proceedings of the 16th Annual Meeting of the Special Interest Group on Discourse and Dialogue. Association for Computational Linguistics.
Luong, M.-T., Pham, H., Manning, C.D., 2015. Effective Approaches to Attention-based Neural Machine Translation. ArXiv150804025 Cs.
Luong, T., Pham, H., Manning, C.D., 2015a. Effective Approaches to Attention-based Neural Machine Translation. Association for Computational Linguistics, Lisbon, Portugal.
Luong, T., Sutskever, I., Le, Q., Vinyals, O., Zaremba, W., 2015b. Addressing the Rare Word Problem in Neural Machine Translation. Association for Computational Linguistics.
Manning, C.D., Schiitze, H., 1999. Foundations of Statistical Natural Language Processing 704.
Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., 2013. Distributed Representations of Words and Phrases and their Compositionality, in: Burges, C.J.C., Bottou, L., Welling, M., Ghahramani, Z., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems 26. Curran Associates, Inc.
Nallapati, R., Zhou, B., dos Santos, C., Gulcehre, C., Xiang, B., 2016. Abstractive Text Summarization using Sequence-to-sequence RNNs and Beyond, in: Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. Association for Computational Linguistics.
OpenSubtitles [WWW Document], 2019. URL http://opus.nlpl.eu/OpenSubtitles.php (accessed 3.29.19).
Ott, M., Edunov, S., Grangier, D., Auli, M., 2018. Scaling Neural Machine Translation. Association for Computational Linguistics.
Papineni, K., Roukos, S., Ward, T., Zhu, W.-J., 2002. BLEU: a Method for Automatic Evaluation of Machine Translation. Association for Computational Linguistics.
Rumelhart, D.E., Hinton, G.E., Williams, R.J., 1988. Neurocomputing: Foundations of Research, in: Anderson, J.A., Rosenfeld, E. (Eds.), . MIT Press, pp. 696–699.
Schuster, M., Paliwal, K.K., 1997. Bidirectional recurrent neural networks. IEEE Trans. Signal Process. 45.
Serban, I.V., Sordoni, A., Bengio, Y., Courville, A., Pineau, J., 2016. Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models. AAAI6.
Serban, I.V., Sordoni, A., Lowe, R., Charlin, L., Pineau, J., Courville, A., Bengio, Y., 2017. A Hierarchical Latent Variable Encoder-Decoder Model for Generating Dialogues. AAAI 7.
Song, Y., Yan, R., Li, X., Zhao, D., Zhang, M., 2016. Two are Better than One: An Ensemble of Retrieval- and Generation-Based Dialog Systems. ArXiv161007149 Cs.
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., Salakhutdinov, R., 2014. Dropout: A Simple Way to Prevent Neural Networks from Overﬁtting. J. Mach. Learn. Res. 30.
Sutskever, I., Vinyals, O., Le, Q.V., 2014. Sequence to Sequence Learning with Neural Networks, in: Ghahramani, Z., Welling, M., Cortes, C., Lawrence, N.D., Weinberger, K.Q. (Eds.), Advances in Neural Information Processing Systems. Curran Associates, Inc., pp. 3104–3112.
Tang, G., Müller, M., Rios, A., Sennrich, R., 2018. Why Self-Attention? A Targeted Evaluation of Neural Machine Translation Architectures, in: Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Brussels, Belgium, pp. 4263–4272.
Tao, C., Gao, S., Shang, M., Wu, W., Zhao, D., Yan, 2018a. Get the point of my utterance! learning towards effective responses with multi-head attention mechanism. In: IJCAI 2018
Tao, C., Mou, L., Zhao, D., Yan, R., 2018b. RUBER: An Unsupervised Method for Automatic Evaluation of Open-Domain Dialog Systems, In: AAAI 2018, pp. 722–729
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I., 2017. Attention is All you Need, Advances in Neural Information Processing Systems 30. Curran Associates, Inc., pp. 5998–6008.
Vinyals, O., Le, Q., 2015. A Neural Conversational Model. ArXiv150605869 Cs.
Weng, L., 2018. Attention? Attention! [WWW Document]. URL https://lilianweng.github.io/lil-log/2018/06/24/attention-attention.html#summary (accessed 3.19.19).
Xing, C., Wu, Y., Wu, W., Huang, Y., Zhou, M., 2018. Hierarchical Recurrent Attention Network for Response Generation, in: Thirty-Second AAAI Conference on Artificial Intelligence. Presented at the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI.
Xu, K., Lei, J., Kiros, R., Cho, K., Courville, A., Salakhutdinov, R., Zemel, R.S., Bengio, Y., 2015. Show, Attend and Tell: Neural Image CaptionGeneration with Visual Attention 10.
Yao, K., Zweig, G., Peng, B., 2015. Attention with Intention for a Neural Network Conversation Model. ArXiv151008565 Cs.
Zhao, T., Lu, A., Lee, K., Eskenazi, M., 2017. Generative Encoder-Decoder Models for Task-Oriented Spoken Dialog Systems with Chatting Capability. ArXiv170608476 Cs.

指導教授

柯士文

審核日期

2019-7-23

推文