句子嵌入 (Sentence Embeddings) 在自然語言理解(Natural Language Understanding, NLU)扮演重要的角色。預訓練模型 (例如: BERT、RoBERTa) 會將原始句子轉換成句子嵌入，並將該嵌入 (Embeddings) 套用在多個NLU任務中有顯著的效果提升，但在語意文本相似性（Semantic Textual, Similarity, STS）任務上表現卻不如預期。先前的研究發現BERT和RoBERTa模型對單詞順序具有敏感性。為此，我們提出了一種名為StructCSE的方法，它將學習單詞順序 (word order) 和語意訊息 (semantic information) 相結合，以增強對比學習的句子嵌入。 在實驗中，我們分別使用了語意文本相似性 (STS) 以及遷移學習 (Transfer Learning) 任務來驗證 StructCSE的有效性。根據實驗結果，在大部分的資料集StructCSE有與過去作法相抗衡或優於基線模型的表現，其中發現StructCSE在以BERT為基礎模型時，在STS任務有顯著的進步，而遷移學習至情感分析的子任務中，StructCSE有突出的表現。 ;Sentence embeddings play an important role in Natural language understanding (NLU). Pretrained models like BERT and RoBERTa encode input sentences into embeddings, which significantly enhances performance across multiple NLU tasks. However, they underperform in the task of semantic textual similarity (STS). Previous research has found that BERT and RoBERTa are sensitive to the word ordering. In response, we propose StructCSE, a method that incorporates word order and semantic information to enhance contrastive learning of sentence embeddings. In our experiments, we evaluate the effectiveness of StructCSE using STS and transfer learning tasks. The results demonstrate that StructCSE performs competitively or outperforms baseline models on most datasets. Especially, with fine-tuning BERT, StructCSE achieves better performance in the STS tasks and exhibits outstanding performance in the sentiment analysis subtasks of transfer learning.
