隨著處理長序列數據需求的增加，許多研究集中於提高模型性能和效率。在這項研究中，我們提出了一種基於 Transformer 架構的高效模型——Butterflyer。Butterflyer 旨在處理圖像、文本、路徑數據和數學運算等長序列數據。我們引入了一種創新的 Butterfly Attention 機制，該機制利用 Butterfly 矩陣的計算來替代傳統的自注意力機制，從而計算和捕捉不同的交互模式，提高了模型的性能。通過採用 Sophia 優化器，我們進一步提升了 Butterflyer 的訓練效率和性能。我們使用 Long Range Arena (LRA)基準數據集來評估 Butterflyer 的性能。實驗結果顯示，Butterflyer 在各種應用中表現出色，尤其在文本分類、圖像分類和文檔檢索等任務中，超越了目前最先進的模型。 ;With the increasing demand for handling long sequential data, many studies have focused on improving model performance and efficiency. In this research, we propose Butterflyer, an efficient model based on the Transformer architecture. Butterflyer is designed to process long sequential data such as images, text, path data, and mathematical operations. We introduce an innovative Butterfly-Attention mech anism that replaces the traditional self-attention mechanism, utilizing Butterfly Matrices to compute and capture different interaction patterns, thereby enhancing computational efficiency and model performance. By employing the Sophia opti mizer, we further improve the training efficiency and performance of Butterflyer. We evaluate Butterflyer’s performance using the Long Range Arena (LRA) bench mark dataset. Experimental results show that Butterflyer demonstrates superior performance across various applications, particularly in tasks such as text classifi cation, image classification, and document retrieval, outperforming state-of-the-art models.
