| 摘要: | 隨著企業對資訊科技服務管理自動化需求日增，AIOps已成為降低系統中斷與縮短事件處理時間的關鍵技術；然而，現行 IT 工單分類流程高度仰賴人工判讀，且資料類別分布不均，常導致分類效率與準確度下降。為填補此研究空缺，本文以兩組公開 IT 工單資料集為基礎，探討混合式自然語言處理模型結合資料平衡技術於多類別分類任務中的應用成效。
研究設計採用 Stratified 5-Fold 交叉驗證，建構並比較三種主流 Transformer 模型（BERT、RoBERTa、DeBERTa）與其分別結合 CNN、LSTM、BiLSTM 等下游架構之混合模型，並引入隨機過採樣、隨機欠採樣與三類資料增強技術（Word2Vec、EDA、T5），評估其於Macro F1 Score、MCC等指標下的分類表現。
實驗結果顯示，在 IT Ticket Classification 資料集中，BERT-CNN 模型即使未使用資料平衡技術，亦展現出最佳整體效能（Macro F1 Score 達 0.408）；使用BERT-BiLSTM導入資料平衡策略後，其表現可再提升約 2%。在 IT Service Ticket Classification Dataset 資料集中，BERT-CNN 模型達到 Macro F1 Score 0.866，顯示其在不同資料集間具備良好的穩健性與泛化能力。整體而言，若能適當結合混合架構與資料平衡策略，將有助於提升不平衡 IT 工單多類別分類任務之整體效能與模型穩定性；而在 AIOps 流程前端導入自動分類機制，亦顯示出潛在應用價值，不僅可提升事件判讀效率，更可作為企業推動智慧化 ITSM 的可行技術參考。
;With the growing demand for automation in IT Service Management (ITSM), AIOps has emerged as a critical technology to reduce system downtime and shorten incident resolution time. However, current IT ticket classification processes still heavily rely on manual interpretation, and the uneven distribution of ticket categories often leads to inefficiencies and decreased classification accuracy. To address this research gap, this study investigates the effectiveness of hybrid natural language processing (NLP) models combined with data balancing techniques for multi-class classification tasks, using two public IT ticket datasets.
The research design adopts Stratified 5-Fold Cross-Validation to construct and compare three mainstream Transformer models (BERT, RoBERTa, and DeBERTa) and their hybrid architectures incorporating CNN, LSTM, and BiLSTM. It also integrates random oversampling, random undersampling, and three data augmentation techniques (Word2Vec, EDA, and T5) to evaluate model performance using Macro F1 Score and Matthews Correlation Coefficient (MCC).
Experimental results show that on the IT Ticket Classification dataset, the BERT-CNN model achieved the best overall performance (Macro F1 Score of 0.408) even without applying data balancing techniques. When data balancing was introduced, the BERT-BiLSTM model exhibited a further improvement of approximately 2%. On the IT Service Ticket Classification Dataset, the BERT-CNN model attained a Macro F1 Score of 0.866, demonstrating its robustness and generalizability across datasets. Overall, the combination of hybrid architectures and appropriate data balancing strategies can enhance the performance and stability of multi-class classification tasks for imbalanced IT tickets. Moreover, integrating automated classification mechanisms at the front end of the AIOps workflow reveals promising application potential, not only improving incident interpretation efficiency but also serving as a viable technical reference for enterprises promoting intelligent ITSM. |
