| 摘要: | 憂鬱症為當前社會嚴重的心理健康問題，傳統識別方法存在主觀性與即時性不足的限制。本研究提出一個多模態情緒識別系統，整合語音（SVM）、視覺（CNN）、文字（BERT）三模態，實現異常情緒的即時識別與早期憂鬱症篩檢，涵蓋七種情緒類別（Angry、Calm、Disgust、Fear、Happy、Neutral、Sad）。相較於傳統單模態方法，本系統透過跨模態資訊學習與深度融合，在非侵入式、即時互動的環境下預測憂鬱傾向，實驗結果顯示在平衡資料集上的分類準確率達 90%。
為提升臨床適用性，系統以 PHQ-8 分數切點（≥10）作為 Ground Truth，將三模態機率分佈串接為 21 維向量，透過 CNN 進行二元分類，縮小了臨床評估與 AI 模型的落差。針對長照機構的資源限制，系統採用晚期融合策略與 ESP32-CAM 等邊緣設備，實現低成本部署與輕量化運算，確保即時性與可擴充性。整體設計導入 MIAT 方法論，結合 IDEF0、GRAFCET 與 HLS，提供結構化、可追蹤的醫療 AI 建模範式，支撐了多模態系統的開發與應用。本研究為長照機構的心理健康監測提供了高效且實用的解決方案，特別適用於資源受限環境。
系統能即時識別長者的異常情緒並預測憂鬱傾向，協助照護人員進行早期介入，同時其低成本部署與邊緣運算設計顯著降低了導入門檻，展現了在長照場景中的應用價值。
;Depression is a severe mental health issue in contemporary society, with traditional recognition methods suffering from subjectivity and lack of real-time capability. This study proposes a multimodal emotion recognition system integrating audio (SVM), visual (CNN), and textual (BERT) modalities to enable real-time identification of abnormal emotions and early depression screening, covering seven emotion categories (Angry, Calm, Disgust, Fear, Happy, Neutral, Sad). Compared to single-modality approaches, the system leverages cross-modal learning and deep fusion to predict depression tendencies in non-invasive, real-time interactive settings, achieving a classification accuracy of 90% on balanced datasets.
To enhance clinical applicability, the system uses PHQ-8 score cutoffs (≥10) as Ground Truth, integrating multimodal probability outputs into a 21-dimensional vector for binary classification via CNN, bridging the gap between clinical assessment and AI models. Tailored for long-term care facilities, it employs late fusion and edge devices like ESP32-CAM for low-cost deployment and lightweight computation, ensuring real-time performance and scalability. The design incorporates the MIAT methodology, utilizing IDEF0, GRAFCET, and HLS to provide a structured, traceable paradigm for medical AI system development, supporting the multimodal framework. This system offers an efficient and practical solution for mental health monitoring in long-term care settings, particularly in resource-constrained environments. It enables real-time detection of abnormal emotions in the elderly and predicts depression tendencies, aiding caregivers in early intervention, while its low-cost deployment and edge computing design significantly lower adoption barriers, demonstrating strong practical value in long-term care scenarios |
