| 摘要: | 在當今時代，遠端機器人超音波系統(Robotic Ultrasound System, RUS)的研究正在 蓬勃發展，特別是對於控制與視覺處理等領域備受關注。本研究應用現有的深度學習 技術來消除操作視野中的遮蔽物，首先攝影機的即時串流影像會透過我們訓練的 YOLOv8實例分割模型偵測是否有障礙物出現，並對障礙物區域進行遮罩消除，接著 透過現有的視訊修復模型（Decoupled Spatial-Temporal Transformer for Video Inpainting, DSTT）對消除的缺失區域進行影像修復，DSTT模型將根據過去的影像計算Attention 值來填補當前影像缺失區域，對於最初的影像序列我們會插入沒有障礙物在內的初始 影像幫助後續的修復工作，如此來達成消除遮蔽物的目標。為了實現自動掃描減少人 員操作，我們利用Intel深度相機對待檢測物體做表面輪廓的三維建模，建模結果會發 送至強化學習的虛擬環境進行模擬，也會傳送回操作端的圖形使用者介面（Graphical User Interface, GUI）視窗，操作者可以透過GUI來選取欲掃描之終點，選取後在強化 學習的虛擬環境進行模擬掃描，確認路徑與安全後，發送模擬結果的關節角數據給機 械手臂執行相應的掃描動作。實驗結果驗證三維建模對不同形狀物體（平面、斜面、 球體）的適用範圍，平面與斜面深度誤差小於0.041公分，面積誤差小於3.3%，球體 部分在切線角度小於25.6的深度誤差小於為0.17公分。在即時操作影像中，當遮蔽物 的11%進入即可被偵測，並且消除於操作者即時影像中，後續實驗結果顯示，我們將 初始影像插入影序列使PSNR與SSIM變的更好。本研究將現有的深度學習技術應用 於遠端RUS，成功解決了臨床上遮蔽物阻擋視野和掃描自動化等問題，提升了系統的 效率與安全性。;In the current era, research on Remote Robotic Ultrasound Systems (RUS) is flourishing, particularly in areas concerning control and visual processing. This study employs existing deep learning techniques to eliminate occlusions in the operating field of view. First, the real time streaming images from the camera are processed by our trained YOLOv8 instance segmentation model to detect whether any obstacles appear, and the obstacles in the detected regions are masked out. Next, the missing regions caused by the removal of these obstacles are restored using a state-of-the-art video inpainting model called Decoupled Spatial Temporal Transformer (DSTT). The DSTT model calculates an attention value based on previous frames to fill in the missing regions in the current frame. For the initial image sequence, we insert frames without any obstacle to facilitate subsequent inpainting, thereby achieving the goal of removing occlusions. In order to realize automated scanning and reduce manual operation, we use an Intel depth camera to perform 3D surface modeling of the object to be inspected. The modeling results are transmitted to a reinforcement learning environment for simulation, as well as sent back to the operator’s Graphical User Interface (GUI). Through the GUI, the operator selects the desired endpoint for scanning. The scanning process is then simulated within the reinforcement learning environment to validate the path and ensure safety. Once validated, the joint angle data from the simulation is sent to the robotic arm to perform the corresponding scanning action. Experimental results confirm that the 3D modeling approach is applicable to objects of different shapes (planar, inclined, spherical). For planar and inclined surfaces, the depth error is less than 0.041 cm, and the area error is under 3.3%. For spheres, at tangential angles below 25.6°, the depth error is less than 0.17 cm. In the real-time operational images, once 11% of the occlusion appears, it can be detected and removed from the operator’s live view. Further experiments demonstrate that inserting an obstacle-free initial image into the sequence improves both PSNR and SSIM. This study applies existing deep learning technologies to remote RUS, successfully addressing clinical challenges related to occlusion and automated scanning, thereby enhancing both efficiency and safety of the system. |
