使用強化學習於機械手臂在腹部之超音波掃描;Reinforcement Learning for Robotic Arm Ultrasound Scanning of the Abdomen

NCUIR > College of Electrical Engineering & Computer Science > Graduate Institute of Electrical Engineering > Electronic Thesis & Dissertation > Item 987654321/96344

Please use this identifier to cite or link to this item: http://ir.lib.ncu.edu.tw/handle/987654321/96344

Title:	使用強化學習於機械手臂在腹部之超音波掃描;Reinforcement Learning for Robotic Arm Ultrasound Scanning of the Abdomen
Authors:	陸柏崴;LU, Po-Wei
Contributors:	電機工程學系
Keywords:	機械手臂;逆向運動學;機器人運動學;強化學習;robotic arm;inverse kinematics;robot kinematics;reinforcement learning
Date:	2025-01-20
Issue Date:	2025-04-09 17:50:50 (UTC+8)
Publisher:	國立中央大學
Abstract:	本研究於使用強化式學習結合機械手臂，使其能夠在腹部上進行超音波掃描。首先使用深度攝影機進行三維建模，構建虛擬腹部模型，這些虛擬物體用於訓練機械手臂，使其能夠在不確定的表面上進行超音波掃描。在訓練過程中，根據UR5e機械手臂的Denavit-Hartenberg（DH）參數表進行逆向運動學控制，以精確地控制虛擬機械手臂的運動。強化學習方法採用PPO(近端優化策略)網路之學習方式，透過獎懲機制使虛擬機械手臂自主學習。在這個架構中，行動者（Actor）負責選擇動作，評論家（Critic）則評估選擇動作的價值，並通過獎懲機制來調整行動者的策略。這種方法使得機械手臂能夠在不確定的環境中不斷學習和改進其行動策略，以實現最佳的超音波掃描效果。在完成模型訓練後，使用Intel Realsense D435 深度攝影機對實際的表面物體進行拍攝，並將拍攝到的深度圖像導入 PyBullet 中進行建模，這樣可以替換訓練過程中使用的隨機虛擬物體，達到真實環境下的應用。在 PyBullet 中生成的物體模型上，應用訓練好的強化學習模型來獲得虛擬機械手臂掃描路徑的關節角度。這些關節角度通過 RTDE（Real-Time Data Exchange）通訊方式傳送給實際的UR5e機械手臂，使其能夠精確地復現虛擬機械手臂的動作，進行超音波掃描。在實際機械手臂的控制中，為了避免TCP（工具中心點）施加過大的力，使用了UR5e機械手臂內建的力感測系統，將施加的力量控制在一定範圍內，確保掃描過程中的安全性和準確性。實驗結果顯示，經過訓練的機械手臂能夠在腹部上進行精確的超音波掃描。此外，建模誤差僅為0.00000079 m²，掃描過程中的施力穩定維持在 9N 至 10N，平均為9.65N。在餘弦距離方面，平均餘弦距離為 0.0017，接近 0，表明超音波探頭與腹部法向量高度對齊。;This study integrates reinforcement learning with a robotic arm to enable ultrasound scanning on abdominal surfaces. A depth camera was first used for 3D modeling to construct virtual abdominal models. These virtual objects were utilized to train the robotic arm to perform ultrasound scans on uncertain surfaces. During the training process, inverse kinematics control was implemented based on the Denavit-Hartenberg (DH) parameter table of the UR5e robotic arm, ensuring precise control of the virtual robotic arm′s movements. The reinforcement learning method employed the Proximal Policy Optimization (PPO) network, where the virtual robotic arm autonomously learned through a reward mechanism. In this framework, the Actor selects actions, while the Critic evaluates the value of these actions and adjusts the Actor′s strategy through a reward mechanism. This approach enables the robotic arm to continuously learn and improve its strategies in uncertain environments, achieving optimal ultrasound scanning performance.After completing model training, the Intel Realsense D435 depth camera was used to capture depth images of actual surface objects, which were then imported into PyBullet for modeling. This allowed the replacement of the random virtual objects used during training, facilitating real-world applications. On the object models generated in PyBullet, the trained reinforcement learning model was applied to determine joint angles for the virtual robotic arm′s scanning path. These joint angles were transmitted to the physical UR5e robotic arm via the Real-Time Data Exchange (RTDE) communication protocol, enabling it to replicate the actions of the virtual robotic arm and perform ultrasound scanning.For the control of the actual robotic arm, the built-in force sensing system of the UR5e robotic arm was used to maintain the applied force within a safe range, preventing excessive force at the tool center point (TCP). This ensured safety and accuracy during the scanning process. The force control mechanism effectively prevented damage to the scanned objects due to overexertion while enhancing the accuracy and reliability of the scanning viii data.Experimental results demonstrated that the trained robotic arm could perform precise ultrasound scanning on various abdominal surfaces, even with uncertainties. Additionally, the modeling error was only 0.00000079 m², and the applied force during scanning was consistently maintained between 9N and 10N, with an average of 9.65N. Regarding cosine distance, the mean cosine distance was 0.0017, close to 0, indicating a high alignment of the ultrasound probe with the abdominal surface′s normal vectors.
Appears in Collections:	[Graduate Institute of Electrical Engineering] Electronic Thesis & Dissertation

Files in This Item:

File	Description	Size	Format
index.html		0Kb	HTML	21	View/Open

社群 sharing

Loading...