基於深度學習之六軸機械手臂應用於臉部追蹤

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：107

、訪客IP：18.226.251.74

姓名

李柏毅(Bo-Yi Li) 查詢紙本館藏

畢業系所

電機工程學系

論文名稱

基於深度學習之六軸機械手臂應用於臉部追蹤

相關論文

★ 直接甲醇燃料電池混合供電系統之控制研究	★ 利用折射率檢測法在水耕植物之水質檢測研究
★ DSP主控之模型車自動導控系統	★ 旋轉式倒單擺動作控制之再設計
★ 高速公路上下匝道燈號之模糊控制決策	★ 模糊集合之模糊度探討
★ 雙質量彈簧連結系統運動控制性能之再改良	★ 桌上曲棍球之影像視覺系統
★ 桌上曲棍球之機器人攻防控制	★ 模型直昇機姿態控制
★ 模糊控制系統的穩定性分析及設計	★ 門禁監控即時辨識系統
★ 桌上曲棍球：人與機械手對打	★ 麻將牌辨識系統
★ 相關誤差神經網路之應用於輻射量測植被和土壤含水量	★ 三節式機器人之站立控制

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

本論文的目的為藉由導入機器視覺與深度學習網路，控制六軸機械手臂與人臉保持固定距離且能對準額頭中央，並在有限範圍內主動跟隨人臉移動。此追蹤系統可應用於人與機器人的互動任務，在本論文中則是以考慮該手臂握持手機或平板與人臉保持上述距離與方向，即使人臉有些許移動或轉動，仍然可以追蹤人臉並維持設定好的距離與方向，方便使用者可以不用自己手持也能看手機或平板。為達成此目的，機械手臂端需經由人臉偵測與辨識臉部特徵，計算人臉與機械手臂末端的相對位置，並由逆向運動學完成人臉的追蹤任務。
在人臉偵測與臉部特徵辨識部分，此二功能皆透過深度學習達成，並且完成(1)取得面部特徵在影像中的位置；(2)計算特徵點與攝影機的相對位置。另外機械手臂的運動控制亦需以下程序。(1)建置虛擬環境；(2)計算模型的轉換矩陣；(3)求得目標點的座標，並以逆運動學控制機器手臂到目標點。綜合上述條件，便可在機械手臂自身機構限制內完成追蹤臉部任務。除了影像辨識與機械手臂的控制外，本研究也添加了手機應用程式供使用者微調機械手臂的位置，可透過手動或語音輸入來調整機械手臂進行上下左右偏移，亦可於手機上同步顯示當前機械手臂上的攝影機所拍攝的影像。本研究亦將此技術應用於目前疫情期間(2020 COVID-19)的自動額溫量測，當人靠近機械手臂前，便能偵測該人的臉部位置，並將測溫模組對準額頭為使用者量測額溫，若有發燒將會出現警示，如此便可免除人工量測的負擔。
本研究在Linux環境下使用機器人作業系統(Robot Operating System, 以下皆簡稱ROS)，由於ROS能共享資訊的特點，便可透過節點與節點間訊息的發送與接收，完成系統中不同軟硬體間的資訊溝通。本研究藉由ROS完成了筆記型電腦、六軸機械手臂、攝影機以及手機程式間資訊的傳遞與彙整，實現軟硬體協同應用。

摘要(英)

The purpose of this thesis is to control the six-axis robotic arm to keep a fixed distance from the human face and aim at the center of the forehead with the aids of computer vision and deep learning. Furthermore, when the human face has a little movement, the robot can also follow it within the limited range. This tracking system can be applied to interactive tasks between humans and robots. In this thesis, it is considered that the arm holding a mobile phone or tablet maintains the above distance and direction to face the user’s face. Even if the face moves or rotates a little, it can still track the face with the set distance and direction, so that the user can watch the mobile phone or tablet without holding them. To achieve this, the robotic arm needs to detect the human face, recognize the facial features, and calculate the related positions between the face and the terminal of robotic arm. Finally, the robotic arm will implement the face tracking by using inverse kinematics.
In face detection and facial features recognition, both are achieved by deep learning technique, after that we also (1) obtain the position of facial features points in the image and (2) calculate the related positions between the facial features and the camera. In addition, the movement control of the robotic arm also needs the following process. (1) building a virtual environment; (2) calculating the transfer matrix of the model; (3) finding the coordinates of the target point and using inverse kinematics to control the robotic arm moving to the target point. This study also added a function in the mobile phone to fine-tune the pose of the robotic arm. The fine tune operation can be implemented by manual input or voice input. The mobile phone can also display the images taken by the current camera on the robot arm simultaneously. This technology can also be applied to the automatic forehead temperature measurement during the current epidemic situation (2020 COVID-19).
In this thesis, Robot Operating System (ROS) is used in the Linux operating system and completed the transmission and integration of information between laptop, robotic arm, camera, and mobile phone.
Keywords: Face detection, Facial landmark, 6 DOF robotic arm, Kinematics, Robot operating system, Coordinate transformation, Face tracking.

關鍵字(中)

★ 臉部偵測
★ 臉部特徵點辨識
★ 六軸機械手臂
★ 運動學
★ ROS
★ 座標轉換
★ 臉部追蹤

關鍵字(英)

★ Face detection
★ Facial landmark
★ 6 DOF robotic arm
★ Kinematics
★ Robot operating system
★ Coordinate transformation
★ Face tracking

論文目次

摘要 i
Abstract i
致謝 iii
目錄 iv
圖目錄 vi
表目錄 ix
第一章緒論 1
1.1研究背景與動機 1
1.2文獻回顧 2
1.3論文目標 3
1.4論文架構 4
第二章系統架構與軟硬體介紹 5
2.1系統架構 5
2.2硬體架構 5
2.4軟體介紹 11
2.4軟體介紹 11
2.4.1 ROS簡介 11
2.4.2 Moveit 14
第三章臉部偵測與特徵辨識 16
3.1臉部偵測 16
3.1.1深度學習網路 16
3.1.2網路的輸入輸出與後處理 18
3.1.3臉部偵測流程 20
3.2臉部特徵點辨識 21
3.2.1深度學習網路 21
3.2.2網路的輸入輸出與後處理 22
3.2.3臉部特徵點辨識流程 23
3.3透視投影法與臉部姿態估測 23
第四章機械手臂運動學與應用 28
4.1轉換矩陣 29
4.2正向運動學 30
4.3逆向運動學 32
4.4座標轉換 34
4.5手機控制端 36
4.5.1同步顯示影像 36
4.5.2 可調控的功能介紹 38
4.6 應用於額溫量測 41
4.7機器人作業系統的應用 43
4.7.1 ROS節點功能說明 43
4.7.2實驗節點與主題流程 47
第五章實驗結果 49
5.1虛擬工作環境建置 49
5.2攝影機影像修正 49
5.3 RGB-D攝影機誤差測量 53
5.4臉部追蹤結果 55
第六章結論與未來展望 58
6.1結論 58
6.2未來展望 58
參考文獻 60

參考文獻

[1] TCS 臺灣傳播調查資料庫-台灣民眾手機使用習慣調查
http://www.crctaiwan.nctu.edu.tw/epaper/%E7%AC%AC90%E6%9C%9F2
0190815.htm，2020 年 5 月。
[2] goron smart plus
https://search.rakuten.co.jp/search/mall/goron+smart+plus/ ， 2020 年 5
月。
[3] Second generation Italian smart bed
https://www.hi-interiors.com/hi-bed/，2020 年 5 月。
[4] Gaming beds
https://www.bauhutte.jp/bauhutte-life/making-gaming-desk/ ， 2020 年 5
月。
[5] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich feature hierarchies
for accurate object detection and semantic segmentation," Proc. IEEE
Conference Computer Vision and Pattern Recognition, Columbus, Jun. 2014,
pp. 580-587.
[6] R. B. Girshick, "Fast R-CNN," Proc. International Conference on Computer
Vision Pattern Recognition, Santiago, Dec. 2015, pp. 1440-1448.
[7] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: towards real time
object detection with region proposal networks," Proc. IEEE Transactions
on Pattern Analysis Machine Intelligence, 2017, vol. 39, pp. 1137-1149.
[8] J. Redmon, S. Divvala, R. Girshick and A. Farhadi, "You only look once:
unified, real-time object detection," arXiv preprint, 2015, arXiv:1506.02640.
[9] W. Liu, et al., "SSD: single shot multiBox detector," Lecture Notes in
Computer Science, 2016, pp. 21–37.
[10]R. C. Luo, T. W. Lin, and Y. H. Tsai, "Analytical inverse kinematic solution
for modularized 7-DOF redundant manipulators with offsets at shoulder and
wrist," in Proc. IEEE/RSJ International Conference on Intelligent Robots
and System, Chicago, Sep. 2014, pp. 516-521.
[11]A. Khan, C. Xiangming, Z. Xingxing and W. L. Quan, "Closed form inverse
kinematics solution for 6-DOF underwater manipulator," International
Conference on Fluid Power and Mechatronics (FPM), Harbin, 2015, pp.
1171-1176.
61
[12]W. Chen, C. Chen, F. Lee and Y. Kung, "Digital hardware implementation
of the forward/inverse kinematics for a SCARA robot manipulator," IEEE
International Conference on Applied System Invention (ICASI), Chiba, 2018,
pp. 54-57.
[13]P. Beeson and B. Ames, "TRAC-IK: An open-source library for improved
solving of generic inverse kinematics," IEEE-RAS 15th International
Conference on Humanoid Robots (Humanoids), Seoul, 2015, pp. 928-935.
[14]S. Kumar, N. Sukavanam and R. Balasubramanian, "An optimization
approach to solve the inverse kinematics of redundant manipulator,
" International Journal of Information and System Sciences (Institute for
Scientific Computing and Information), 2010, vol. 6, no. 4, pp. 414-423.
[15]J. Vannoy and J. Xiao, "Real-time adaptive motion planning (RAMP) of
mobile manipulators in dynamic environments with unforeseen changes," in
Proc. IEEE Transactions on Robotics, Oct. 2008, vol. 24, pp. 1199-1212.
[16]J. Chaichawananit and S. Saiyod, "Solving inverse kinematics problem of
robot arm based on a-star algorithm," International Joint Conference on
Computer Science and Software Engineering (JCSSE), Khon Kaen, 2016,
pp. 1-6.
[17]H. Li, Z. Song, Z. Jiang, Y. Mo and W. Ni, "Solving inverse kinematics of
industrial robot based on bp neural network," IEEE 7th Annual
International Conference on CYBER Technology in Automation, Control,
and Intelligent Systems (CYBER), Honolulu, 2017, pp. 1167-1171.
[18]S. Klemm, J. Oberlander, A. Hermann, A. Roennau, T. Schamm, J. M. ¨
Zollner, and R. Dillmann, "RRT*-connect: faster, asymptotically optimal
motion planning," Proc. IEEE International Conference on Robotics and
Biomimetics, Zhuhai, Dec. 2015, pp. 1670-1677.
[19]J. Demby’s, Y. Gao and G. N. DeSouza, "A study on solving the inverse
kinematics of serial robots using artificial neural network and fuzzy neural
network," IEEE International Conference on Fuzzy Systems (FUZZ-IEEE),
New Orleans, 2019, pp. 1-6.
[20]J. Xiao, W. Han and A. Wang, "Simulation research of a six degrees of
freedom manipulator kinematics based on Matlab toolbox," in Proc.
International Conference on Advanced Mechatronic Systems, Xiamen, Mar.
2017, pp. 376-380.
[21]Deep Neural Networks
62
https://docs.opencv.org/master/d2/d58/tutorial_table_of_content_dnn.html
，2020 年 5 月。
[22]F. Haoqiang and E. Zhou. "Approaching human level facial landmark
localization by deep learning," Image Vis. Comput., Mar. 2016, vol. 47, pp.
27-35.
[23]L. Xiao, "A review of solutions for perspective-n-point problem in camera
pose estimation," Journal of Physics: Conference Series., Sep. 2018, vol.
1087.
[24]J. J. Kuffner Jr and S. M. LaValle, "RRT-Connect: An efficient approach to
single-query path planning," Proc. IEEE International Conference on
Robotics and Automation, San Francisco, Aug. 2000, pp. 995-1001.
[25]ASUS – GU501
https://www.asus.com/tw/ROG-Republic-Of-Gamers/ROGGU501/?SearchKey=gu501/，2020 年 5 月。
[26]達明機器人
https://www.tm-robot.com/zh-hant/regular-payload/，2020 年 5 月。
[27]Intel® RealSense™ Depth Camera D435i
https://www.intelrealsense.com/zh-hans/depth-camera-d435i/ ，2020 年 5
月。
[28]ROS - Wiki
http://wiki.ros.org/ROS/Tutorials，2020 年 5 月。
[29]J. Redmon, A. Farhadi, "YOLOv3: An incremental improvement," arXiv
preprint, 2018, arXiv:1804.02767.
[30]K. Simonyan and A. Zisserman, "Very deep convolutional networks for
large-scale image recognition," CoRR, Apr. 2015.
[31]K. He, X. Zhang, S. Ren and J. Sun, "Deep residual learning for image
recognition," IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), Las Vegas, Nov. 2016, pp. 770-778.
[32]I. H. Choi and Y. G. Kim, "Head pose and gaze direction tracking for
detecting a drowsy driver," International Conference on Big Data and
Smart Computing, Bangkok, 2014, pp. 241-244.
[33]Camera Calibration and 3-D Vision - MATLAB & Simulink
https://www.mathworks.com/help/vision/ref/cameracalibrator-app.html ，
2020 年 5 月。
63
[34]Z. Zhang, "A flexible new technique for camera calibration," IEEE
Transactions on Pattern Analysis and Machine Intelligence, Nov. 2000, vol.
22, no. 11, pp. 1330-1334.
[35]B. Amos, B. Ludwiczuk, M. Satyanarayanan, "Openface: a general-purpose
face recognition library with mobile applications," CMU-CS-16-118, CMU
School of Computer Science, June 2016.
[36]D. Jacques and H. R. Scheunemann, "A kinematic notation for lower-pair
mechanisms based on matrices," Trans ASME J. Appl. Mech, 23, 1955, pp.
215-221.
[37]J. Peng, W. Xu, Z. Wang and D. Meng, "Analytical inverse kinematics and
trajectory planning for a 6DOF grinding robot," IEEE International
Conference on Information and Automation (ICIA), Yinchuan, 2013, pp.
834-839.
[38]std_msgs - ROS Wiki
http://wiki.ros.org/std_msgs，2020 年 5 月。
[39]Melexis-MLX90621
https://www.melexis.com/en/product/mlx90621/far-infrared-sensor-arrayhigh-speed-low-noise，2020 年 5 月。
[40]Intel RealSense Help Center D400 Series
https://support.intelrealsense.com/hc/enus/community/posts/360037076293-Align-color-and-depth-images，2020
年 5 月。
[41]Datasheet – Intel
https://www.intel.com/content/dam/support/us/en/documents/emergingtechnologies/intel-realsense-technology/Intel-RealSense-D400-SeriesDatasheet.pdf，2020 年 6 月。
[42]Depth Post-Processing for Intel® RealSense™ Depth Camera D400 Series
https://dev.intelrealsense.com/docs/depth-post-processing，2020 年 6 月。

指導教授

王文俊(Wen-June Wang)

審核日期

2020-7-21

推文