利用多影像平均值以及高斯濾波方法提升RPA軟體的影像辨識準確度

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：51

、訪客IP：3.14.141.177

姓名

李資瀚(Zi-Han Li) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

利用多影像平均值以及高斯濾波方法提升RPA軟體的影像辨識準確度

相關論文

★ 使用PolyTraceAid進行程式文件覆蓋率計算與分群	★ Support Visual Debugging in Electronic Design Automation Software by xDIVA
★ 設計與實作視覺化追蹤點以支援xDIVA進行程式動畫	★ Combine Internal Test Oracles and Capture/Replay GUI Testing for Precise Replay and Higher Validation Power
★ Dissimilarity of Polymorphic Execution Sequences	★ The Analysis of Capturing System Test Cases into Unit Test Cases
★ 動態延遲載入的測試覆蓋率	★ 建構於JMeter之自動化分散式壓力測試架構
★ 模組化因修改具耦合的程式碼所伴隨的成本漣漪	★ Design a Pluggable Architecture for Layout Algorithms in xDIVA
★ 重複性程式碼檢測之外掛模組設計	★ Visual Perception of Realistic Rendering in xDiva 3D Environment
★ Why and When GUI Test Automation Fails in Practice and Our Solutions to The Problem	★ Why and When GUI Test Automation Fails in Practice and Our Solutions to The Problem
★ Very High Precision Optical Character Recognition For Clean-Fixed-Sized True Type Characters	★ Enhance Korat by Branch Capability in Capture/Replay User Scenario to Industrial Test Case Automation

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

RPA (Robotic Process Automation) 智慧軟體流程機器人為工業 4.0 中不可或缺的成員之一。RPA 可以應用於高重複性的人工作業流程，透過 RPA 的幫助可以減少人力的耗損以及出錯的可能性。RPA 軟體的應用除了在辦公室自動化之外，還可以應用在工業製造的環境。
本實驗室所開發的 Robotiive 即是適用於工業製造的 RPA 軟體。Robotiive 會擷取當下電腦螢幕畫面，並透過 OpenCV 中的模板匹配 (Template Matching) 的方式，找到使用者所框選的 GUI 元件，並進行操作。
在實際應用的案例中，使用者為了維持機台端電腦的穩定性，在大多數的情況下不會希望將軟體直接安裝在機台端電腦，而是會透過 KVM (Keyboard Video Mouse Switch)、遠端軟體或是網頁來讓 RPA 軟體來對機台電腦進行操作。在這些案例中，我們發現不同的遠端軟體會因為網路因素對回傳的影像進行壓縮，或是網頁文字中產生的反鋸齒 (Anti-aliasing)，造成 Robotiive 的影像辨識失敗。
以上問題雖然透過降低 OpenCV 中模板匹配的閥值或是重新截圖可以改善，不過當環境因素改變後，又會使得新截取的影像辨識失敗。因此，本論文將會以多影像平均值以及高斯濾波基於 OpenCV 的模板匹配，開發出一套不降低閥值且不需要重新截圖就可以正確辨識出目標影像的方法。

摘要(英)

RPA (Robotic Process Automation) is an indispensable member of Industry 4.0.RPA can be applied to a highly repetitive manual process. With the help of RPA, attrition and the possibility of human errors can be reduced and automation is enabled. Commercial RPA tools are mainly applied in office automation domain and they are not generally applicable in manufacturing environment.
Robotiive originated from our lab is the RPA software suitable for industrial manufacturing. Robotiive takes a screenshot from current computer and use template matching method in OpenCV to find the GUI component selected by the user.
In practical manufacturing applications, in order to maintain stability of the machine-side computer, users do not want to install the software directly on the machine-side computer in most cases, but will use KVM (Keyboard Video Mouse Switch), remote software or webpage to allow RPA software to operate the machine and computer. In these cases, we found that different remote software
may compress the returned screen image due to network factors, or activate antialiasing effect in the text of the webpage, causing image recognition failed in Robotiive.
Although the above problems can be improved by lowering the threshold of template matching in OpenCV or re-taking screenshot, environmental change will cause the threshold to unstable and sensitive again. Therefore, this thesis use averaging multiple images and Gaussian filter based on OpenCV template
matching to develop a method that can identify the target image correctly without reducing the threshold or re-taking screenshot.

關鍵字(中)

★ 多影像平均值
★ 高斯濾波
★ 影像辨識

關鍵字(英)

★ Average of multiple image
★ Gaussian Blur
★ Image recognition

論文目次

摘要 i
Abstract ii
目錄 iii
圖目錄 v
表目錄 vii
一、緒論 1
1-1 RPA 軟體 1
1-2 Robotiive 核心原理及實際應用的問題 2
1-3 解決雜訊及反鋸齒問題的方法 3
二、研究背景 4
2-1 腳本產生方式 4
2-2 和 GUI 元件互動的方式 5
2-2-1 直接指定螢幕座標 5
2-2-2 透過系統指定目標 GUI 之 ID 6
2-2-3 影像辨識 8
2-3 UiPath 8
2-4 Robotiive 10
三、問題描述與分析 13
3-1 遠端軟體 13
3-1-1 Windows Remote Desktop 13
3-1-2 TeamViewer 14
3-1-3 VNC 14
3-2 反鋸齒 15
3-3 分析 15
3-3-1 計算 RGB 變化的模式 15
3-3-2 分析結果 18
四、解決方案 19
4-1 利用多影像平均值消除雜訊 20
4-1-1 對擷取目標影像的流程進行修正 20
4-1-2 對擷取螢幕影像的流程進行修正 22
4-2 利用高斯濾波去除反鋸齒 25
4-2-1 高斯濾波 25
4-2-2 對模板匹配的流程進行修正 26
4-3 對整體流程改變的影響 29
五、評估 30
5-1 不同圖片大小在不同強度雜訊下的情境 30
5-2 不同圖片大小的情境套用解決方案後的評估結果 32
5-2 中英文在不同反鋸齒效果的情境 34
5-4 對於相似文字的辨識率評估 37
六、結論與未來展望 39
七、參照 40

參考文獻

[1] Chen, Xin-Chih. "Korat: An O.S.-independent Capture/Replay Test Automation System." Institute of Computer Science & Information Engineering, National Central University, 2014.
[2] Cheng, Yung-Pin, Deron Liang, and Wei-Jen Wang. "KORAT—A platform independent test automation tool by emulating keyboard/mouse hardware signals." 2016 IEEE AUTOTESTCON. IEEE, 2016.
[3] Chen, Li-Hsuan. "Enhance Korat by Branch Capability in Capture/Replay User Scenario to Industrial Test Case Automation." Institute of Computer Science & Information Engineering, National Central University, 2018.
[4] OpenCV. "Template Matching ". 1 June 2021 < https://docs.opencv.org/3.4/de/da9/tutorial_template_matching.html >.
[5] Cheng, Yung-Pin, Ching-Wei Li, and Yi-Cheng Chen. "Apply computer vision in GUI automation for industrial applications." Mathematical biosciences and engineering: MBE 16.6 (2019): 7526-7545.
[6] OpenCV. "Smoothing Images”. 1 June 2021 < https://docs.opencv.org/master/d4/d13/tutorial_py_filtering.html >
[7] Automation Anywhere. June 2021 < https://www.automationanywhere.com/>
[8] UiPath. June 2021 < https://www.uipath.com/ >
[9] Windows Remote Desktop. June 2021 < https://docs.microsoft.com/windows-server/remote/remote-desktopservices/clients/remote-desktop-clients >
[10] TeamViewer. June 2021 < https://www.teamviewer.com/ >
[11] VNC. June 2021 < https://www.realvnc.com/ >
[12] ORL/AT&T Labs Cambridge. "The RFB Protocol". June 2021 < https://camorl.co.uk/rfb.html >
[13] OpenCV. "Image Filtering" June 2021 < https://docs.opencv.org/master/d4/d86/group__imgproc__filter.html >

指導教授

鄭永斌(Yung-Pin Cheng)

審核日期

2021-7-19

推文