基於頁面層級之快速網頁資料擷取與綱要驗證

以作者查詢圖書館館藏

、以作者查詢臺灣博碩士

、以作者查詢全國書目

、勘誤回報

、線上人數：11

、訪客IP：13.59.36.203

姓名

陳天盛(Tian-Sheng, Chen) 查詢紙本館藏

畢業系所

資訊工程學系

論文名稱

基於頁面層級之快速網頁資料擷取與綱要驗證
(Efficient Web Data Extraction Via Page-Level Schema Induction & Verification)

相關論文

★ 行程邀約郵件的辨識與不規則時間擷取之研究	★ NCUFree校園無線網路平台設計及應用服務開發
★ 網際網路半結構性資料擷取系統之設計與實作	★ 非簡單瀏覽路徑之探勘與應用
★ 遞增資料關聯式規則探勘之改進	★ 應用卡方獨立性檢定於關連式分類問題
★ 中文資料擷取系統之設計與研究	★ 非數值型資料視覺化與兼具主客觀的分群
★ 關聯性字組在文件摘要上的探討	★ 淨化網頁：網頁區塊化以及資料區域擷取
★ 問題答覆系統使用語句分類排序方式之設計與研究	★ 時序資料庫中緊密頻繁連續事件型樣之有效探勘
★ 星狀座標之軸排列於群聚視覺化之應用	★ 由瀏覽歷程自動產生網頁抓取程式之研究
★ 動態網頁之樣版與資料分析研究	★ 同性質網頁資料整合之自動化研究

檔案

[Endnote RIS 格式]

[Bibtex 格式]

[相關文章]

[文章引用]

[完整記錄]

[館藏目錄]

[檢視]

[下載]

本電子論文使用權限為同意立即開放。
已達開放權限電子全文僅授權使用者為學術研究之目的，進行個人非營利性質之檢索、閱讀、列印。
請遵守中華民國著作權法之相關規定，切勿任意重製、散佈、改作、轉貼、播送，以免觸法。

摘要(中)

在網頁資訊擷取(Web Data Extraction)的領域之中，如何自動從各式各樣不同的網頁中擷取出資料的相關議題至今已被探討了十多年，然而由於網頁的內容與架構的複雜，現有的方法均有其限制之處，再加上大量網頁擷取的需求，使得網頁資訊擷取的研究仍有相當大的挑戰。
網頁資料擷取系統主要分成記錄層級(Record Level)和頁面層級(Page Level)兩大類別，雖然頁面層級相較於記錄層級能夠得到更完整的網頁資訊，但由於問題的複雜及實作的困難兩大瓶頸，使得頁面層級的議題鮮少被關注，其擷取的效能與效率都有改進的空間。另一方面，雖然許多頁面層級的擷取系統標榜免標記的訓練，但是對於測試網頁的運作並無太多著墨。
有鑑於此，在本篇論文當中，我們提出了一套學習概念的頁面層級擷取系統，針對大量網頁擷取的情況，我們只需對一部分的網頁進行非監督式的訓練，並且利用訓練出的綱要(Schema)，透過擷取程式驗證(Wrapper Verification)的機制來測試其他剩餘的網頁，並同時擷取出網頁資料。本論文的實驗顯示，在表列網頁(List page)的處理上，本系統產生的綱要都比過去的頁面層級系統要來得準確且資料擷取的效率也較快，擷取效率相較於過去針對每一個網頁都進行分析的系統架構要快上數十倍。

摘要(英)

The problem of automatically extracting data from web pages has been studied more than ten years. However, existing researches have limitations due to high structural complexity in web pages. On the other hand, the necessity of extracting data from large amount of web pages make it a challenging task for researchers.
Web data extraction can be classified into two categories based on the extraction targets, record-level task and page-level task. Although the web data extracted by page-level approach is more complete than record-level approach, very few researches focus on this task because of the difficulties and complexities in the problem, and there are still much to be desired on effectiveness and efficiency. On the other hands, previous page-level systems focus on how to achieve unsupervised training and pay less concern about how to extract data from testing pages by matching with a wrapper.
In this paper, we propose a learning based architecture for page-level extraction systems. Given a large amount of web pages for data extraction, the system use part of the input pages for training the schema, and then extract data from the rest of the input pages through wrapper verification. In our experiments, our system works better than other page-level extraction systems in terms of schema accuracy and extraction efficiency for multi-record pages. Overall, the extraction efficiency is dozens of times higher than state-of-the-art unsupervised approaches that extract data page by page without learning scheme (wrapper verification).

關鍵字(中)

★ 擷取規則推導
★ 擷取規則驗證
★ 網頁資料擷取

關鍵字(英)

★ Wrapper Induction
★ Wrapper Verification
★ Web Data Extraction

論文目次

中文摘要 i
Abstract ii
誌謝 iii
目錄 iv
圖目錄 v
表目錄 vi
一、緒論 1
二、相關研究 5
2.1 頁面層級擷取系統 5
2.2 擷取規則驗證系統 6
三、擷取規則推導 (Wrapper Induction) 9
3.1 重複樣式探勘 (Repeat Pattern Mining) 13
3.1.1 列舉樣式 (Enumerate Patterns) 13
3.1.2 擷取有序樣式的天際線 (Get Sequential Patterns of Skyline) 14
3.1.3 輸出樣式紀錄 (Output Pattern Records) 16
3.2 排比程序 (Alignment Procedure) 17
3.2.1 樣板過濾 (Template Filtering) 17
3.2.2 排比 (Alignment) 18
3.3 綱要生成 (Schema Generation) 21
四、擷取規則驗證 (Wrapper Verification) 25
4.1 建立有限狀態機 (Build Finite State Machine) 25
4.1.1 建立狀態與基本轉移 (Build States and Basic Transitions) 27
4.1.2 建立出入口資訊表 (Build Entrance and Exit Information) 28
4.1.3 加入跳過轉移 (Add Jump Transitions) 31
4.2 候選狀態建立 (Build Candidate States) 33
4.3 最佳候選狀態提取 (Select the Best Candidate) 37
4.4 網頁資料輸出 (Output Data) 39
五、實驗 40
5.1 網頁綱要效能評估 41
5.2 資料擷取效率評估 42
5.2.1 驗證機制擷取效率成長評估 42
5.2.2 擷取效率比較 46
六、結論與未來工作 47

參考文獻

[1] R. Agrawal and R. Srikant, "Mining Sequential Patterns," presented at the Proceedings of the Eleventh International Conference on Data Engineering, 1995.
[2] A. Arasu and H. Garcia-Molina, "Extracting structured data from Web pages," presented at the Proceedings of the 2003 ACM SIGMOD international conference on Management of data, San Diego, California, 2003.
[3] G. O. Arocena and A. O. Mendelzon, "WebOQL: Restructuring documents, databases, and Webs," Theory and Practice of Object Systems, vol. 5, pp. 127-141, 1999.
[4] C.-H. Chang and S.-C. Lui, "IEPAD: information extraction based on pattern discovery," presented at the Proceedings of the 10th international conference on World Wide Web, Hong Kong, Hong Kong, 2001.
[5] C.-H. Chang, Y.-L. Lin, K.-C. Lin, and M. Kayed, "Page-Level Wrapper Verification for Unsupervised Web Data Extraction," in Web Information Systems Engineering – WISE 2013. vol. 8180, X. Lin, Y. Manolopoulos, D. Srivastava, and G. Huang, Eds., ed: Springer Berlin Heidelberg, 2013, pp. 454-467.
[6] M.-C. Chen, "Exploiting Dynamic Encoding and Multiple Pages for Record Boundary Detection & Data Extraction," Master′s Computer Science and Information Engineering at National Central University, 2014.
[7] V. Crescenzi, G. Mecca, and P. Merialdo, "RoadRunner: Towards Automatic Data Extraction from Large Web Sites," presented at the Proceedings of the 27th International Conference on Very Large Data Bases, 2001.
[8] C. N. Hsu and M. T. Dung, "Generating finite-state transducers for semi-structured data extraction from the Web," Information Systems, vol. 23, pp. 521-538, Dec 1998.
[9] M. Kayed and C. H. Chang, "FiVaTech: Page-Level Web Data Extraction from Template Pages," Ieee Transactions on Knowledge and Data Engineering, vol. 22, pp. 249-263, Feb 2010.
[10] N. Kushmerick, "Wrapper induction: Efficiency and expressiveness," Artificial Intelligence, vol. 118, pp. 15-68, Apr 2000.
[11] N. Kushmerick, "Wrapper verification," World Wide Web, vol. 3, pp. 79-94, 2000/10/01 2000.
[12] K. Lerman, S. N. Minton, and C. A. Knoblock, "Wrapper maintenance: a machine learning approach," J. Artif. Int. Res., vol. 18, pp. 149-181, 2003.
[13] L. Ling, C. Pu, and W. Han, "XWRAP: an XML-enabled wrapper construction system for Web information sources," in Data Engineering, 2000. Proceedings. 16th International Conference on, 2000, pp. 611-621.
[14] W. Liu, X. F. Meng, and W. Y. Meng, "ViDE: A Vision-Based Approach for Deep Web Data Extraction," Ieee Transactions on Knowledge and Data Engineering, vol. 22, pp. 447-460, Mar 2010.
[15] I. Muslea, S. Minton, and C. A. Knoblock, "Hierarchical wrapper induction for semistructured information sources," Autonomous Agents and Multi-Agent Systems, vol. 4, pp. 93-114, Mar-Jun 2001.
[16] N. Pasquier, Y. Bastide, R. Taouil, and L. Lakhal, "Discovering frequent closed itemsets for association rules," Database Theory - Icdt′99, vol. 1540, pp. 398-416, 1999.
[17] E. H. Pek, X. Li, and Y. Z. Liu, "Web wrapper validation," Web Technologies and Applications, vol. 2642, pp. 388-393, 2003.
[18] J. Roberto J. Bayardo, "Efficiently mining long patterns from databases," SIGMOD Rec., vol. 27, pp. 85-93, 1998.
[19] H. A. Sleiman and R. Corchuelo, "TEX: An efficient and effective unsupervised Web information extractor," Knowledge-Based Systems, vol. 39, pp. 109-123, Feb 2013.
[20] L. Wang and T. Jiang, "On the complexity of multiple sequence alignment," J Comput Biol, vol. 1, pp. 337-48, Winter 1994.
[21] Y. Zhai and B. Liu, "Web data extraction based on partial tree alignment," presented at the Proceedings of the 14th international conference on World Wide Web, Chiba, Japan, 2005.

指導教授

張嘉惠(Chia-Hui Chang)

審核日期

2014-8-20

推文