以點對點技術為基礎之整合性資訊管理 及分析系統;Peer-to-Peer-Based Big Data Management and Analysis in Astronomical Applications

NCUIR > College of Electrical Engineering & Computer Science > Graduate Institute of Computer Science and Information Engineering > Electronic Thesis & Dissertation > Item 987654321/65634

Please use this identifier to cite or link to this item: http://ir.lib.ncu.edu.tw/handle/987654321/65634

Title:	以點對點技術為基礎之整合性資訊管理及分析系統;Peer-to-Peer-Based Big Data Management and Analysis in Astronomical Applications
Authors:	唐正憲;Tang,Cheng-Hsien
Contributors:	資訊工程學系
Keywords:	點對點傳輸;資料分群;資料分類;分散式系統;Peer-to-Peer;Hierarchical Clustering;Data classification;Distributed Computing
Date:	2014-07-22
Issue Date:	2014-10-15 17:06:54 (UTC+8)
Publisher:	國立中央大學
Abstract:	由於各種資訊觀測技術的進步以及硬體價格下降的關係，使得天文觀測的精確度及資料量產生爆炸性的成長，同時也對於資料管理及分析產生極大負擔，以往經由手動處理及人工分析的研究方式也不再適用於如今的超巨量資料集。目前對於天文研究最大的挑戰是：一.降低資料維護及處理所產生的各種成本。二. 在合理時間內，從各種不同的巨量觀察紀錄中搜索並擷取使用者需要的資料。三. 改進已知傳統分析方法或設計新型演算法，以對應各種大型資料及分散式儲存環境。四. 能夠快速適應不同狀況及降低開發難度的彈性化架構。雖然目前已經有多種不同的分散式架構能夠提供科學家當作使用的工具進行開發，但是系統設計仍然不是科學家的長向，結果便是到目前為止仍然缺少一個真對天文領域，全方位的管理及分析方案。本研究將因應天文資料的特性，設計出具有以下功能的整合性系統：一. 可對應不同資料量、不同資源的系統架構，並加入自動化軟、硬體管理及高容錯力的點對點資訊管理系統。二. 針對天文資料的格式，建立快速索引。三. 將各種功能切割為多種獨立單位，讓系統能夠快速對應各種不同的需求。四. 以此系統架構為基礎，設計出針對天文資料的分類、分群範例演算法。本系統是由三個子系統所組成，其中包含一個以點對點傳輸為基礎之大量資料管理系統，此系統提供快速的資料搜尋以及自動化管理技術。第二個系統是大型資料分類系統，提供類似機器學習的大量資料分析技術，第個系統是大型資料分群系統，利用訊息傳撥演算法，讓系統能夠在分散式環境下針對大量資料建立階層式群集。本研究所提供的系統可以提供使用者針對大型資料管理以及分析所需的各種工具，同時讓使用者能夠針對不同的需求進行各式客製化動作，將系統快速的建立起來。;The improvement of information technology provides scientific observations of high quality that demand larger storage space and faster data processing power than ever before. However, it also massively increases the cost of the corresponding management and analytical processes. Thus, it becomes impractical to process tera-bytes of data using traditional approaches. From the perspective of astronomical data processing, the most important challenges are: 1. To maintain large amount of data with lower cost and overhead, 2. To locate and to extract desired data from a huge collection of data pool in a reasonable time, 3. To develop new analysis methods for large-scale of data based on distributed environment, and 4. To use a flexible architecture that can adapt into different situation quickly and decrease the overhead of development. Even though the existing distributed computing techniques, such as grid and cloud technologies, have provided the scientists a better way to access powerful computing resources, the development of big-data management and analysis software is still lagging far behind. The awkward predicament obstructs the connected computing resources from being utilized efficiently. To deal with the problem, we used integrated, efficient information management and analysis system for astronomical data processing. Therefore, this study focuses on the development of a management system design as well as the distributed classification and clustering methods for efficient data analysis in various astronomical application. The proposed system can be viewed as a integrated system that supports management and analysis of large data collections. It consists of one data management sub-system and two analytical sub-systems. The first sub-system is called the Peer-to-Peer-Based Management System (P2PBMS), which adapt the Chord system design to construct a scalable platform for fast data retrieval and management. The second sub-system is called the Similarity Classification System (SCS), which uses a decentralized Multiple Classifier System (MCS) framework to provide fast and stable classification in a distributed environment using multiple classifiers. The last one is called the Distributed Hierarchical Clustering System (DHCS), which uses a distributed message-passing algorithm to efficiently calculate a hierarchical cluster, given a set of astronomical data. The proposed integrated system can support large-scale data management and analysis for astronomical data processing. With the three sub-systems, we can provide necessary analytical tools and combination frameworks to deal with different kinds of complex analysis tasks. The Unit-Based structure can decrease the overhead of system customization for different purposes.
Appears in Collections:	[Graduate Institute of Computer Science and Information Engineering] Electronic Thesis & Dissertation

Files in This Item:

File	Description	Size	Format
index.html		0Kb	HTML	522	View/Open

社群 sharing

Loading...