博碩士論文 107525005 詳細資訊




以作者查詢圖書館館藏 以作者查詢臺灣博碩士 以作者查詢全國書目 勘誤回報 、線上人數:35 、訪客IP:3.149.26.246
姓名 許家繻(Chia-Ju Hsu)  查詢紙本館藏   畢業系所 軟體工程研究所
論文名稱 在Kubernetes與Kafka上實作基於Consumer lag之微服務動態轉移機制
(Consumer lag aware microservice relocation mechanism over Kubernetes and Kafka)
相關論文
★ 以伸展樹為基礎的Android Binder Driver★ 應用增量式學習於多種農作物判釋之研究
★ 應用分類重建學習偵測航照圖幅中的新穎坵塊★ 一個建立在平行工作系統上的動態全球計算平台
★ 用權重參照計數演算法執行主動物件垃圾收集★ 一個動態負載平衡之最大可能性估算計算架構
★ 利用多項系統負載資訊進行動態P2P系統重組的策略研究★ 基於Hadoop系統的雲端應用程式特徵擷取與計算監測架構
★ 適用於大型動態分散式系統的調適性計算模型★ 一個提供彈性虛擬資料中心的雲端服務平台
★ 雲端彈性虛擬機房服務平台之資源控管中心★ 一個適用於自動供應雲端系統的動態調適計算架構
★ 線性相關工作與非相關工作的探索式排程策略★ 適用於大資料集高效率的分散式階層分群演算法
★ 混合雲端環境上的多重代理人動態調適計算管理架構★ 基於圖形的平行化最小生成樹分群演算法
檔案 [Endnote RIS 格式]    [Bibtex 格式]    [相關文章]   [文章引用]   [完整記錄]   [館藏目錄]   [檢視]  [下載]
  1. 本電子論文使用權限為同意立即開放。
  2. 已達開放權限電子全文僅授權使用者為學術研究之目的,進行個人非營利性質之檢索、閱讀、列印。
  3. 請遵守中華民國著作權法之相關規定,切勿任意重製、散佈、改作、轉貼、播送,以免觸法。

摘要(中) 近年來容器化技術發展迅速,許多軟體服務開始以容器和微服務的形式部署運行,而開源容器管理系統Kubernetes為容器自動部署和管理提供強大的功能,使容器部署變得更加彈性且可靠,許多雲端供應商也提供Kubernetes代管服務,使在雲端叢集中部署容器變得更加普遍。同時,一些物聯網和邊緣運算的應用情境也開始嘗試使用Kubernetes部署工作,這些工作可能會面臨邊緣端環境變化導致的性能下降、節點計算能力不均或工作量動態變化等問題導致工作量堆積。然而,Kubernetes的計算資源管理依賴管理者的估算,難以適應動態的變化。因此,本研究設計出一套Kubernetes擴充功能,以Kafka訊息串流平台作為容器之間的資料傳遞管道,透過監控其Consumer Lag以支援在Kubernetes上實現根據容器的工作進度對容器進行轉移,並設計轉移流程以保證轉移後不會有資料遺失或重複處理的狀況。
摘要(英) Container technology has developed rapidly in recent years, and many software services are deployed as containers and microservices. Kubernetes, an open-source container orchestration system that provides powerful features to automatically deploy and manage containerized applications, makes container deployment more flexible and reliable. So many cloud vendors provide managed Kubernetes service make it more popular to deploy containers on a cloud cluster. Some scenarios of the Internet of Things or edge computing also use Kubernetes to deploy their jobs. These jobs may face some problems, such as degradation of work node performance caused by edge environment variety, uneven computing power between work nodes, or dynamic workload. However, managing resources for containers in Kubernetes is rely on the manager’s estimation, and it is difficult to adjust to dynamic workload changes. Therefore, this thesis proposes a set of extension functions for Kubernetes, make it possible to relocate containers according to the work progress, which is achieved by using Kafka as message passing channel and monitoring Consumer Lag of each job. This thesis also proposes a relocation control flow to avoid data loss or data re-process after container relocation.
關鍵字(中) ★ Kubernetes
★ Kafka
★ 微服務
★ 動態轉移
關鍵字(英) ★ Kubernetes
★ Kafka
★ microservice
★ dynamic relocation
論文目次 摘要 i
Abstract ii
目錄 iii
表目錄 v
圖目錄 v
一、 緒論 1
1-1 研究背景 1
1-2 研究動機與目的 2
1-3 論文架構 2
二、 背景知識 3
2-1 Kubernetes 3
2-2 Kubernetes Pod轉移機制 5
2-2-1 基於資源使用量 5
2-2-2 節點不可用 6
2-3 Prometheus 7
2-3-1 kube-prometheus 8
2-4 Apache Kafka 9
2-4-1 Kafka Offset Commit 機制 10
2-4-2 Kafka Consumer Group Rebalance機制 11
2-5 服務發現 12
2-5-1 Client-Side模式 12
2-5-2 Server-Side模式 12
2-5-3 Kubernetes中的服務發現 12
三、 相關研究與討論 15
3-1 Circuit Breaker 15
3-2 邊緣運算工作轉移相關研究 15
四、 系統設計 17
4-1 系統架構 17
4-2 Job Operator 22
4-2-1 ManagedJob 22
4-2-2 Region 24
4-2-3 ConsumerLagMonitor 26
4-3 自訂kube-prometheus 28
4-4 ConsumerLag Monitor 29
4-5 Consumer Helper 32
4-5-1 轉移情況討論 32
4-5-2 流程設計 33
五、 使用範例及實驗 35
5-1 實驗環境 35
5-2 模擬計算量之overhead量測 36
5-2-1 模擬中斷後重新啟動狀況 38
5-2-2 正常轉移狀況 40
5-3 Consumer數量變化 42
5-4 模擬車流量偵測應用情境之overhead量測 43
六、 結論與未來研究方向 48
參考文獻 49
參考文獻 [1] D. Bernstein, “Containers and Cloud: From LXC to Docker to Kubernetes,” IEEE Cloud Computing, vol. 1, no. 3, pp. 81–84, Sep. 2014, doi: 10.1109/MCC.2014.51.
[2] D. Jaramillo, D. V. Nguyen, and R. Smart, “Leveraging microservices architecture by using Docker technology,” in SoutheastCon 2016, Mar. 2016, pp. 1–5, doi: 10.1109/SECON.2016.7506647.
[3] F. Montesi and J. Weber, “Circuit Breakers, Discovery, and API Gateways in Microservices,” arXiv:1609.05830 [cs], Sep. 2016, Accessed: Apr. 26, 2020. [Online]. Available: http://arxiv.org/abs/1609.05830.
[4] “What is Kubernetes.” https://kubernetes.io/docs/concepts/overview/what-is-kubernetes/ (accessed Mar. 26, 2020).
[5] B. Burns, J. Beda, and K. Hightower, “Kubernetes: Up and Running,” p. 1.
[6] “Managing Compute Resources for Containers.” https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/ (accessed Apr. 11, 2020).
[7] “Configure Out of Resource Handling.” https://kubernetes.io/docs/tasks/administer-cluster/out-of-resource/ (accessed Apr. 11, 2020).
[8] “Overview | Prometheus.” https://prometheus.io/docs/introduction/overview/ (accessed Mar. 26, 2020).
[9] “prometheus/node_exporter.” https://github.com/prometheus/node_exporter (accessed Mar. 26, 2020).
[10] “coreos/kube-prometheus,” May 29, 2020. https://github.com/coreos/kube-prometheus (accessed May 29, 2020).
[11] “Jsonnet - The Data Templating Language.” https://jsonnet.org/ (accessed May 29, 2020).
[12] “Apache Kafka,” Apache Kafka. https://kafka.apache.org/intro (accessed Mar. 28, 2020).
[13] M. Kleppmann and J. Kreps, “Kafka, Samza and the Unix Philosophy of Distributed Data,” p. 7.
[14] “Kafka 0.9 Consumer Rewrite Design - Apache Kafka - Apache Software Foundation.” https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design#Kafka0.9ConsumerRewriteDesign-HeartbeatResponse (accessed Jun. 13, 2020).
[15] “Kafka Client-side Assignment Proposal - Apache Kafka - Apache Software Foundation.” https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Client-side+Assignment+Proposal#KafkaClientsideAssignmentProposal-Phase1:JoiningtheGroup (accessed Jun. 13, 2020).
[16] C.-Y. Fan and S.-P. Ma, “Migrating Monolithic Mobile Application to Microservice Architecture: An Experiment Report,” in 2017 IEEE International Conference on AI Mobile Services (AIMS), Jun. 2017, pp. 109–112, doi: 10.1109/AIMS.2017.23.
[17] “Service.” https://kubernetes.io/docs/concepts/services-networking/service/ (accessed Apr. 30, 2020).
[18] Xuan-Qui Pham and Eui-Nam Huh, “Towards task scheduling in a cloud-fog computing system,” in 2016 18th Asia-Pacific Network Operations and Management Symposium (APNOMS), Oct. 2016, pp. 1–4, doi: 10.1109/APNOMS.2016.7737240.
[19] D. Haja, M. Szalay, B. Sonkoly, G. Pongracz, and L. Toka, “Sharpening Kubernetes for the Edge,” in Proceedings of the ACM SIGCOMM 2019 Conference Posters and Demos, Beijing, China, Aug. 2019, pp. 136–137, doi: 10.1145/3342280.3342335.
[20] “The confluent_kafka API — confluent-kafka 1.4.0 documentation.” https://docs.confluent.io/current/clients/confluent-kafka-python/ (accessed Jun. 02, 2020).
指導教授 王尉任 審核日期 2020-8-17
推文 facebook   plurk   twitter   funp   google   live   udn   HD   myshare   reddit   netvibes   friend   youpush   delicious   baidu   
網路書籤 Google bookmarks   del.icio.us   hemidemi   myshare   

若有論文相關問題,請聯絡國立中央大學圖書館推廣服務組 TEL:(03)422-7151轉57407,或E-mail聯絡  - 隱私權政策聲明