虛擬實境(Virtual reality)應用以頭戴式裝置(head-mounted displays, HMD)顯示360度視訊(360 degree video)，提供使用者身臨其境的體驗(immersive experiences)，但360度視訊衍生以下挑戰: (1) 高空間解析度(spatial resolution)與高畫面率(frame rate) 360度視訊，引致4K或更大的資料量。(2) 經球體對平面投影(sphere-to-plane projections)(例如: 等距長方(equirectangular)或多面體(polyhedron))的2D平面影像，具有無可避免的幾何形變(geometric deformation)。(3) 因應使用者頭部移動(head movement)，使用者觀賞方向的viewport須即時串流(streaming)。目前最新的(state-of-the-art) 2D視訊壓縮標準如HEVC，仍難以使360度視訊位元串流(bitstream)符合現今通訊的網路延遲(latency)上限，而現有的2D視訊的物件追蹤相關技術，亦多未考量目前360度視訊編碼標準考量的投影影像。因此，本一年期計畫之研究目標，將針對360度視訊處理與分析的兩個主題，進行研究探討並提出設計: (1)視訊編碼(Video coding)的位元率控制(rate control)。(2)視訊分析(Video analysis)的物件追蹤(object tracking)。本計畫研究成果預期將有效增加360度視訊壓縮的位元率控制準確率，平滑影像品質與緩衝區滿度(buffer fullness)，提升360度視訊的物件追蹤準確率，進而提升虛擬實境應用之使用者觀賞品質與360度視訊分析的準確率。 ;For virtual reality applications, head-mounted displays (HMDs) with 360 degree videos provide users immersive experiences. However, several new challenges also arise: (1) 360 degree videos with high spatial resolution and high frame rate lead to a large amount of data (4K and beyond). (2) Sphere-to-plane projections such as equirectangular projection and polyhedron projection may have inevitable geometric deformation. (3) Viewport streaming should promptly respond to head movement of viewers. State-of-the-art methods of 2D video coding (e.g., HEVC) still cannot make the compressed bitstream of 360 degree videos meet the constraint of network latency. Moreover, most existing object tracking techniques cannot be applied to current 360 degree videos with variety of projection formats. Thus, the major goals of this project consist of two parts of 360 degree video processing and analysis, and they are stated as follows: (1) Rate control of 360 degree video coding. (2) Object tracking on 360 degree videos. This project expects to increase accuracy of rate control of 360 degree video coding, smooth visual quality and buffer fullness of 360 degree video coding, and increase accuracy of object tracking. Thus, quality of viewing experiences for virtual reality applications and accuracy of 360 degree video analysis can be significantly improved.
