本研究提出多運算子影像與視訊尺寸調整(retargeting)演算法，目的在於有效率地調整影像畫面至目標解析度，並將演算法延伸應用於視訊。對於數位影像，我們適當地施予基於內容之邊緣裁切(content-based cropping)和縮放(scaling)，首先計算影像中的視覺顯著特徵(visual saliency feature)，並將影像透過SLIC(Simple Linear Iterative Clustering)演算法切割成較大的超級像素(superpixel)，擷取畫面中的前景物作為畫面切割的依據，接著逐一比較視覺特徵圖進行邊緣裁切與等比例縮放。若時間允許，圖縫裁減(seam carving)可被使用讓畫面更接近目標長寬比。圖縫裁減主要計算畫面梯度，採用動態規劃刪除最小能量圖縫並進行圖縫的局部更新，最後定義突出點以限制圖縫數量並決定裁減停止點。對於某些適合的影像，我們亦可增加圖縫來降低畫面直接縮放程度。由實驗結果顯示，我們確實有效率地維持影像主體，演算法也達到較高的實用性。另外，我們將影像處理延伸至視訊資料，考量視訊壓縮域動態資料計算，透過H.264/AVC視訊壓縮編碼時所產生的運動向量(motion vector)和運動補償資訊(motion compensation)判斷鏡頭種類，若為非固定式場景，我們使用邊緣裁切以及縮放的方式處理畫面；若為固定場景，則可使用圖縫裁減機制。為了防止運動中的前景物在裁切過程中被移除而造成失真，我們將壓縮域中的位移向量製作運動特徵圖(motion feature map)，結合視覺特徵圖協助圖縫裁減和邊緣裁切。實驗結果顯示我們的方法可以廣泛處理不同種類的鏡頭，在畫面前景物形狀的維持以及背景保留上，亦優於其他視訊畫面調整演算法。;This research presents a multi-operator image retargeting scheme, which can be further expanded to video retargeting. The objective is to effectively and efficiently adjust the image or video frame to the targeted resolution. Given an image or frame, the content-based cropping and scaling will be applied. The visual saliency map is calculated and the superpixels are formed via Simple Linear Iterative Clustering (SLIC) to serve as the reference to extract the visually significant foreground objects. Next, the degree of cropping and scaling will be determined by the saliency map. Seam carving can also be employed to make the resolution closer to the target if the efficiency is not an important issue. Seam caving checks the one-directional gradients and uses dynamic programming to remove the saliency with minimal significance. Local update helps to reduce the computational burden. Saliency points are identified and helps to decide when to stop the seam carving process. For certain images, inserting seams is also useful to decrease the the degree of scaling. Experimental results show that the proposed method does maintain the significant objects of the image and is also more feasible.
For video retargeting, the data in compressed video stream, including the motion vectors and motion compensation, are used to classify the types of shots. If the shot belongs to a fixed scene, seam carving can be applied. Otherwise, only cropping and scaling are used. To avoid removing the foreground objects, the motion feature map is formed, combined with the visual saliency map, to achieve seam carving and cropping. The experimental results shows that the proposed scheme can deal a variety of shots and outperform existing algorithms.
