\ul
Space-time Reinforcement Network for Video Object Segmentation
Abstract
Recently, video object segmentation (VOS) networks typically use memory-based methods: for each query frame, the mask is predicted by space-time matching to memory frames. Despite these methods having superior performance, they suffer from two issues: 1) Challenging data can destroy the space-time coherence between adjacent video frames. 2) Pixel-level matching will lead to undesired mismatching caused by the noises or distractors. To address the aforementioned issues, we first propose to generate an auxiliary frame between adjacent frames, serving as an implicit short-temporal reference for the query one. Next, we learn a prototype for each video object and prototype-level matching can be implemented between the query and memory. The experiment demonstrated that our network outperforms the state-of-the-art method on the DAVIS 2017, achieving a score of 86.4%, and attains a competitive result 85.0% on YouTube VOS 2018. In addition, our network exhibits a high inference speed of 32+ FPS.
Index Terms:
Video object segmentation, memory-based methods, auxiliary frame, prototype learningI Introduction
Semi-supervised video object segmentation (VOS) stands as a challenging task in computer vision, drawing widespread attention in autonomous driving, robotics and video editing. The key to this task is to fully utilize the given limited signals between frames, where the first-frame annotation is provided by the user, and it segments objects in the remaining frames as accurately as possible.
Recently, matching-based methods [1, 2, 3] have achieved great success in semi-supervised VOS, achieving object segmentation through pixel-level matching between query frames and memory frames. Among these methods, memory-based networks [4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18] attracted a lot of attention. For instance, the space-time storage network (STM) [4] firstly is proposed to construct a feature memory for each object, and applies space-time matching between query and memory frames, which can better solve the problems of object occlusion and drift. After STM [4], there are many variants [8, 9, 10, 11, 14, 16, 17, 18] developed aiming to improve accuracy, reduce memory usage, and so on. Particularly, the latest related state-of-the-art methods, known as STCN [13] and XMem [15], share multiple memory stores for all objects to capture different spatial-temporal contexts, have achieved prominent performance.
Although these methods have achieved good results, there are still two issues that need to be carefully considered. Firstly, the existing methods [6, 11, 15] investigate the space-time coherence between adjacent video frames to exclude distractor objects, however the coherence is often completely destroyed due to occlusion, fast motion and irregular deformation. This can be referred to Fig.1(a), where the features between the query and its adjacent frame are inconsistent. Secondly, the dense pixel-level matching between the memory features and the query features will leads to undesired mismatching caused by the noises or distractors as shown in Fig.1(b). Thus, a high-level matching is needful as an auxiliary manner for solving the problem.
Motivated by the above discussions, we propose a Space-time Reinforcement Network (SRNet) for video object segmentation to address the two mentioned weaknesses. In more details, the core ideas of this paper come from two aspects: i) Instead of directly exploring the space-time coherence between adjacent video frames, we propose to generate an auxiliary frame from adjacent frames, serving as an implicit short-temporal reference for the query one. The proposed auxiliary frame is more consistent with the query frame as shown in Fig.1(a). ii) Beyond pixel-level matching, we propose to additionally learn a prototype for each video object. Thus, a high-level semantic matching, i.e., prototype-level matching, can be implemented between the query and memory to effectively improve the robustness of the method, also shown in Fig.1(b).
Our contributions can be summarized as follows:
-
•
We propose to generate an implicit auxiliary frame between adjacent frames to improve the space-time coherence.
-
•
We introduce a high-level semantic matching, i.e., prototype-level matching, to reduce undesired mismatching caused by the noises or distractors.
-
•
Our SRNet achieves state-of-the-art performance on DAVIS 2017 and competitive results on YouTube 2018, while maintaining a high inference speed of 32+ FPS.
II Method
Fig.2 shows an overview of our proposed method SRNet. We firstly outline the recent memory-based advanced VOS methods. Furthermore, we elaborate on the improvements made by our SRNet based on these methods.
Revisit memory-based VOS methods. Since the seminal work of STM [4], memory-based methods have emerged as the predominant solution for VOS. Specifically, they use the current frame as the query and store reference frames and masks in memory, where and represent the initial size of the query frame. Then, the prediction of the mask can be obtained by space-time matching between query and memory. We revisit the framework of memory-based methods, e.g., STCN [13] and Xmem [15], as they are one of the simplest and most effective memory-based methods.
As shown in Fig.2, given memory frames and a query frame, the query frame is passed to the query encoder to generate query key and query value . Meanwhile, memory frames and masks are fed to the memory encoder to obtain memory key and memory value , where and are spatial dimensions with a stride of 16, is the dimension of the key space and is the dimension of value space. Then, memory read performs space-time correspondence matching between and . We compute the affinity matrix :
(1) |
where indicates the similarity between the i-th memory pixel and the j-th query pixel, and is a similarity measure, e.g., distance.
Then, it multiplies with to obtain the memory feature for the query frame. Finally, is upsampled in decoder to generate prediction mask .
II-A Overview
As shown in Fig.2, our SRNet divides the memory into two blocks, the local memory stores the adjacent frame and its mask , while the global memory holds frames and their masks . Then we feed them into the memory decoder to obtain local memory key , local memory value , global memory key and global memory value . We further learn memory prototype by . Next, we pass , , and into a Feature Alignment Module (FAM) to obtain the value of auxiliary frame and compute the local feature . Then, the memory feature is obtained by space-time matching beween query and global memory. Through a Prototype Transformer Module (PTM), pixel-level feature , which is obtained by fusing and , are used for prototype-level matching with . They are updated iteratively to obtain the final prototype feature . Finally, decoder upsamples to generate prediction mask .
II-B Feature Alignment Module
We introduce FAM in this section, as shown in Fig.3, which aligns the adjacent frame with the query frame to obtain an auxiliary frame, serving as an implicit short-temporal reference for the query one. We do not pursue image-level alignment, but rather learn feature-level alignment between the adjacent frame and query frame. Specifically, what we are learning is not the auxiliary frame, but its value . Subsequently, We refine to obtain the local feature of the query frame.
Given , , and , a uniform grid of points is generated as a reference. Specifically, the grid size is downsampled by a factor based on the input feature map size, , . The value of the reference point is a linearly spaced 2D coordinate , and utilizing the grid shape is normalized to a range of [-1,+1], where (+1,+1) represents the lower right corner and (-1,-1) represents the upper left corner. In order to obtain the offset of each reference point, and are input into the lightweight separable convolution to generate the offset . Then, the local memory value and the offset are inputted into the sampling function to generate the value of auxiliary frame .
(2) |
Specifically, we will set to bilinear interpolation to make it differentiable:
(3) |
where and indices all positions on .
Given and , we estimate the convolutional kernel sampling offset and weight coefficient :
(4) |
where is a convolutional layer with a kernel size of 3 × 3, the sampling offset and weight coefficient respectively represent the positional shift and intensity fluctuation of each pixel in relative to .
Subsequently, given the value of the auxiliary frame , sampling offset and weight coefficient as inputs, local feature can be computed:
(5) |
where is a modulated deformable convolution layer[19].
YT-VOS 2018 Val | YouTube 2019 Val | DAVIS 2017 Val | DAVIS 2016 Val | DAVIS 2017 Test | ||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Model | FPS | |||||||||||||||||||
STM [4] | 79.4 | 79.7 | 84.2 | 72.8 | 80.9 | - | - | - | - | - | 81.8 | 79.2 | 84.3 | 89.3 | 88.7 | 89.9 | 72.2 | 69.3 | 75.2 | 11.1 |
CFBI [5] | 81.4 | 81.1 | 85.8 | 75.3 | 83.4 | - | - | - | - | - | 81.9 | 79.1 | 84.6 | 89.4 | 88.3 | 90.5 | 78.0 | 74.4 | 81.6 | 5.9 |
RMNet [6] | 81.5 | 82.1 | 85.7 | 75.7 | 82.4 | - | - | - | - | - | 83.5 | 81.0 | 86.0 | 88.8 | 88.9 | 88.7 | 75.0 | 71.9 | 78.1 | 4.4 |
HMMN [8] | 82.6 | 82.1 | 87.0 | 76.8 | 84.6 | 82.5 | 81.7 | 86.1 | 77.3 | 85.0 | 84.7 | 81.9 | 87.5 | 90.8 | 89.6 | 92.0 | 78.6 | 74.7 | 82.5 | 9.3 |
STCN [13] | 83.0 | 81.9 | 86.5 | 77.9 | 85.7 | 82.7 | 81.1 | 85.4 | 78.2 | 85.9 | 85.4 | 82.2 | 88.6 | \ul91.6 | 90.8 | \ul92.5 | 76.1 | 73.1 | 80.0 | 20.2 |
AOT [9] | 84.1 | 83.7 | 88.5 | 78.1 | 86.1 | 84.1 | 83.5 | 88.1 | 78.4 | 86.3 | 84.9 | 82.3 | 87.5 | 91.1 | 90.1 | 92.1 | 78.8 | 75.3 | 82.3 | 12.1 |
RDE [10] | - | - | - | - | - | 81.9 | 81.1 | 85.5 | 76.2 | 84.8 | 84.2 | 80.8 | 87.5 | 91.1 | 89.7 | 92.5 | 77.4 | 73.6 | 81.2 | 27.0 |
XMem [15] | 85.7 | 84.6 | 89.3 | 80..2 | 88.7 | 85.5 | 84.3 | 88.6 | 80.3 | 88.6 | \ul86.2 | \ul82.9 | \ul89.5 | 91.5 | 90.4 | 92.7 | 81.0 | 77.4 | 84.5 | 22.6 |
SRNet | \ul85.0 | \ul83.9 | \ul88.5 | \ul79.7 | \ul87.9 | \ul84.7 | \ul82.9 | \ul87.4 | \ul80.3 | \ul88.3 | 86.4 | 83.1 | 89.7 | 91.8 | \ul90.6 | 93.0 | \ul79.2 | \ul76.5 | \ul81.9 | 32.3 |
FAM | PTM | FPS | ||
---|---|---|---|---|
✓ | ✓ | 86.4 | 85.0 | 32.3 |
✓ | 86.0 | 84.6 | 37.5 | |
✓ | 85.8 | 84.3 | 38.7 |
II-C Prototype Transformer Module
In fact, pixel-level matching will leads to undesired mismatching caused by the noises or distractors. To address this issue, we propose a Prototype Transformer Module (PTM) as shown in Fig.4 which performs prototype-level matching between query and memory.
We can use mask annotations on the memory frames to learn the foreground prototype. There are two strategies for utilizing segmentation masks, namely early fusion and late fusion. Early fusion will mask the supporting images before inputting them into the feature extractor. Late fusion directly masks feature maps to generate foreground features separately. In this work, we adopt an early fusion strategy because we can reuse the obtained through the memory encoder.
Specifically, given , the formula for calculating the initial prototype and initial pixel feature :
(6) |
(7) |
where, is a 2-layer, -dimensional MLP, is a 2-layer, 1-dimensional MLP and includes two ResBlocks and a CBAM block.
Our PTM has learnable prototype and pixel feature , which iteratively updated by initial prototype and initial pixel feature through cross attention. We added position embeddings to the key of each attention layer.
(8) |
(9) |
where, , , , is the learnable parameters in SPA.
Then, guided by the updated prototype , the semantic information of the corresponding pixels in pixel feature is activated. Specifically, is extended and combined with to activate the target area:
(10) |
where is a simple activation network consisting of two 3 × 3 convolutional layers and one channel attention layer.
Finally, we use as the new and as the new , and update them three times to obtain the final prototype feature .
III Experiment
For assessment purposes, we utilize standard metrics: the Jaccard index , contour accuracy , and their combined average . In YouTubeVOS [20], the computation of and is performed separately for “seen” and “unseen” categories. represents the averaged across both “seen” and “unseen” categories.
III-A Quantitative Comparison
Table I tabulate our results on YouTube 2018/2019 [20] validation, DAVIS 2016/2017 [21] validation, and DAVIS 2017 test-dev. On the DAVIS 2017 and 2016 validation set, our SRNet outperforms the baseline STCN [13] by 1% and 0.2% in and runs about 60% faster and outperforms the state-of-the-art method Xmem [15] by 0.2% and 0.3% in and runs about 43% faster. On the YouTube 2018 validation and DAVIS 2017 test-dev set, SRNet performs 2% and 2.1% better on than STCN [13], but 0.7% and 1.8% less than Xmem [15]. Thanks to setting our at 256, rather than 512, our inference speed far exceeds other methods. We choose STCN [13] as the baseline because its GPU usage is 20% lower than XMem [15] and nearly twice as fast when training.
III-B Qualitative Comparison
We choose two videos as examples, one from DAVIS 2017 validation and another from YouTube 2018. and present the segmentation results in comparison with SRNet, STCN [13] and Xmem [15] in Fig.5. As can be seen, our SRNet produces more accurate masks: similar objects can be distinguished, and undesired mismatching can be reduced.
III-C Ablations
We conduct ablation experiments on DAVIS 2017 validation and YouTube 2018 validation. In particular, we report and FPS for DAVIS 2017, alongside for YouTube 2018.
Ablation study on the two modules of SRNet is conducted to evaluate their individual effectiveness, as shown in Table II. When two modules are activated (i.e., in the default configuration), scoring 86.4% on DAVIS 2017 and 85.0% on YouTube 2018, with an inference speed reaching 32.3 FPS. When FAM is disabled, the result decreases 0.4% and 0.4%. Then, When PTM is disabled, the result decreases 0.6% and 0.7%.
To verify the impact of sampling intervals of local memory in FAM, we choose different intervals for the experiment, as shown in the Fig.6.
An analysis of different numbers of PTM layers on the effectiveness of our network in Fig.7. In order to better balance accuracy and FPS, we choose to iterate three times. Additionally, we visualized each layer’s auxiliary mask in Fig.8. It can be observed that the object becomes more coherent (red arrows), and errors are suppressed (yellow arrows). In Fig.9, it can be seen that the distance between pixels of the same class in pixel feature is far, while the distance between pixels of different classes is close. Obviously,the prototype feature after PTM has effectively reduced intra-class variation and enlarged inter-class difference.
III-D Limitations
As shown in Fig.10, SRNet typically fails in small-scale foreground objects. We believe this is mainly because our method cannot capture fine-grained information. We hope to find more effective methods to distinguish small objects and achieve better performance in future work.
IV Conclusion
In this paper, we tackle the issues of memory-based methods for semi-supervised video object segmentation (VOS). For improving the space-time coherence, we first propose to generate an implicit auxiliary frame between adjacent frames. Secondly, we introduce a high-level semantic matching, i.e., prototype-level matching to reduce undesired mismatching caused by the noises or distractors. The experiment results show that our SRNet achieves competitive performance.
V Acknowledgement
This work was supported in part by the National Natural Science Foundation of China under Grant 62332015, and Grant 62072449; in part by the Science and Technology Development Fund, Macau, SAR, under Grant 0075/2023/AMJ, Grant 0003/2023/RIB1, and Grant 001/2024/SKL; in part by the Zhuhai Science and Technology Innovation Bureau under Grant ZH2220004002524; and in part by the University of Macau under Grant MYRG2022-00059-FST and Grant MYRG-GRG2023-00237-FST-UMDF.
References
- [1] P. Voigtlaender, Y. Chai, F. Schroff, H. Adam, B. Leibe, and L.-C. Chen, “Feelvos: Fast end-to-end embedding learning for video object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9481–9490.
- [2] L. Fu, Z. Li, Q. Ye, H. Yin, Q. Liu, X. Chen, X. Fan, W. Yang, and G. Yang, “Learning robust discriminant subspace based on joint - and -norm distance metrics,” IEEE transactions on neural networks and learning systems, vol. 33, no. 1, pp. 130–144, 2020.
- [3] Z. Yang, Y. Wei, and Y. Yang, “Collaborative video object segmentation by foreground-background integration,” in European Conference on Computer Vision. Springer, 2020, pp. 332–348.
- [4] S. W. Oh, J.-Y. Lee, N. Xu, and S. J. Kim, “Video object segmentation using space-time memory networks,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, pp. 9226–9235.
- [5] Z. Yang, Y. Wei, and Y. Yang, “Collaborative video object segmentation by multi-scale foreground-background integration,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 44, no. 9, pp. 4701–4712, 2021.
- [6] H. Xie, H. Yao, S. Zhou, S. Zhang, and W. Sun, “Efficient regional memory network for video object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1286–1295.
- [7] Q. Huang, L. Shen, R. Zhang, J. Cheng, S. Ding, Z. Zhou, and Y. Wang, “Hdmixer: Hierarchical dependency with extendable patch for multivariate time series forecasting,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 38, no. 11, 2024, pp. 12 608–12 616.
- [8] H. Seong, S. W. Oh, J.-Y. Lee, S. Lee, S. Lee, and E. Kim, “Hierarchical memory matching network for video object segmentation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 889–12 898.
- [9] Z. Yang, Y. Wei, and Y. Yang, “Associating objects with transformers for video object segmentation,” Advances in Neural Information Processing Systems, vol. 34, pp. 2491–2502, 2021.
- [10] M. Li, L. Hu, Z. Xiong, B. Zhang, P. Pan, and D. Liu, “Recurrent dynamic embedding for video object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1332–1341.
- [11] Y. Chen, D. Zhang, Z.-x. Yang, and E. Wu, “Robust and efficient memory network for video object segmentation,” arXiv preprint arXiv:2304.11840, 2023.
- [12] Q. Ye, P. Huang, Z. Zhang, Y. Zheng, L. Fu, and W. Yang, “Multiview learning with robust double-sided twin svm,” IEEE transactions on Cybernetics, vol. 52, no. 12, pp. 12 745–12 758, 2021.
- [13] H. K. Cheng, Y.-W. Tai, and C.-K. Tang, “Rethinking space-time networks with improved memory coverage for efficient video object segmentation,” Advances in Neural Information Processing Systems, vol. 34, pp. 11 781–11 794, 2021.
- [14] Y. Chen, C. Hao, Z.-X. Yang, and E. Wu, “Fast target-aware learning for few-shot video object segmentation,” Science China Information Sciences, vol. 65, no. 8, p. 182104, 2022.
- [15] H. K. Cheng and A. G. Schwing, “Xmem: Long-term video object segmentation with an atkinson-shiffrin memory model,” in European Conference on Computer Vision. Springer, 2022, pp. 640–658.
- [16] M. Li, L. Hu, Z. Xiong, B. Zhang, P. Pan, and D. Liu, “Recurrent dynamic embedding for video object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1332–1341.
- [17] K. Park, S. Woo, S. W. Oh, I. S. Kweon, and J.-Y. Lee, “Per-clip video object segmentation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 1352–1361.
- [18] Y. Chen, D. Zhang, Y. Zheng, Z.-X. Yang, E. Wu, and H. Zhao, “Boosting video object segmentation via robust and efficient memory network,” IEEE Transactions on Circuits and Systems for Video Technology, 2023.
- [19] X. Zhu, H. Hu, S. Lin, and J. Dai, “Deformable convnets v2: More deformable, better results,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 9308–9316.
- [20] N. Xu, L. Yang, Y. Fan, D. Yue, Y. Liang, J. Yang, and T. Huang, “Youtube-vos: A large-scale video object segmentation benchmark,” arXiv preprint arXiv:1809.03327, 2018.
- [21] J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool, “The 2017 davis challenge on video object segmentation,” arXiv preprint arXiv:1704.00675, 2017.