1st Place Solution for MeViS Track in CVPR 2024 PVUW Workshop: Motion Expression guided Video Segmentation

Abstract

Motion Expression guided Video Segmentation (MeViS), as an emerging task, poses many new challenges to the field of referring video object segmentation (RVOS). In this technical report, we investigated and validated the effectiveness of static-dominant data and frame sampling on this challenging setting. Our solution achieves a $\mathcal{J}\&\mathcal{F}$ score of 0.5447 in the competition phase and ranks 1st in the MeViS track of the PVUW Challenge. The code is available at: https://github.com/Tapall-AI/MeViS_Track_Solution_2024.

1 Introduction

Pixel-level Video Understanding in the Wild (PVUW) is a workshop providing large-scale datasets, competitions, and discussions for video scene parsing, one of the fundamental problems in computer vision. Since 2021, PVUW has encouraged much improvement in video semantic segmentation and video panoptic segmentation. This year, two new subjects join PVUW: 1) Complex Video Object Segmentation (MOSE) [7] and 2) Motion Expression guided Video Segmentation (MeViS) [6], supplementing PVUW with object-centric pixel-level video understanding, vital to many real-world applications such as video editing and human-computer interactive systems. With large-scale videos, diverse/realistic challenges, and high-quality annotations, MOSE and MeViS build a competitive platform and encourage comprehensive and robust solutions.

This technical report focuses on the MeViS subject: Motion Expression guided Video Segmentation, which aims to segment target objects in videos, guided by natural language expressions. Before MeViS, several benchmarks [9, 14, 22] have been proposed to encourage explorations in this field. Despite fostering surging research works, these benchmarks focus more on short videos with less same-category objects and static attributes (e.g., location and appearance). As a result, frame-level segmentation also predicts high-quality masks. This would weaken the investigation of temporal properties, vital to understanding real-world videos.

Recently, MeViS (Motion expressions Video Segmentation) [6] has been proposed to emphasise temporal properties in RVOS. Compared with previous benchmarks, MeViS brings several unique challenges: 1) Motion-dominant language expressions, 2) Complex scenes with multiple same-category instances, 3) One-to-more text-object pairs, and 4) Long videos. These challenges encourage RVOS to focus on dynamic attributes, comprehensive multi-modal interactions, and efficient long-term video understanding.

The success of MTTR [1] and ReferFormer [25] motivates the community to consider transformer-based end-to-end architecture [2, 31] as the dominant paradigm. Given an input video and text, the paradigm encodes object queries from all frames and decodes text-relevant ones into masks. The difference between MTTR and ReferFormer lies in query encoding: The former encodes general object queries, and the latter considers language-guided queries, enlightening most following works. These could be roughly divided into two categories: 1) Robust vision-language alignments [16, 15, 20], which focus on the alignments between visual/textual properties; and 2) Temporal-aware interactions [10, 17, 27], where improvements leverage spatial and temporal properties to segment the targets. The more recent ones, SOC [17] and MUTR [27], achieve SoTA performance on previous benchmarks due to their efficient interactions between object sequences and texts. This idea comes from VITA [12], a video instance segmentation method, which also inspired the latest RVOS SoTA: DsHMP [11]. Despite good results, they struggle with MeViS since they are trained on static-dominant data. In addition, MeViS consists of long videos, challenging RVOS in comprehensive video understanding and efficiency.

With these new and realistic challenges, this report improves existing RVOS methods in training and inference schemes. Specifically, we consider MUTR [27] as the baseline. With pre-trained weights on Ref-COCO series [30, 19] and Ref-YouTube-VOS [22], we fine-tune them on MeViS. Masks with one-to-more text-object pairs are considered as a whole to encourage adaptive object perception based on texts. To balance comprehensive understanding and efficiency, we split long input videos into sub-videos via frame sampling. With these improvements, our solution ranks 1st in the MeViS Track.

Experiments on the MeViS valid set indicate that previous RVOS data still contribute to this challenging setting due to their sufficient and well-aligned object masks and texts. In addition, ablations on sampling schemes reveal that there is much room for improvement in temporal modelling over long videos. Specifically, limited by computational resources, the temporal modules are trained with pseudo videos with less frames. During inference, however, videos have much more temporal contexts. This inconsistency leads to considering fewer frames (sampled) in temporal modules outperform the one with all frames. We hope these findings are helpful for future research.

2 Related Works

This sections overviews representative methods and trends in referring and semi-supervised video object segmentation.

2.1 Referring Video Object Segmentation

Recent RVOS methods build their end-to-end architectures upon transformers. Specifically, given input videos and texts, the methods initialise fixed-number object queries to integrate vision-language contexts. Queries from different frames are considered a trajectory if they have the same index. The trajectory best matching with texts is decoded into masks on each frame. As pioneer works, MTTR [1] and ReferFormer [25] build foundation architectures with visual/textual encoders, multi-modal transformers, and mask decoders, motivating many following works. They improve RVOS in mainly two aspects: 1) Robust multi-modal alignments and 2) Temporal-aware interactions.

With cyclic structural consensus, R2VOS [16] shows better results when text-referred objects are absent from frames. In SgMg [20], a spectrum-based multi-modal attention is proposed to improve query-guided mask predictions. FS-RVOS [15] improves RVOS to adapt new visual/textual concepts via the cross-modal affinity. As a versatile model, UNINEXT [26] unifies different object perception tasks and data, generalising well on previous RVOS benchmarks.

Previous methods rarely consider temporal properties or achieve this implicitly, e.g., with video-swin-transformer as backbone. In HTML [10], vision and language information are interacted over hierarchical temporal contexts. Motivated by VITA [12], which validates video understanding can be achieved via associating frame-level objects, recent methods encode temporal properties only from object queries, leading to end-to-end and efficient architectures: SOC [17], MUTR [27], and DsHMP [11]. The former two emphasise mutual multi-modal fusion and achieve SoTA performance on previous benchmarks, while the latter perform hierarchical multi-modal interactions and show high-quality results on ALL RVOS benchmarks.

2.2 Semi-supervised Video Object Segmentation

Unlike RVOS, which specifies the target objects with texts, semi-supervised video object segmentation (SVOS) considers manually annotated masks (usually on the first frame) as targets [8]. Therefore, SVOS methods focus on dense correspondence between frames and can propagate high-quality masks from one or several frames to the whole video.

With this feature, most winner solutions [24, 18] from previous RVOS competitions use SVOS methods to refine their results. In brief, they first select high-confident masks from overall predictions. Then, the masks are propagated to remaining frames to refine their corresponding results. The intuition behind the idea is that the offline RVOS methods struggle to generate spatial-temporal consistent object masks. This could be mitigated significantly via powerful SVOS methods (once the selected masks are high-quality).

Memory-based paradigm (since STM [21]) has dominated SVOS due to its efficient, robust, and dense correspondence between frames. In particular, the paradigm considers not only the first frame annotations but also predictions from intermediate frames as references. This way, SVOS could better adapt to object changes.

Figure 1: Overview of our solution. Given an input video, we divide all frames into $N$ subsets via non-continuous sampling. Here we take two subsets as an example. They are marked with Blue and Green boxes. In particular, each subset is segmented individually, guided by the input text, and combined for the final results.

Earlier SoTAs [23, 4] improve STM with robust cross-frame correspondence. The recent focus has been shifted to more challenging and realistic settings: long videos and complex scenes, motivating high-quality benchmarks [7, 13] and solutions. Specifically, XMem [3] is proposed to segment long videos with dynamic memory management. AOT series [28, 29] consider object representations to enhance the robustness against complex scenes. By integrating object queries into dense correspondence, Cutie [5] significantly reduces the matching noise between frames and achieves the SoTA SVOS performance.

3 Method

Fig. 1 shows our solution, where we consider MUTR [27] as the base model, with Swin-Transformer-Large as vision encoder and RoBERTa-base as text encoder. Given an input video with $T$ frames ($\mathcal{V}=\{v_{t}\in\mathbb{R}^{H\times W\times 3}\}^{T}_{t=1}$) and referring text $\mathcal{E}=\{e_{i}\}^{L}_{i=1}$ with $L$ words, we first sample $\mathcal{V}$ into $N$ subsets: $\{\mathcal{V}_{n}\}_{n=1}^{N}$. Then, we segment each subset individually under the guidance from $\mathcal{E}$, achieving mask subsets: $\{\mathcal{M}_{n}\}_{n=1}^{N}$. Finally, the masks are combined for the final predictions: $\mathcal{M}=\{m_{t}\in\mathbb{R}^{H\times W}\}_{t=1}^{T}$.

Training details.

With MUTR’s weights jointly trained on Ref-COCO [30], Ref-COCO+ [30], Ref-COCOg [19], and Ref-YouTube-VOS [22], we perform fine-tuning on MeViS training videos. For the expressions specifying multiple objects, we consider all masks as a whole to encourage the model to perceive and segment all objects from videos. To better leverage pre-trained parameters, we follow MUTR to sample five frames as a pseudo video and use the same losses. The fine-tuning is performed for two epochs, where the learning rate is reduced to 10% during the last one.

Inference details.

Given an input video, we resize each frame to keep its shorter size at 360. Unlike previous RVOS benchmarks, MeViS videos consist of much frames and thus cannot be inferred with one feed-forward pass. As diagrammed in Fig. 1, we sample the video into $N=T\mid T_{c}$ subsets and perform referring segmentation individually. $T_{c}=30$ is the length of each subset.

4 Experiments

This section first shows our quantitative and qualitative results on the MeViS test set. Then, we provide ablations on MeViS validation set to show the solution’s effectiveness and try to derive some insights for future research.

4.1 Main Results

Tab. 1 shows our quantitative results on the MeViS test set.

Team	$\mathcal{J}\&\mathcal{F}$	$\mathcal{J}$	$\mathcal{F}$
Tapall.ai	0.5447 (1)	0.5048 (2)	0.5846 (1)
BBBiiinnn	0.5420 (2)	0.5097 (1)	0.5743 (2)
PPPPPsanG	0.5280 (3)	0.4853 (3)	0.5707 (3)
times	0.5151 (4)	0.4610 (4)	0.5691 (4)
Phan	0.5075 (5)	0.4562 (5)	0.5588 (5)
LIULINKAI	0.4267 (6)	0.3927 (6)	0.4607 (6)
ntuLC	0.3700 (7)	0.3407 (7)	0.3994 (7)

Table 1: Quantitative results on the MeViS test set.

Method	Backbone	Prev.	MeViS	$\mathcal{J}\&\mathcal{F}$
MUTR [27]	Swin-L	✓	✗	0.4343
MUTR [27]	Swin-L	✓	✓	0.4857
SOC [17]	V-Swin-B	✓	✗	0.4394
SOC [17]	V-Swin-B	✓	✓	0.4664

Table 2: Ablations on training data. ‘Prev.’ indicates the use of Ref-COCO, Ref-COCO+, Ref-COCOg, and Ref-YouTube-VOS.

Method	1	5	10	20	30	40
N-continuous	0.4725	0.4788	0.4811	0.4849	0.4857	0.4864
Continuous	0.4725	0.4831	0.4855	0.4875	0.4864	0.4892
No sampling	0.4607	0.4685	0.4730	0.4792	0.4822	0.4732

Table 3: Ablations on sampling methods. Five sub-video lengths are considered. The highlighted results and the results evaluated in Tab. 1 come from the same model.

Figure 2: Difference between (a) Non-continuous sampling, (b) Continuous sampling, and (c) No sampling. Each box here denotes one frame, and the same colour boxes are sampled as pseudo videos for referring segmentation. Note that the best object trajectory selection is performed individually in each sampled video. For ‘No sampling’, we still divide videos into subsets and predict masks upon those. During the selection, we feed all object queries into temporal modules and consider the resulting probabilities to select the best mask trajectory.

4.2 Ablations

To validate the effectiveness of our solution, we show results in Fig. 3 and ablations on the MeViS valid set.

Figure 3: Qualitative predictions of our solution on the MeViS valid set. Orange and Green masks are the predictions guided by the texts with the same colour. The percentage indicates the position of corresponding frame in the video.

Figure 4: Qualitative ablations on training data. The percentage indicates the position of corresponding frame in the video.

Figure 5: Qualitative ablations on subset video length. The percentage indicates the position of corresponding frame in the video.

Training method.

Tab. 2 compares $\mathcal{J}\&\mathcal{F}$ on different training data. To generalise the conclusion, we take another RVOS method with temporal properties (SOC [17]) into account. MUTR and SOC share the same training and inference procedure. It is observed that the training data in previous benchmarks still contribute to this challenging setting, due to their sufficient and well-aligned object-text pairs. Fig. 4 shows that previous benchmarks enable RVOS methods to perceive objects. With MeViS and unified mask supervision, the methods work on more challenging applications with motion expressions or ones with plural nouns.

Sampling method.

Tab. 3 ablates methods and hyper-parameters for sampling frames. The difference between these methods is diagrammed in Fig. 2. Although temporal modules in MUTR enable us to collect and infer long-term object queries from videos, they are only trained with pseudo videos with five frames. The gap between training and inference temporal contexts struggles with temporal interactions over long videos. Results in Tab. 3 and Fig. 5 show that temporal modules works better than frame-level predictions (sub-video length=1) but the performance cannot be improved further with more temporal contexts.

5 Conclusion

This technical report explores the value of training data and temporal contexts for the challenging MeViS benchmark. The competitive results and ablations demonstrate that the well-aligned object-text data (even with primarily the static attributes) are helpful in motion expression-guided referring video segmentation. In addition, we investigate the effectiveness of temporal contexts and reveal room for improvement in the temporal multi-modal analysis of long videos.