How is Visual Attention Influenced by Text
Guidance? Database and Model

In order to understand and model visual attention behavior, many eye-tracking databases have been constructed under the free-viewing task. MIT1003 [23] is a large-scale saliency database that contains 1003 images coming from Flickr and LabelMe. MIT300 [24] and CAT2000 [25] are two widely used benchmark databases, which contain 300 and 2000 test images respectively. SALICON [22] is currently the largest crowd-sourced saliency database, which contains 10000 training images, 5000 validation images, and 5000 test images, which is widely adopted for pretraining saliency prediction models. The saliency data is collected through mouse tracking using Amazon Mechanical Turk (AMT).

In our previous work [38], we have established a text-image saliency database. As shown in Table I, the previous dataset mainly aims to investigate whether the text description can influence the saliency, thus only contains one pure image saliency map and one text-image saliency map for one image. Since the previous work has validated the influence of texts on image saliency, our new dataset SJTU-TIS further aims to investigate whether and how different texts can influence the image saliency, which has not been involved and studied in the previous work. Moreover, the current database is a rebuilt dataset with more images. To investigate the influence of different texts on image saliency, we remove some images and keep about 200 images from the previous dataset. In summary, the SJTU-TIS database contains 300 pure image saliency maps and 300 text-image saliency maps for a group of 300 images, and includes 300 pure image saliency maps and 300$\times$3 text-image saliency maps for another group of 300 images.

Most traditional methods model visual saliency based on the bottom-up mechanism, which generally extract simple low-level feature maps, such as intensity, color, direction, etc., and integrate them to generate saliency maps. Itti et al. [39] considered underlying features on multiple scales to predict saliency maps. Harel et al. [26] introduced a graph-based saliency model, which defined Markov chains on various image maps and regarded the balanced distribution of map locations as activation values and saliency values. Many other classical methods such as AIM [40], SMVJ [41], CovSal [42], SeR [43], HFT [44], etc., are also commonly used saliency prediction models.

With the development of deep neural networks (DNN), saliency prediction tasks have made significant progress in recent years [45, 46]. Huang et al. [30] used VGG as the backbone and proposed a two-stream network to extract coarse features and fine features to calculate saliency maps. Cornia et al. [47] proposed an Attentive ConvLSTM, which focuses on different spatial positions of a bunch of features to enhance prediction. Pan et al. [48] proposed the generative adversarial network (GAN) to calculate the saliency map. Che et al. [49] studied the influence of transformation on visual attention and proposed a GazeGAN model based on the U-Net for saliency prediction. Duan et al. [1] proposed a vector quantized saliency prediction method and generalized it for AR saliency prediction. These top-down-based saliency prediction models have been widely used in various research fields in recent years [50].

We analyze four image attributes, including contrast, colorfulness, spatial information, and brightness to characterize the content diversity of the images in the SJTU-TIS database. The image attributes of the SJTU-TIS and SALICON databases [22] are shown in Fig. 6. It can be observed that our SJTU-TIS database covers a wide range of content diversity.

If the overall sampling rate of the eye movement is less than 90%, the data and the subject will be regarded as outlier. None of the 60 subjects is identified as outlier and removed from the experiment. We first overlay all fixation points of one image fixated by all viewers into one map to generate the fixation map of this image. Then the fixation map is smoothed with a 1${}^{\circ}$ Gaussian kernel to obtain a continuous fixation density map (visual attention map).

Fig. 5 shows some schematic diagrams of the fixation maps. We use green points to represent the fixations obtained under the pure image condition and red points to represent the fixations obtained under the text-guided condition. The first line represents the general scenario descriptions (Type1), while the second to fourth rows represent three different descriptions of the same image (Type2-Type4). It can be observed that for Type1, there is not much difference between the distributions of red and green points, which are both relatively uniform. For Type2, under the specified descriptions of salient objects, red points are more concentrated on the described object compared to green points, such as the meat in the third image. For Type3, when describing non-salient objects, there is a significant difference between the red points and green points. Green points are rarely distributed on the described non-salient object, while red points are concentrated on the described object, such as the apple slices in the third image. For Type4, under the condition of common descriptions containing both salient and non-salient objects, the red points will partially shift to non-salient objects, while most of them concentrated on salient objects.

In order to understand and predict visual attention behavior, many eye-tracking databases have been constructed in recent years. In this paper, two databases are used to validate the effectiveness of the proposed TGSal model, including the largest publicly available saliency database SALICON [22] which is constructed for the general saliency prediction purpose, and the proposed SJTU-TIS database which is established for the text-guided saliency prediction purpose.

In the field of visual attention and saliency prediction, many consistency metrics are generally adopted to evaluate the performance of saliency algorithms. We select seven commonly used metrics including AUC-J, sAUC, CC, IG, KL, NSS, and SIM [57]. These saliency evaluation metrics can be categorized into two types including the location-based metrics and the distribution-based metrics [58, 57, 59], as summarized in Table II. Location-based metrics consider saliency values at discrete fixation locations, while distribution-based metrics consider both ground truth fixation density maps and predicted saliency maps as continuous distributions.

We first conduct experiments on SALICON [22] to validate the general effectiveness of our TGSal model. We train the model only on the training set of SALICON and test the model performance on the validation set of it. We further conduct a cross-dataset evaluation experiment by training the model on the training set of the SALICON [22] and testing it on MIT1003 [23]. All comparative deep learning models mentioned in the Sections V-B and V-C are trained and tested using the same way. During the training process, we set the text input of our TGSal model as empty and freeze the text feature extraction module and the image feature extraction module. The model is trained on SALICON for a total of 20 epochs, with an initial learning rate of 1$e^{-4}$, and gradually reduces the learning rate to 1$e^{-5}$ with the cosine annealing [66, 67, 68]. The hyperparameters $\alpha$ and $\beta$ in the loss function are empirically set as 0.06 and 1 respectively, to balance the contributions of the two loss items.

Then we conduct experimental validation on the SJTU-TIS database. As aforementioned, our database contains five types of visual attention data, which includes visual attention on pure images, visual attention on general scenario text-guided images, visual attention on images guided by specified text descriptions of salient objects, visual attention on images guided by specified descriptions of non-salient objects, visual attention on images guided by common descriptions containing both salient and non-salient objects. Therefore, it is necessary to consider both the individual training strategy and the joint training strategy on five conditions. Firstly, We use the weights pretrained on SALICON [22] as the initial weights, and then individually finetune the model on five conditions. Secondly, we still use the weights pretrained on SALICON [22] as the initial weights, but then jointly finetune the model on five conditions. Finally, in order to validate that the proposed database can independently support training and evaluation, we further perform the joint training procedure on five categories, however, without pretraining on SALICON [22]. During the experiments, we randomly split each group of data into two sets: a training set (80% of data) and a testing set (20% of data), without overlap** during the experiment. Each experiment is repeated 10 times to prevent performance bias in each group, and mean results are recorded for comparison.

It should be noted that all of the DNN models are first pretrained on SALICON [22] and then finetuned on the five groups of our SJTU-TIS database. Table IV shows the quantitative comparisons of different models under the pure image condition and the general scenario description condition. Table V presents the quantitative comparisons of different models under the condition of specified descriptions of salient objects and the condition of specified descriptions of non-salient objects. Table VI demonstrates the quantitative comparisons of different models under the condition of common descriptions containing both salient and non-salient objects and averaged among all conditions. In these tables, TGSal-I means leaving the text blank and only inputting images to the network, while TGSal means the complete network. Detailed analyses are given below.

Firstly, as shown in Table IV, in the case of pure images and general scenario descriptions, the unimodal baseline models, especially SAM-VGG also attain good performance. However, as shown in Tables V and VI, these models have relatively poor performance, especially in terms of the CC and SIM metrics. These phenomena indicate that the text-guided saliency prediction is a difficult task for the unimodal baseline models. Moreover, the performance of all models under the conditions of the salient-object description and the common description containing both salient and non-salient objects is relatively better than under the non-salient-object description condition. This is mainly due to that the salient objects generally occupy a larger or more central area in the image, while non-salient objects are usually small objects. Generally, it is difficult for deep neural networks to extract features for small objects compared to large objects, which makes it difficult to concentrate on non-salient object. We can also observe that deep learning models achieve better performance than traditional models in most cases, and mutimodal model performs better in most conditions among the baseline deep learning models.

Secondly, from the quantitative perspective, our proposed model achieves the best performance on all five subsets, verifying the great representation of learning ability under different description scenarios, and when the text is left blank, our model TGSal-I still outperforms other unimodal baseline models. The qualitative results shown in Fig. 10 also manifest the superiority of our TGSal. Moreover, comparing the TGSal-I and TGSal model, we can observe that the TGSal model outperforms TGSal-I under all text-induced conditions. This phenomenon manifests that the introducing text information can improve the performance of predicting text-guided visual saliency, and our proposed text-image feature fusion modules can effectively introduce text information into the image saliency prediction task. To quantify the performance improvement, we compare TGSal (utilizing both image and text as input) with TGSal-I (only image input) under four conditions, respectively. As illustrated in Table VII, the performance improvement percentage on the non-salient description set (Type3) is significantly greater than that in other categories, indicating that the text features provide more assistance for the saliency prediction under the “specific descriptions of non-salient objects” condition. However, the influence of text features on Type1 (general scenario descriptions) and Type4 (common descriptions containing both salient and non-salient objects) is comparatively modest.

We further conduct experiments considering the joint training strategy. Specifically, we first pretrain the model on SALICON [22], and then jointly finetune it on all conditions of SJTU-TIS, which can further eliminate the description category learning bias and improve the generalization capability of the TGSal model. The results are shown in Tables VIII, IX and X. Since the cross-attention map cannot be used in the pure image condition, we only report the results of ALBEF-CAM [63] and BLIP-CAM [64] on Type1 to Type4 in Tables VIII, IX, and X. First, we observe that the CAM-based saliency prediction method for the multimodal text-image pretraining models can generate significant results, but the performance of most metrics is not comparable to the state-of-the-art methods. The finetuned three multimodal-based methods including ALBEF [63], BLIP [64], and ImageBind [65] can achieve the comparable performance, and may even perform better in terms of some metrics, compared to unimodal saliency prediction methods. Moreover, as a multimodal saliency prediction method, our TGSal achieves better performance compared to other three multimodal models, i.e., ALBEF [63], BLIP [64], and ImageBind [65]. This is reasonable since BLIP [64] is a pretrained model designed for image and text visual question and answering (VQA), and ImageBind [65] is a pretrained model that contains more modalities besides image and text modalities. Their features contain more redundant information compared to CLIP [52], which makes it hard to train to select right features. Finally, we observe that the overall performance of the jointly trained model is slightly lower than the performance of the individually trained models, shown in Tables IV, V, and VI, which is mainly due to the description mixing.

In order to validate that the proposed database can independently support training and evaluation without the help of other databases, we further train all models jointly on all conditions of the SJTU-TIS dataset without pretraining on SALICON dataset [22], and show the results in Tables XI, XII and XIII. It can be observed that when training directly on the SJTU-TIS dataset without pretraining on SALICON [22], our proposed model achieves significant results and still performs better than other models in terms of all metrics, which validates that the proposed database can independently support training and evaluation. However, the performance of all models has decreased to some extend compared to the results after pretraining on SALICON [22] in Tables VIII, IX and X.

To verify the rationality and effectiveness of the text feature fusion modules in the proposed TGSal model, we conduct ablation experiments from the following two aspects, including w/o global which means removing the global text concatenation part of the global text feature fusion (GTFF) module, and w/o local which means removing the attention module in the local text feature fusion (LTFF) module. Table XIV demonstrates the results of this ablation study, which manifests that these two modules are indispensable. Moreover, the results of w/o local indicate that local text fusion is more crucial compared to the global text fusion in our proposed framework. Moreover, the comparison between TGSal-I and TGSal in Tables IV, V, and VI also demonstrates the significance of introducing text features in the text-guided saliency prediction task.

Since the number of heads in the attention module is a hyperparameter, we also conduct an ablation experiment to study the impact of the number of heads on our TGSal model based on the SALICON database. As shown in Table XV, we compare the results of three different numbers of heads including 2, 4, and 8, respectively. The baseline represents the best performance result among all baseline deep learning models from Table III. We bold the best results in this table. It can be observed that when the number of heads is 2 and 4, some metrics are lower than the baseline model, when the number of heads is set to 8, all metrics are higher than the baseline model and most of them achieve better performance compared to the third line. Since increasing the head number may also add the GPU memory consumption, the head number in our TGSal is set to 8.

Due to the use of a series of operations such as convolution and upsampling in the feature fusion module to fuse multi-scale image information, we also explore the impact of the design of the feature fusion module based on the SALICON database. We attempt three structures, as shown in Fig. 12. Specifically, the first structure uses residual modules instead of the convolution module in the LTFF module and HFR module of our TGSal model. The second structure uses deconvolution structures instead of convolutional layers and upsampling layers. The third structure attempts to replace one convolution with two, and insert an attention module in the middle of the convolutions. The experimental results shown in Table XVI demonstrate that the deconvolution structure has the poorest performance, while the third structure achieves the best in terms of a few metrics. However, the performance of the above three structures in term of SIM is much lower than that of the baseline model. Therefore, the final structure of the LTFF module and HFR module are designed as shown in Fig. 8.

To further validate the effectiveness of the adopted loss function, we compare it with other common loss functions: L1 norm loss and mean square error loss. Fig. 11 shows the results of each metric under the condition of non-salient description. Despite observing that the proposed and used loss function exhibits relatively lower values in terms of the sAUC metric, the performance is higher than other two loss functions in terms of other metrics. Since it is important to consider the performance across a comprehensive set of metrics and our presented loss function achieves better performance compared to the traditional L1 and L2 loss functions in terms of most metrics, we adopt this new loss function in our work. The better performance of L2 loss in terms of the sAUC metric may come from the explicit prediction loss since sAUC is an accuracy metric, but it may not work well on other similarity-based metrics.

The above experiments are all conducted on the condition that the database is split into training/testing sets with the ratio of 4:1. We further add the experiment containing a validation set for comparison. We split the database into training/testing sets with the ratio of 4:1, and into training/validation/testing sets with the ratio of 3:1:1 for each type, respectively, and compare the results. Table XVII demonstrates the comparison results between different database partitions. It can be observed that for the condition that we split the data into three sets, there is a minor performance drop for most cases and metrics, but for a few cases and metrics, the performance may be improved. The performance difference is not large, which manifests the stability of our model. However, the slight performance drop for most cases may be due to the reduction of the training samples.