Holmes-VAD: Towards Unbiased and Explainable
Video Anomaly Detection via Multi-modal LLM

Abstract

Towards open-ended Video Anomaly Detection (VAD), existing methods often exhibit biased detection when faced with challenging or unseen events and lack interpretability. To address these drawbacks, we propose Holmes-VAD, a novel framework that leverages precise temporal supervision and rich multimodal instructions to enable accurate anomaly localization and comprehensive explanations. Firstly, towards unbiased and explainable VAD system, we construct the first large-scale multimodal VAD instruction-tuning benchmark, i.e., VAD-Instruct50k. This dataset is created using a carefully designed semi-automatic labeling paradigm. Efficient single-frame annotations are applied to the collected untrimmed videos, which are then synthesized into high-quality analyses of both abnormal and normal video clips using a robust off-the-shelf video captioner and a large language model (LLM). Building upon the VAD-Instruct50k dataset, we develop a customized solution for interpretable video anomaly detection. We train a lightweight temporal sampler to select frames with high anomaly response and fine-tune a multimodal large language model (LLM) to generate explanatory content. Extensive experimental results validate the generality and interpretability of the proposed Holmes-VAD, establishing it as a novel interpretable technique for real-world video anomaly analysis. To support the community, our benchmark and model will be publicly available at https://holmesvad.github.io/.

1 Introduction

Video Anomaly Detection (VAD) [14] aims to identify abnormal events in videos, which has been extensively researched in recent years due to its considerable application value in public safety [43] and video content understanding [55]. Current VAD approaches can be broadly classified into three categories according to the annotation type of the training data, i.e., unsupervised, weakly-supervised and fully-supervised. Unsupervised methods [14, 35, 30, 12, 49, 60] train solely on normal videos (one-class) or unlabeled normal/abnormal videos, while weakly supervised methods [43, 71, 45, 23, 54, 72, 37] train on normal/abnormal videos with video-level labels. Fully-supervised methods [29, 19] are less studied due to the high cost of precise frame-by-frame annotations. Recently, inspired by the strong representation of multi-modal large language models (MLLMs) pretrained on massive data [46, 7, 16, 59, 26, 74, 28, 69, 62, 9, 4, 51] and their impressive advancements in many downstream visual tasks [13, 52, 53], many efforts [41, 17, 57, 61, 56, 65] start to integrate the multi-modal knowledge into VAD systems, which enables more precise anomaly detection.

Despite significant progress, existing VAD models still face the following primary challenges:

• Biased anomaly space: Due to the lack of reliable frame-level abnormal supervision, unsupervised methods fail to reconstruct or predict unseen normal data, while weakly supervised methods also struggle to select trustworthy snippets for training under the video-level supervision. Consequently, the learned anomaly space of these methods develop a prevalent bias toward unseen or easily-confused normality, remain "when does the anomaly happen" still facing challenges. Although there are some fully supervised studies [29, 19], the number of annotated videos is very small due to the inefficiency of the annotation process, resulting in a lack of scalability.

• Lack of explainability: Existing video anomaly detection approaches do not offer transparent explanations and reasoning for detected anomalies, i.e., "what is the anomaly", and "why is it considered anomalous". This opacity restricts human comprehension and engagement with the system.

Figure 1: Towards unbiased and explainable VAD. In contrast to prevailing VAD approaches (a) that primarily concentrate on identifying anomalies, our method (b) facilitates not only unbiased (i.e., less false alarms toward easily cofused or unseen normality) predictions of anomaly scores but also explanation of detected anomalies, through constructing a large scale VAD dataset with single-frame annotations for untrimmed videos and explanable instruction data for trimmed videos.

Drawing from the above analysis, our insight is that a strong AI-powered anomaly detection system requires not only identifying deviations, but also providing insightful explanations, mirroring the deductive reasoning like the detective Sherlock Holmes. To this end, we present Holmes-VAD, an unbiased and explainable VAD framework based on MLLMs (see Fig.1).

More specifically, to tackle the first issue, we propose a more label-friendly single-frame supervision (one-click for each abnormal event) [38, 20, 67, 8, 21] in the domain of video anomaly detection instead of the prohibitive frame-by-frame annotation. Following this labeling paradigm, we manually make single-frame annotations for the exsiting two largest VAD datasets, e.g., UCF-Crime [43] and XD-Violence [55]. To address the second problem of lacking explainability, we construct a large amount of anomaly-awared instruction conversation data for the finetuning of Multimodal LLM. We leverage the single-frame annotated videos and exsiting off-the-shell large foundation model to build an efficient semi-automated data engine. This data engine can be divided into three main steps: 1) Data Collection: gathering video data, primarily from open-source datasets. 2) Annotation Enhancement: generate reliable video event clips around the single-frame annotated frames and give textual descriptions to them through human effort or foundation models. 3) Instruction Construction: utilizing powerful LLM with open-world knowledge to generate explanable analysis in the context of the enhanced video annotations. Subsequently, the obtained analysis is filted manually and structured into conversational format. After on the above steps, a new benchmark containing single-frame temporal annotations and explanatory text descriptions is constructed, and we name the final obtained dataset as VAD-Intruct50k.

Built upon the proposed VAD-Intruct50k, we develop a customized solution for interpretable video anomaly detection, which has three key components, i.e., Video Encoder, Temporal Sampler and Multi-modal LLM. The Video Encoder and Multi-modal LLM are used to encode the input video and generate text response to the input text prompt, respectively. Additionally, the Temporal Sampler is used to predict the abnormal scores of video frames and sample high-responsive parts as the input for Multi-modal LLM, which is lightweight and enables effient inference. Specially, these three components can be replaced by any other Video-MLLMs or VAD-Networks. Our primary focus is on how to construct a supervised multi-modal dataset to train these components. Extensive experiments demonstrate that our Holmes-VAD achieve outstanding performance in video anomaly detection and can provide detailed explanations for the detected abnormal events.

To summarize, our major contributions are as follows:

• We propose Holmes-VAD, a video anomaly detection system that is capable of identifying anomalies and providing insightful explainations across even hour-long videos.

• To bridge the dataset gap toward an unbiased and explanable VAD system, we introduce VAD-Intruct50k, a large-scale multimodal video anomaly detection datasets, including single-frame annotations for untrimmed videos, and a large amount of instruction conversation data for trimmed abnormal/normal video clips.

• Extensive quantitative and qualitative experiments demonstrate that the proposed Holmes-VAD achieves superior performance and interpretability over recent state-of-the-art methods.

2 Related Works

Video Anomaly Detection. This task aims to temporally detect abnormal frames in a long untrimmed video [2, 39, 24, 33, 49, 14]. The early VAD attempts are based on hand-crafted features [2, 18, 70, 39, 33, 24]. Recently, deep learning approaches [14, 60, 37] have become dominant in Video Anomaly Detection (VAD), broadly classified into unsupervised, weakly-supervised, and fully-supervised methods. Unsupervised methods train only on normal videos to learn normal patterns and are often designed as reconstruction-based [14, 58, 12, 60], prediction-based [30], or a combination [31]. Some methods [64, 44, 47] also explore a fully unsupervised setting, including both unlabeled normal and abnormal videos in the training set. Weakly-supervised methods [43, 71, 10, 55, 45, 23, 54, 72, 37, 68] use both normal and abnormal videos with video-level annotations. Fully-supervised methods [29, 19] are less common due to the high cost of precise frame-level annotations.

Multi-modal Large Language Model. The universal and powerful conversational capabilities of ChatGPT [1] have inspired the entire AI community. This has prompted the emergence of the open-source Large Language Models (LMMs), such as LLaMA [46], Vicuna [7], and Mistral [16], based on autoregressive models [48], they are pretrained and instruction tuned via large amounts of text tokens, thus posses universal and powerful text generation capabilities. Recently, Multi-modal LLMs [59, 26, 74, 28, 27, 69, 62, 9, 4, 51] empower LLMs with visual understanding capabilities. Additionally, MLLMs for videos (e.g., VideoChat [22], Video-LLaMA [66], and Video-LLaVA [25]) pave the way for multi-modal temporal understanding.

Multi-modal Video Anomaly Detection. Large-scale visual-language pretrained models such as CLIP [42] serve as a bridge between visual and textual modalities. Some recent works [41, 17, 57, 61] in the realm of video anomaly detection have leveraged textual information as prompts to enhance the model’s anomaly representation. Based on this,  [56] firstly proposed the open vocabulary VAD task. Furthermore,  [65] extracted captions from video frames using a caption model and designed prompts for LLMs to provide anomaly scores. However, these approaches primarily focus on generating anomaly scores and lack fine-tuning on large-scale domain-specific instruction datasets, resulting in their performance being highly dependent on the base LLMs.

3 VAD-Instruct50k Benchmark

Figure 2: Data engine for the proposed VAD-Instruct50k. We collect numerous abnormal/normal videos from exsiting datasets, following by a series of annotation enhancement including temporal single-frame annotation, event clips generation and event clips captioning. Then we construct the instruction data by prompting the powerful LLM with the enhanced annotation. Throughout the pipeline, manual work and large fundation models coordinated with each other to ensure efficiency and quality in construction.

In this section, we will illustrate the process of VAD-Instruct50k dataset generation. Firstly, the data collection process of VAD-Instruct50k will be presented. Subsequently, we will elaborate on how to enhance the annotations of the collected videos. Finally, the generation process of the instruction conversation data will be introduced. The overall pipeline of the data engine is shown in Fig. 2.

3.1 Data Collection

We first collect videos from the training sets of the two largest weakly-supervised VAD datasets, UCF-Crime [43] and XD-Violence [55], because their video quantity far exceeds that of other existing datasets [50, 32, 34], and their video-level annotations provide a solid foundation for further data processing. After filtering out some low-quality videos via human inspection, we collected a total of 5547 untrimmed videos, include 810/800 abnormal/normal videos from UCF-Crime and 1905/2032 abnormal/normal videos from XD-Violence.

3.2 Annotation Enhancement

The collected videos from UCF-Crime [43] and XD-Violence [55] only offers video-level anomaly labels, which denotes whether the video includes anomalies. Going beyond these coarse annotations, we purify these annotations to enable more discriminative anomaly detection model training.

Temporal single-frame annotation. We adopt an efficient temporal annotation method involving sparse single-frame annotation for the collected abnormal videos, inspired by [40, 38, 67, 20, 21, 68] that use this approach to balance model performance and annotation cost. Specifically, we annotate only one frame for each abnormal event in the video. Through this process, we collect an average of 2.35 single-frame annotations per video.

Event clip generation. Based on the single-frame annotation, we design a reliable pseudo frame-level label generation method and leverage it to train a VAD network $\phi_{s}$ . For each abnormal video with single-frame annotations $\mathcal{G}=\{g_{i}\}^{N_{g}}$ and its anomaly score estimated by the trained VAD network, we generate multiple anomaly event proposals around the annotated frame. Formally, each proposal is represented via a starting and ending timestamp, i.e., $s$ and $e$. For each normal video, we randomly extract several normal event proposals. After this process, we collect all trimmed event clips with anomaly labels: $\mathcal{E}=\{s_{i},e_{i},y_{i}\}^{N_{e}}$, where $y_{i}$ is set to the anomaly class of the video (e.g., Explosion) if the event clip is from an abnormal video, otherwise, it is set to Normal.

Event clip captioning. To fully extract semantic information from the event clips, we utilize a video-based multimodal large language model (MLLM)  [25] to generate detailed captions for each event clip. We also include the SurveillanceVision dataset  [63], which provides manually annotated detailed fine-grained event descriptions for video clips from UCF-Crime  [43]. After combining these resources, we obtain all event clips with corresponding captions $c$ and anomaly labels: $\mathcal{E}=\{s_{i},e_{i},y_{i},c_{i}\}_{i}^{N_{e}}$.

3.3 Instruction-tuning Data Construction

The process of annotation enhancement effectively fills the gap of insufficient information in the original video-level annotation. However, there is still a lack of anomaly-awared explanation for these event clips, i.e., what is the anomaly and why. To address this issue, we utilize the powerful LLM with sufficient open-world knowledge for further instruction dataset construction. Technically, for each event clip in $\mathcal{E}$, we design a task prompts $P_{t}$ combined with the referenceable anomaly context, i.e., the abnormal label $y_{i}$ and the detail caption $c_{i}$. Then we input the combined prompt into the LLM $\mathcal{M}$ to make a judgment on anomalies in the video clip and providing an explanation. The generated response is paired with a corresponding anomaly-awared quesion $P_{d}$, result in an instruction item:

$$\mathcal{I}_{i}=\{\text{"user"}:P_{d},\text{"assistant"}:\mathcal{M}(P_{t},y_{i},c_{i})\}$$ (1)

We use Llama3-Instruct-70B [3] as $\mathcal{M}$ here because of its open-source availability and comparable performance to GPT4. We design multiple $P_{d}$ to ensure the diversity of the instruction data, a typical prompts of $P_{d}$ is: "<video>\n Are there any unexpected or unusual events in the video clip?".

4 Holmes-VAD

Utilizing the proposed VAD-Intruct50k dataset for training, we develop a customised solution for interpretable video anomaly detection, namely Holmes-VAD, which has three key components, Video Encoder, Temporal Sampler and Multi-modal LLM with tunable LoRA [15] modules (see Fig. 3).

Figure 3: Overview of Holmes-VAD. Holmes-VAD takes untrimmed video and user prompt as inputs, and takes the anomaly scores and explanation for detected anomalies outputs. The Temporal Sampler takes class tokens of frames as input and estimates the anomaly scores, and the dense visual tokens are resampled accroding to their anomaly scores before entering the projector.

4.1 Model Architecture

Visual Encoder. We utilize the frozen video encoder in LanguageBind [73] following [25]. It inherits the ViT-L/14 structure from CLIP [42], we refer to it as $\phi_{v}$. Different from the orginal ViT, it models the temporal relationship between frames through additional self-attention layer in the temporal dimension. Give a video frame sequence ${V\in{\mathbb{R}}^{N\times H\times W\times C}}$, the output features of each frame can be denotes as follow:

$$F^{d}_{i}=\{f_{i}^{cls},f_{i}^{1},f_{i}^{2},...,f_{i}^{N_{p}}\}=\phi_{v}(V_{i})\qquad i\in\{1,2,...,N\}$$ (2)

where $f_{i}^{cls}$ indicates the class token feature of $i$-th video frame, $f_{i}^{j}$ ($j\in\{1,2,...,N_{p}\}$) denotes the visual embedding of each patch, and $N_{p}$ reperesents the number of patches of each frame.

Temporal Sampler. Due to the excessive computational burden caused by numerous visual tokens in video, past video-based MLLM approaches [22, 66, 25] have resorted to uniform temporal frame sampling of videos, e.g., 8 frames. This method is clearly unsuitable for long videos in video anomaly detection task, as it increases the probability of ignoring key information.  [65] conduct dense anomaly detection via MLLM in a frame-by-frame mode, which also inevitably leads to a large amount of redundant computation. To address this issue, we first input the dense video frames into the visual encoder, then we introduce the trained VAD network in 3.2 here, which receives the cls token of the video frames $f_{1}^{cls},f_{2}^{cls},...,f_{N}^{cls}$ and outputs anomaly scores $s_{1},s_{2},...,s_{N}$:

$$\{s_{1},s_{2},...,s_{N}\}=\phi_{s}(\{f_{1}^{cls},f_{2}^{cls},...,f_{N}^{cls}\})$$ (3)

where $\phi_{s}$ denotes the trained VAD network.

Then, we sample the video tokens according to the anomaly scores. Specifically, only the tokens $f_{k}$ from frames with corresponding anomaly score $s_{k}$ above a set threshold $\theta$ are then fed into the subsequent network:

$$F^{s}=\{f_{k}\in F^{d}\mid s_{k}>\theta\}$$ (4)

where $F^{s}$ denotes the sampled sparse visual tokens from the original dense visual tokens $F^{d}$. In this way, the model can generate anomaly-awared response to long untrimmed video.

Projector and LLM. To enable the LLM to understand the features output by the visual encoder, a projector ${\phi_{proj}}$ composed of two layers of MLPs is designed between them, after this, the feature dimention is aligned with the input dimension of LLM. We utilize Vicuna [7] as our LLM following [25].

$$T_{i+1}=LLM([\phi_{proj}(F^{s}),\phi_{T}(T_{0:i})])$$ (5)

where $T_{0:i}$ represents the input text tokens to LLM and $T_{i+1}$ indicates the predicted next token. $\phi_{proj}$ and $\phi_{T}$ represents the Projector and the Text Encoder, respectively. $[\cdot,\cdot]$ denotes concatenation.

4.2 Training

Training of the Temporal Sampler. In this stage, we only train the Temporal Sampler under the single-frame supervision. In essence, we employed a pseudo-labeling supervision strategy. The pseudo-labels are initialized through single-frame annotations during the training process and are online updated around the annotated frames. We use the generated pseudo label to supervise the predicted anomaly score, which can effectively reduce the bias of the temporal sampler towards easily confued normality.

Instruction Tuning. During this stage, we take the trimmed event clips as input and do not perform Temporal Sampler because each clip has been labeled as abnormal or normal. In this stage, we train the projector and use LoRA [15] to fine-tune the Multi-modal LLM. We conduct different tuning strategy and compare them in the next section. Given the projected visual features $F_{v}$ and the textual input embedding $F_{t}$, the LLM decode them into a sequence words $\mathcal{A}$. we follow mainstream works to use the original auto-regressive training objective. The objective aims to maximize the likelihood of generating the ground truth answer sequence given the input features, encouraging the model to produce coherent and accurate responses based on the input features.

5 Experiments

In this section, we conduct extensive experiments to thoroughly demonstrate the capabilities of our proposed model, i.e., Holmes-VAD.

5.1 Experiment Setup

Datasets. We conduct the comparative experiments on two standard VAD datasets, namely, UCF-Crime [43] and XD-Violence [55]. (1) UCF-Crime [43] comprises 1900 untrimmed videos totaling 128 hours from outdoor and indoor surveillance cameras. It encompasses 13 classes of real-world anomalies, including Abuse, Explosion, Fighting, and Shooting. In the weakly-supervised setting, there are 1610/290 videos for training/testing, with the training set consisting of 810 abnormal videos and 800 normal videos, respectively. (2) XD-Violence [55] is the largest VAD benchmark, comprising 4754 videos totaling 217 hours sourced from surveillance, movies, car cameras, and games. It encompasses 6 anomaly classes: Abuse, Car Accidents, Explosions, Fighting, Riots, and Shooting. The training/testing video count stands at 3954/800, adhering to a weakly-supervised framework. The training set comprises 1905 abnormal videos and 2049 normal videos, respectively.

Metrics. To evaluate the anomaly detection performance of the temporal sampler, we use the Area Under the Curve (AUC) as the main evaluation metric for UCF-Crime following [45, 54, 23, 72, 68]. Meanwhile, the AUC of the frame-level precision-recall curve (AP) is utilized for XD-Violence. To evaluate the quality of explanation response, we randomly extract 86 abnormal/normal video segments from the test videos of UCF-Crime and XD-Violence, and then invite 10 users to vote on the responses of different models from 3 aspects include Judgement Accuracy (JA), Content Perception (CP) and Anomaly Explanatory (AE). Please see the Appendix for more details about the metrics.

Implementation details. In our study, we take the ViT in LanguageBind model [73] as the Video Encoder and initialize the Multi-modal LLM with Video-LLaVA [25]. UR-DMU [72] serves as the foundation structure for our Temporal Sampler. To optimize the Temporal Sampler, we ramdomly sample one frame at 16-frame intervals, and Adam optimizer with a learning rate of 1e-4 is adopted. Note that when evaluating performance on XD-Violence and UCF-Crime, only videos in the corresponding training sets are used to train our model for fair comparisons. For instruction tuning, we train with a batch size of 128 for 1 epoch, using the AdamW optimizer with cosine learning rate decay and a warm-up period, setting the projector’s learning rate to 2e-5. The LoRA [15] parameters are set as: $r$=64, $\alpha$=128, and learning rate=2e-4. The abnormal threshold $\theta$ is set to 0.8 during inference. Experiments are conducted on 2 NVIDIA A100 GPUs.

5.2 Main Results

Table 1: Comparision with state-of-the-art Video Anomaly Detection approches. We include semi-supervised (Semi.) methods, unsupervised (Un.) methods, weakly-supervised (W.) methods and some other methods. “$\ast$” represents the result reported in  [65].

Methods	Backbone	Supervision	Explanation	XD-Violence	UCF-Crime
				AP/%	AUC/%
Non-explainable VAD
Conv-AE [14]	-	Semi.	✗	27.25	50.60
GODS [49]	I3D	Semi.	✗	N/A	70.46
GCL [64]	ResNext	Un.	✗	N/A	71.04
DYANNET [44]	I3D	Un.	✗	N/A	84.50
MIST [10]	I3D	W.	✗	N/A	82.30
Wu et al. [55]	I3D	W.	✗	78.64	82.44
RTFM [45]	I3D	W.	✗	77.81	84.30
MSL [23]	I3D	W.	✗	78.28	85.30
S3R [54]	I3D	W.	✗	80.26	85.99
MGFN [6]	I3D	W.	✗	79.19	86.98
UR-DMU [72]	I3D	W.	✗	81.66	86.97
CLIP-TSA [17]	ViT	W.	✗	82.19	87.58
VadCLIP [57]	ViT	W.	✗	84.51	88.02
Yang et al. [61]	ViT	W.	✗	83.68	87.79
Wu et al. [56]	ViT	Open-Vocabulary	✗	66.53	86.40
Explainable Multi-modal VAD
ZS CLIP [42]∗	ViT	Training-Free	✓	17.83	53.16
ZS IMAGEBIND [11]∗	ViT	Training-Free	✓	25.36	55.78
LLAVA-1.5 [27]∗	ViT	Training-Free	✓	50.26	72.84
LAVAD [65]	ViT	Training-Free	✓	62.01	80.28
Holmes-VAD (Ours)	ViT	Instruction-Tuned	✓	90.67	89.51

We compare our method with state-of-the-art methods, including semi-supervised methods [14, 49], unsupervised methods [64, 44], weakly-supervised methods [45, 23, 54, 72, 17, 57] and recently training-free method [65]. We have indicated their backbones, supervision methods, and performance on the UCF-Crime and XD-Violence datasets, as shown in Table 1. Our method has an AP of 90.67% on XD-Violence and an AUC of 89.51% on UCF-Crime, significantly outperforming the prior state-of-the-art methods, which demonstrates that our method can generate less biased anomaly scores. It is worth noting that while achieving precise localization of anomalies, Holmes-VAD is also capable of providing explanations and analysis for the detected anomalies by the model, a feature unavailable in existing non-explainable VAD methods. Although LAVAD [65] has explainability, the training-free large language model lacks an understanding of anomaly knowledge due to the limitation of insufficient supervised data.

5.3 Analytic Results

Table 2: Human evaluation on models under different training settings.

Training Strategy	Average Words of Model Reponse	JA(%)	CP(%)	AE(%)
Training-free	38.29	65.1	11.6	15.9
Projector	40.84	81.4	27.2	32.2
Projector+LoRA (Default)	46.13	86.0	61.2	51.9

Table 3: Ablation study of backbone and supervision in Temporal Sampler.

Backbone	Single-frame	XD-AP(%)	UCF-AUC(%)
I3D [5]	✗	82.40	86.54
ViT [73]	✗	84.96	84.61
I3D [5]	✓	89.40	90.80
ViT [73]	✓	90.67	89.51

Table 4: Effect of random temporal shifting of single-frame annotations.

Shifted Timestamp	XD-AP(%)	UCF-AUC(%)
0	90.67	89.51
10	90.55	89.45
50	90.45	89.32
100	90.12	88.95

Table 5: Temporal Sampler v.s. Uniform Sampler. We averaged the inference time of all test videos.

Sampling Strategy	XD-AP(%)	UCF-AUC(%)	Avg. Infer Speed (second per video)
Uniform Sampler	67.25	78.38	32.82
Temporal Sampler (Default)	90.67	89.52	4.24

Influence of varied training strategies on anomaly explanation. We conduct a user study to evaluate three different training strategies over 86 test samples and 10 volunteers: a) Training-free: no fine-tuning; b) Projector: fine-tuning on VAD-Instruct50k, only training the projector while kee** the Multi-modal LLM fixed; c) Projector+LoRA: fine-tuning on VAD-Instruct50k, training the projector and using LoRA [15] to fine-tune the Multi-modal LLM. As shown in Table 5.3, Projector+LoRA provide the most detailed response (46.13 words in average) and reaches the highest Judgement Accuracy (86.0%). Addtionally, it also achieves the highest voting rate, including 61.2% on Content Perception and 51.9% on Anomaly Explanatory, these demonstrate better interpretability by fine-tuning Multi-modal LLM on VAD-Instruct50k.

Backbone and supervision matters in Temporal Sampler. In Table 5.3, we ablate the impact of video backbone and the supervision for Temporal Sampler. We use UR-DMU [72] as our baseline method. The results indicate that on XD-Violence dataset, LanguageBind [73] as a backbone outperforms I3D [5] significantly, whereas the opposite is observed on UCF-Crime. Additionally, single-frame supervision significantly enhances performance regardless of the backbone used, demonstrating the effectiveness of point supervision in improving anomaly localization capabilities.

Influence of perturbed single-frame annotations. To assess the robustness of our method to the perturbed temporal position of single-frame annotation, we introduce varied temporal timestamp shifts to the original positions of the annotated frames. As shown in Table 5.3, there is no significant performance degradation of our model under perturbed annotation positions, indicating that our method possesses a notable tolerance towards variations in degraded supervision.

Temporal Sampler v.s. Uniform Sampler. We replace the Temporal Sampler with Uniform Sampler while maintaining the frame rate. The video is then divided into non-overlap** clips, which are sequentially fed into the Multimodal LLM to output results. If the output is "Yes" the anomaly scores of all frames in the input segment are set to 1, otherwise, they are set to 0. Finally, we compare the detection performance and inference efficiency in Table 5.3. The results demonstrate that the Temporal Sampler ensures higher inference efficiency while maintaining accurate detection results.

Figure 4: Qualitative results. We compare our interpretability results with Video-LLaVA [36] (without instruction tuning). Correct and wrong explanations are in green and red, respectively.

Qualitative comparision. To provide a more intuitive understanding of the capabilities of MLLM in explaining complex anomalies, we provide qualitative comparisons between Holmes-VAD and Video-LLaVA in Fig. 4. The results demonstrate that Holmes-VAD can accurately identify anomalies in videos and provide specific explanations for conflicts in sports competitions, explosions, and accidents captured by car cameras (Abnormal Cases). Even for normal videos, Holmes-VAD exhibits robust analytical abilities, correcting erroneous responses from the Temporal Sampler (Normal Cases). These findings highlight the effectiveness and advantage of Holmes-VAD in perceiving video events and analyzing anomalies.

6 Conclusion

In this paper, we introduce a video anomaly detection system called Holmes-VAD to address the biases and lack of interpretability in existing anomaly detection methods. By introducing a more efficient labeling paradigm and constructing a large-scale multimodal video anomaly detection dataset, VAD-Instruct50k, we validated the generality and interpretability of Holmes-VAD. Through extensive experiments, we positioned Holmes-VAD as a valuable tool for real-world applications.

Limitation and Future work. Despite the human effort for filtering the noise instruction data during constructing the VAD-Instruct50k dataset, the reliance on off-the-shelf video captioning models for generating video description may not always capture the nuances and context-specific information. This is a trade-off we have made between labeling costs and efficiency, we believe that the quality of data is no less important than the quantity of data, and we plan to further enhance data quality and quantity within acceptable labor costs in the future. Furthermore, although we control the length of the video input to the Multi-modal LLM through Temporal Sampler and accurately analyze abnormal content in the trimmed video clips, there is still a lack of an effective solution for Multimodal LLM to understand long-term video anomalies without compromising its image-level perceptual capabilities. We leave these for our future exploration.

Acknowledgement This work is supported by the National Natural Science Foundation of China under grant U22B2053 and 623B2039.

Appendix A Appendix

A.1 Broader Impact

The paper proposes a video anomaly detection framework, namely Holmes-VAD, that is capable of temporally identifying anomalies accurately and providing insightful explainations across even hour-long videos. Additionally, this paper provides VAD-Intruct50k, a large scale multimodal video anomaly detection datasets, including single-frame annotations for untrimmed videos, and a large amount of instruction conversation data for trimmed abnormal/normal video clips.

The positive societal impacts of the work include:

• Improved public safety: The development of more accurate and interpretable video anomaly detection systems can enhance public safety by enabling quicker and more precise identification of anomalies in surveillance videos, such as criminal activities or accidents.

• Advancement in supervised and open-world VAD research: The proposed VAD-Intruct50k dataset provide a). accurate temporal timestamp of the abnormal events in videos, and b). video-explanation pair for both abnormal and normal video clips, which can pave the way for further supervised and open-world research in the video anomaly detection area.

The negative societal impacts may include:

• Privacy concerns: The use of video surveillance technology, especially in public spaces, raises concerns about privacy and the potential for intrusive monitoring of individuals without their consent.

• Disregard for minor anomalies: Despite efforts to reduce bias in anomaly detection, there is still a risk of disregard for subtle anomalies such as stealing in the supermarket, leading to potential undetected anomalies.

Consequently, researchers should adhere to relevent laws and regulations, and strive to avoid using our model or dataset for any improper invasion of privacy. Meanwhile, all our model and data will be only used for research purpose to avoid the potential negative societal impacts.

A.2 Process of single-frame annotation.

Figure 5: Screenshot of the single-frame annotation interface.

Annotation tool. We develop an interface designed specifically for single-frame annotation in videos, as shown in Fig. 5. This interface makes it easier to navigate through video lists, adjust video progress rapidly, and automatically record timestamps for annotating individual frames. Furthermore, it enables the preview of annotated frames. By clicking on the annotated frame ID, the video progress automatically synchronizes with the corresponding temporal position. These features greatly streamline the annotation process, enhancing convenience and efficiency. If annotators come across any errors or need to make adjustments, they can delete incorrect annotations and proceed with re-annotation.

Quality control. We initially divide the entire dataset into various portions and distribute them among different annotators for labeling. Once the first round of annotations is completed, we proceed with a secondary review of the video annotations to eliminate incorrect or redundant annotations. In addition, we include ignored clicks to minimize the possibility of overlooking potential anomalies. This process ensures the Reliability and Comprehensiveness of the single-frame annotations.

Examples of single-frame annotation. To facilitate a better understanding of the annotation process, we offer several examples of annotated videos in Fig. 6.

Figure 6: Examples of single-frame annotation.

A.3 Model architecture and training details of the Temporal Sampler.

Figure 7: Architecture of the Temporal Sampler.

Model architecture. We use UR-DMU [72] as the VAD network in our Temporal Sampler. As shown in Fig. 7, UR-DMU utilizes a Global and Local Multi-Head Self Attention (GL-MHSA) module to capture both long-range and short-range temporal relationships among video snippets. Furthermore, UR-DMU introduces two memory banks to store and differentiate abnormal and normal prototypes, thereby maximizing the margins between these two representations. In order to learn discriminative representations, UR-DMU employs triplet loss to increase the feature distance after interacting with different memories. Simultaneously, it utilizes KL loss to constrain the normal memory to follow a Gaussian distribution, accounting for the variance introduced by noise. Thus, the base loss function for the UR-DMU baseline is defined as follows:

$${\mathcal{L}}_{base}={\mathcal{L}}_{mil}+{\mathcal{L}}_{mag}+{\mathcal{L}}_{triplet}+{\mathcal{L}}_{kl}$$ (6)

Training details. During the training stage of the Temporal Sampler, we leverage the sparse single-frame annotations to generate reliable dense snippet-level pseudo label. As illustrated in Alg. 1, we employ a dynamic threshold and perform local bidirectional mining based on the single-frame annotations. Snippets with anomaly scores exceeding a specific proportion of the annotated snippet’s score are identified as pseudo anomaly snippets. We set $\alpha=0.9$ in our implementation.

Algorithm 1 Pseudo Label Mining.

Input: Anomaly score $\mathbf{S}\in{\mathbb{R}}^{T}$, single-frame annotations $\mathcal{G}=\{g_{i}\}^{N_{g}}$, anomaly ratio $\alpha$.
Output: Pseudo anomaly snippets $\mathcal{T}^{a}=\{t_{i}\}_{i}^{N_{a}}$.

1:  Let $\mathcal{T}^{a}\leftarrow\varnothing$.

2:  for every $g_{i}\in\mathbf{G}$ do

3:     for $t=g_{i}$ to $g_{i-1}$ do

4:        if $\mathbf{S}[t]>\alpha\cdot S[g_{i}]$, then $\mathcal{T}^{a}\leftarrow t\cup\mathcal{T}^{a}$, else break

5:        end if

6:     end for

7:     for $t=g_{i}$ to $g_{i+1}$ do

8:        if $\mathbf{S}[t]>\alpha\cdot S[g_{i}]$, then $\mathcal{T}^{a}\leftarrow t\cup\mathcal{T}^{a}$, else break

9:        end if

10:     end for

11:  end for

12:  Return $\mathcal{T}^{a}$

After mining the pseudo anomaly snippets, we adopt Gaussian function to smooth the binary pseudo label:

$$\hat{S}(\mathbf{t})=\textbf{norm}(\sum_{i=1}^{N_{a}}{exp(-\frac{\|\mathbf{t}-\mathbf{t_{i}}\|^{2}}{2r^{2}})})$$ (7)

where $r=0.1$ indicates the smoothing ratio. We use the generated dense and smooth pseudo label to supervise the predicted anomaly score:

$${\mathcal{L}}_{abn}=BCE(S,\hat{S})$$ (8)

where $S$ and $\hat{S}$ denote the predicted anomaly score and the generated pseudo frame-level label, respectively.

A.4 Details of human evaluation.

Figure 8: Screenshot of the human evaluation interface.

Figure 9: Qualitative comparision in human evaluation. We show the results of Judgement Accuracy (JA), Content Perception (CP) and Anomaly Explanatory (AE) above the answer box of each model.

To evaluate the quality of explanation response, we randomly extract 86 abnormal/normal video segments from the test videos of UCF-Crime and XD-Violence, and then invite 10 users to vote on the responses of different models from 3 aspects include Judgement Accuracy (JA), Content Perception (CP) and Anomaly Explanatory (AE).

• Judgement Accuracy (JA): Determine whether the model’s judgment on anomalies is correct, we extract predictions by matching "Yes"/"No" in the answers, and compare them with the ground truth labels (abnormal/normal). Finally, we calculate the accuracy of the judgments.

• Content Perception (CP): The accuracy and clarity of the model’s descriptions of the content, characters, and events in the video scenes, as well as any potential hallucination issues (descriptions of non-existent objects in the video or responses unrelated to the questions).

• Anomaly Explanatory (AE): The model’s ability to analyze and interpret abnormal/normal events in the video.

We provide the screenshot of the human evaluation interface in Fig. 8, to ensure a fair selection, the names of the models are not visible to the users, and choices can only be made from anonymous options. In Fig. 9, we provide several test examples, with the results of the Judgement Accuracy (JA), Content Perception (CP) and Anomaly Explanatory (AE).

A.5 Dats statistical analysis of VAD-Instruct50k

In Table 6, we conduct a statistical analysis of our proposed VAD-Instruct50k and compare it with representative datasets in the VAD field, which shows the significant volume and excellent diversity of our constructed instruction dataset.

Table 6: Datasets Statistics.

Dataset	#Videos	Annotation Type	#Queries	Avg word
CHUK Avenue	37	None	N/A	N/A
ShanghaiTech	437	None/video label	N/A	N/A
UCF-Crime [43]	1,610	video label	N/A	N/A
XD-Violence [55]	4,754	video label	N/A	N/A
UCA [63]	1,854	segment caption	23,542	20.15
VAD-Instruct50k (Ours)	5,547	single-frame&segment instruction	51,567	44.83