VideoPatchCore: An Effective Method to Memorize Normality for Video Anomaly Detection
VideoPatchCore：一种用于视频异常检测的正常性的有效方法

The reconstruction-based approach is based on the assumption that a model trained to reconstruct normal video frames fails to accurately reconstruct abnormal frames. Nguyen [25] proposed a model comprising one encoder and two decoders. This model considered the $t^{th}$ frame as the input and predicted both the reconstructed frame and the optical flow between the $t^{th}$ and $t+1^{th}$ frames. Zaheer [38] used GANs to generate high-quality reconstructed frames. In this approach, the discriminator was trained to classify high-quality frames as normal and low-quality frames as abnormal, utilizing a generator in its previous state that produced low-quality frames.
基于重建的方法基于这样一个假设，即训练用于重建正常视频帧的模型无法准确重建异常帧。Nguyen [25] 提出了一个由一个编码器和两个解码器组成的模型。该模型将 $t^{th}$ 帧视为输入，并预测重建的帧和 $t^{th}$ 和 $t+1^{th}$ 帧之间的光流。Zaheer [38] 使用 GAN 生成高质量的重建帧。在这种方法中，判别器被训练为将高质量帧分类为正常帧，将低质量帧分类为异常帧，利用生成低质量帧的先前状态的生成器。

The prediction-based approach is based on the assumption that when a model is trained to predict future or past frames of normal video sequences, it fails to accurately predict abnormal frames. Liu [18] utilized FlowNet and GANs to predict the $t+1^{th}$ frame from $t$ frames. Yang [36] suggested selecting key frames from $t$ frames and predicting all $t$ frames based only on these key frames.
基于预测的方法基于以下假设：当模型经过训练以预测正常视频序列的未来或过去帧时，它无法准确预测异常帧。Liu [18] 利用 FlowNet 和 GAN 从帧中 $t$ 预测 $t+1^{th}$ 帧。Yang [36] 建议从 $t$ 帧中选择关键帧，并仅根据这些关键帧预测所有 $t$ 帧。

In VAD, memory is utilized to address the issue of accurately generating abnormal frames in OCC method. Gong [8] proposes a method that stores the features of normal frames in memory and generates normal frames using these stored memory items. Park [26] suggests a method for learning various patterns of normal data by clustering similar features and using the cluster centers as memory items, thus storing numerous normal features in a compact memory. Sun [32] employs contrastive learning to cluster features based on their classes and stores each feature in memory. This technique allows for the creation of a compact memory by leveraging the semantic information of objects.
在 VAD 中，利用内存来解决 OCC 方法中准确生成异常帧的问题。Gong [8] 提出了一种方法，将普通帧的特征存储在内存中，并使用这些存储的内存项生成普通帧。Park [26] 提出了一种学习法线数据各种模式的方法，该方法通过将相似特征聚类并使用簇中心作为内存项，从而在紧凑的内存中存储大量法线特征。Sun [32] 采用对比学习根据特征的类对特征进行聚类，并将每个特征存储在内存中。该技术允许通过利用对象的语义信息来创建紧凑的内存。

In the image domain, representation-based approaches have gained attention as effective methods for anomaly detection. This approach utilizes pre-trained networks on large-scale datasets such as ImageNet[7] and compares the embeddings of normal and abnormal images to detect anomalies. Cohen[6] proposes feature pyramid matching, leveraging the different information at each layer of the network. Features from the training data are stored in memory, and anomaly scores for the features of test data are computed using k-NN with respect to the features stored in memory. Roth[30] points out that high-level features of pre-trained networks are specialized for classification purposes, indicating the need to extract features at the middle level. These features are stored in memory, and memory optimization is achieved through Coreset Subsampling.
在图像领域，基于表示的方法作为异常检测的有效方法而受到关注。这种方法利用大规模数据集（如 ImageNet）上的预训练网络[7]，并比较正常和异常图像的嵌入以检测异常。Cohen[6] 提出了特征金字塔匹配，利用网络每一层的不同信息。训练数据中的特征存储在内存中，测试数据特征的异常分数使用 k-NN 相对于内存中存储的特征计算。Roth[30] 指出，预训练网络的高级特征专门用于分类目的，这表明需要在中间阶段提取特征。这些功能存储在内存中，内存优化是通过 Coreset Subssampling 实现的。

Fig. 2 depicts the proposed VPC model, consisting of two streams: the local stream at the object level and the global stream at the frame level. Each stream extracts features through a vision encoder from CLIP, pre-trained on large-scale datasets, and these features are segmented according to the type of memory and accessed in the memory bank. During this process, the memorization and inference stages are performed. In the memorization stage, features are stored in memory, which is then optimized using greedy coreset subsampling from PatchCore. In the inference stage, anomaly scores are derived by calculating distances between memory and features. The following provides a detailed description of each component of the proposed VPC.
无花果。 图 2 描述了建议的 VPC 模型，该模型由两个流组成：对象级别的本地流和帧级别的全局流。每个流都通过 CLIP 的视觉编码器提取特征，在大规模数据集上进行预训练，这些特征根据内存类型进行分割，并在内存库中访问。在此过程中，将执行记忆和推理阶段。在记忆阶段，特征存储在内存中，然后使用 PatchCore 的贪婪核心集子采样进行优化。在推理阶段，异常分数是通过计算内存和特征之间的距离来得出的。下面提供了建议的 VPC 的每个组件的详细说明。

Vision Encoder. We adopt a CNN-based CLIP model as the Encoder and utilize layers 2, 3 $\varphi_{2}$, $\varphi_{3}$ similar to PatchCore. Typically, the shallow layers of the network capture the local features, while deeper layers capture the global features. However, the first layer captures overly detailed features such as edges and corners, and the last layer captures features biased towards classification, which are not helpful for anomaly detection. Thus, we process and utilize features from the intermediate layers $\varphi_{2}$, $\varphi_{3}$.
Vision Encoder （视觉编码器）。我们采用基于 CNN 的 CLIP 模型作为编码器，并使用第 2、3 层 $\varphi_{2}$ ， $\varphi_{3}$ 类似于 PatchCore。通常，网络的浅层捕获局部特征，而较深层捕获全局特征。但是，第一层捕获过于详细的特征（如边缘和角落），最后一层捕获偏向于分类的特征，这对异常检测没有帮助。因此，我们处理和利用中间层 $\varphi_{2}$ 的特征 。 $\varphi_{3}$

Local Stream. The local stream is composed of YOLOv5[15] for object detection, a vision encoder $\varphi$ for feature extraction, and spatial memory (SM) and temporal memory (TM) for storing the object appearance and motion information. Given a sequence of $d$ consecutive frames $F_{1},F_{2},\ldots,F_{d}\in\mathbb{R}^{3\times H\times W}$, YOLOv5 generates a set of $n$ objects for each frame, $O_{1},O_{2},\ldots,O_{d}\in\mathbb{R}^{n\times 3\times H\times W}$. Subsequently, the locally-aware features $l_{1},l_{2},\ldots,l_{d}\in\mathbb{R}^{n\times c\times h\times w}$are produced that represent the features of the objects.
本地流。本地流由用于对象检测的 YOLOv5[15]、 $\varphi$ 用于特征提取的视觉编码器以及用于存储对象外观和运动信息的空间内存 （SM） 和时间内存 （TM） 组成。给定一个连续帧 $F_{1},F_{2},\ldots,F_{d}\in\mathbb{R}^{3\times H\times W}$ 序列 $d$ ，YOLOv5 会为每个帧生成一组 $n$ 对象。 $O_{1},O_{2},\ldots,O_{d}\in\mathbb{R}^{n\times 3\times H\times W}$ 随后，将生成表示对象特征的本地感知特征 $l_{1},l_{2},\ldots,l_{d}\in\mathbb{R}^{n\times c\times h\times w}$ 。

Specifically, average pooling is performed on the features from $\varphi_{2},\varphi_{3}$ and they are concatenated based on $\varphi_{2}$. This strategy aims to integrate both fine-grained and coarse-grained information that is necessary for object-level VAD. However, as $\varphi_{2},\varphi_{3}$ have relatively small receptive fields, pooling is used to consolidate information over a wider range. The equation for constructing $l_{i}$ is as follows: \linenomathAMS
具体来说，对 中的 $\varphi_{2},\varphi_{3}$ 特征执行平均池化，并根据 进行连接 $\varphi_{2}$ 。此策略旨在集成对象级 VAD 所需的细粒度和粗粒度信息。然而，与相对较小的感受野一样 $\varphi_{2},\varphi_{3}$ ，池化用于在更宽的范围内整合信息。构造 $l_{i}$ 方程如下：\linenomathAMS

where $f_{\text{ap}}$ represents the average pooling and $\langle\cdot,\cdot\rangle$ denotes tensor concatenation.
其中 $f_{\text{ap}}$ 表示平均池化， $\langle\cdot,\cdot\rangle$ 表示张量串联。

Subsequently, $l_{1:d}$ are reshaped into $lf\in\mathbb{R}^{n\times c\times d\times h\times w}$. However, the dimensionality of $lf$ is impractically large for video processing. Therefore, we employ a method called split pooling to reduce the number of channels, transforming it into $\mathbb{R}^{n\times c^{\prime}\times d\times h\times w}$ ($c\gg c^{\prime}$). The details of this approach can be found in Section 4.6. Subsequently, $lf$ is split into spatial patches and temporal patches through spatial partition and temporal partition, respectively, and stored in SM and TM.
随后， $l_{1:d}$ 被重塑为 $lf\in\mathbb{R}^{n\times c\times d\times h\times w}$ 。但是，对于视频处理来说，维度 $lf$ 不切实际地大。因此，我们采用一种称为 split pooling 的方法来减少通道的数量，将其转换为 $\mathbb{R}^{n\times c^{\prime}\times d\times h\times w}$ （） $c\gg c^{\prime}$ 。此方法的详细信息可以在 Section 4.6 中找到。随后， $lf$ 分别通过空间分区和时间分区拆分为空间补丁和时间补丁，并存储在 SM 和 TM 中。

Global Stream. The global stream constructs globally-aware features $g_{1},g_{2},\\ \ldots,g_{d}\in\mathbb{R}^{c\times 1\times 1}$ to capture the overall context of frames, thereby differing from the local stream, which focuses on individual objects within frames. We utilize $\varphi_{2},\varphi_{3}$, applying global pooling to obtain global information. Specifically, average pooling is applied to the features of each layer, and these are concatenated based on $\varphi_{3}$, which handles more global information. During global pooling, both average and max pooling techniques are employed to achieve a better representation. The equation for constructing $g_{i}$ is as follows: \linenomathAMS
全局流。全局流构造全局感知特征 $g_{1},g_{2},\\ \ldots,g_{d}\in\mathbb{R}^{c\times 1\times 1}$ 来捕获帧的整体上下文，因此与本地流不同，本地流侧重于帧内的单个对象。我们利用 $\varphi_{2},\varphi_{3}$ ，应用全局池来获取全局信息。具体来说，平均池化应用于每个图层的特征，这些特征基于 $\varphi_{3}$ 连接，从而处理更多的全局信息。在全局池化期间，同时使用平均池化和最大池化技术来实现更好的表示。构造 $g_{i}$ 方程如下：\linenomathAMS

where $f_{\text{gap}}$ and $f_{\text{gmp}}$ denote global average pooling and global max pooling, respectively.
其中 $f_{\text{gap}}$ 和 $f_{\text{gmp}}$ 分别表示 Global Average pooling 和 Global Max pooling。

Subsequently, $g_{1:d}$ are reshaped into $gf\in\mathbb{R}^{c\times d\times 1\times 1}$. The $gf$ constructed in this manner includes the global information of each frame and is converted into high-level patches through high-level partition. Afterwards, the patch is stored in the High-level Semantic Memory (HSM).
随后， $g_{1:d}$ 被重塑为 $gf\in\mathbb{R}^{c\times d\times 1\times 1}$ 。以这种方式构建的 S $gf$ 包含了每帧的全局信息，并通过 high-level partitions 转换为 high-level patch。之后，补丁将存储在高级语义内存 （HSM） 中。

Fig. 3 illustrates the proposed patch partition method, which consists of spatial partition, temporal partition, and high-level partition. The spatial partition focuses on object appearance information, generating patches while disregarding temporal information. The temporal partition emphasizes object motion information, generating patches while ignoring spatial information. The high-level partition utilizes extensive spatiotemporal information to generate patches for understanding the contextual information across frames. The detailed descriptions of each partition are as follows.
无花果。 图 3 说明了所提出的 patch 分区方法，它由空间分区、时间分区和高级分区组成。空间分区侧重于对象外观信息，在生成补丁的同时忽略时态信息。时间分区强调对象运动信息，生成面片，同时忽略空间信息。高级分区利用广泛的时空信息来生成补丁，以便了解跨帧的上下文信息。每个分区的详细说明如下。

Spatial Partition. Appearance information is crucial for evaluating anomalies in objects and can be derived from spatial features. We compress the input feature map $lf$ using temporal global pooling[16] to preserve spatial information while ignoring temporal aspects. Subsequently, average pooling is applied to consider various regions of the object. For instance, when assessing a person’s pose, it is essential to consider multiple joints together. The equation for creating $SpatialPatches$ is as follows: \linenomathAMS
空间分区。外观信息对于评估对象中的异常至关重要，并且可以从空间特征中获取。我们使用时间全局池化[16] 压缩输入特征图 $lf$ ，以保留空间信息，同时忽略时间方面。随后，应用平均池化来考虑对象的各个区域。例如，在评估一个人的姿势时，必须同时考虑多个关节。创建 $SpatialPatches$ 方程如下：\linenomathAMS

where $f_{\text{tgp}}$ denotes the temporal global pooling. Finally, the results are reshaped into $\mathbb{R}^{(n\cdot\hat{h}\cdot\hat{w})\times c^{\prime}}$, where $h\geq\hat{h}$ and $w\geq\hat{w}$.
其中 $f_{\text{tgp}}$ 表示临时全局池化。最后，结果被重塑为 $\mathbb{R}^{(n\cdot\hat{h}\cdot\hat{w})\times c^{\prime}}$ ，其中 $h\geq\hat{h}$ 和 $w\geq\hat{w}$ 。

Temporal Partition. Motion information in objects represents changes over time, making it crucial for VAD. We utilize the adjacent temporal information within $lf$ to generate motion features $mf$. To achieve this, the feature differences are computed and the representative motion values are determined through global average pooling. The equation for creating $TemporalPatches$ is as follows: \linenomathAMS
Temporal Partition 的 Temporal Partition。对象中的运动信息表示随时间的变化，因此对于 VAD 至关重要。我们利用其中 $lf$ 的相邻时间信息来生成运动特征 $mf$ 。为了实现这一点，计算了特征差异，并通过全局平均池化确定了代表性运动值。创建 $TemporalPatches$ 方程如下：\linenomathAMS

$mf_{(t)}\in\mathbb{R}^{n\times c^{\prime}\times h\times w}$ represents the difference between the $t^{th}$ and $t+1^{th}$ time within $lf$, and it belongs to $mf\in\mathbb{R}^{n\times c^{\prime}\times\hat{d}\times h\times w}$, where $\hat{d}=d-1$. Finally, the results are reshaped into $\mathbb{R}^{(n\cdot\hat{d})\times c^{\prime}}$.
$mf_{(t)}\in\mathbb{R}^{n\times c^{\prime}\times h\times w}$ 表示 $t^{th}$ 和 $t+1^{th}$ 时间在 $lf$ 内 之间的差异，它属于 $mf\in\mathbb{R}^{n\times c^{\prime}\times\hat{d}\times h\times w}$ ，其中 $\hat{d}=d-1$ 。最后，将结果重塑为 $\mathbb{R}^{(n\cdot\hat{d})\times c^{\prime}}$ 。

High-level Partition. In frames, the global context processes the relationship between the objects and the scene, and considers interactions between different objects, thereby enabling more accurate VAD. As $gf$ already encompasses the global context from a spatial perspective, temporal pyramid pooling is utilized to obtain high-level temporal information. Additionally, this approach secures multi-scale temporal information, addressing the limitation of only using adjacent temporal information in temporal partition methods. The equation for creating $HighlevelPatches$ is as follows: \linenomathAMS
高级分区。在帧中，全局上下文处理对象与场景之间的关系，并考虑不同对象之间的交互，从而实现更准确的 VAD。由于 $gf$ 已经从空间角度包含了全局上下文，因此利用时间金字塔池化来获取高级时间信息。此外，这种方法还保护了多尺度的时态信息，解决了在时态划分方法中仅使用相邻时态信息的限制。创建 $HighlevelPatches$ 方程如下：\linenomathAMS

Temporal pyramid pooling is implemented using $f_{\text{mp}}^{l}$. In this case, $f_{\text{mp}}^{l}$ represents applying the max pooling operation $l$ times. Finally, the results are reshaped into $\mathbb{R}^{d\times c}$.
时间金字塔池化是使用 $f_{\text{mp}}^{l}$ 实现的。在本例中， $f_{\text{mp}}^{l}$ 表示应用最大池化操作 $l$ 时间。最后，将结果重塑为 $\mathbb{R}^{d\times c}$ 。

Similar to the memorization stage, the Patch Partition method is applied as described earlier. At this time, mismatched pooling technology is used to improve the VAD performance. This technique and the method of calculating the anomaly score are as follows.
与记忆阶段类似，如前所述应用 Patch Partition 方法。此时，使用不匹配池技术来提高 VAD 性能。此技术和计算异常分数的方法如下。

Mismatched Pooling. During the temporal partition process, global pooling is applied differently in the memorization and inference stages. The memorization stage employs global average pooling (GAP), while global max pooling (GMP) is used during inference. As normal objects typically exhibit more static characteristics compared to abnormal ones, we expected normal motion features to show similar average and maximum values. In contrast, abnormal motion features were anticipated to show significant discrepancies in these values. By adopting this approach, VAD performance improves by preventing the retrieval of nearby temporal patches from memory when abnormal data is input. Experimental validation of the effectiveness of this methodology is detailed in Section 4.6.
不匹配的池。在时间分区过程中，全局池化在记忆和推理阶段的应用方式不同。记忆阶段采用全局平均池化 （GAP），而在推理过程中使用全局最大池化 （GMP）。由于与异常物体相比，正常物体通常表现出更多的静态特征，因此我们预计正常运动特征会显示相似的平均值和最大值。相比之下，预计异常运动特征会在这些值中表现出显着差异。通过采用这种方法，VAD 性能通过防止在输入异常数据时从内存中检索附近的临时补丁来提高。该方法有效性的实验验证在第 4.6 节 中详细说明。

Anomaly Scoring. All nearest neighbors are computed between patches $X$ generated from test frames and the memory bank $M$. Subsequently, the representative patch $x^{*}$ is selected from the $X$ and its nearest neighbor $m^{*}$ is selected from the $M$. The distance $s^{*}$ between $x^{*}$ and $m^{*}$ is assigned as the score representing the frames. \linenomathAMS
异常评分。所有最近邻域都是在从 test frames 和 memory bank $M$ 生成的补丁 $X$ 之间计算的。随后，从 中选择代表性补丁 $x^{*}$ ，并从 中选择其最近的邻居 $m^{*}$ $M$ 。 $X$ 和 $m^{*}$ 之间的 $x^{*}$ 距离 $s^{*}$ 被分配为表示帧的分数。\linenomathAMS

As three types of memory are used, $s^{*}$ is computed three times for SM, TM, and HSM, respectively. The $s_{\text{spatial}}^{*}$ calculated from SM and $s_{\text{temporal}}^{*}$ calculated from TM determine the local anomaly score (LAS) in the local stream. Conversely, $s_{\text{high-level}}^{*}$ computed from HSM is utilized as the global anomaly score (GAS). Ultimately, the LAS and GAS combine to derive the anomaly score. The equation for calculating the anomaly score is as follows: \linenomathAMS
由于使用了三种类型的内存， $s^{*}$ 因此将分别对 SM、TM 和 HSM 计算 3 次。根据 SM $s_{\text{spatial}}^{*}$ 计算和 $s_{\text{temporal}}^{*}$ 从 TM 计算确定本地流中的局部异常评分 （LAS）。相反， $s_{\text{high-level}}^{*}$ 从 HSM 计算得出的分数将用作全局异常分数 （GAS）。最终，LAS 和 GAS 结合得出异常分数。计算异常分数的公式如下：\linenomathAMS

The proposed model was validated on three datasets: CUHK Avenue (Avenue)[21], ShanghaiTech (SHTech)[40], and IITB Corridor (Corridor)[29]. As each dataset was constructed for OCC method, the training dataset consisted only of normal videos, while the test dataset contained both normal and abnormal videos.
所提出的模型在三个数据集上进行了验证：香港中文大学大道 （Avenue）[21]、上海科技大学 （SHTech）[40] 和 IITB 走廊 （Corridor）[29]。由于每个数据集都是针对 OCC 方法构建的，因此训练数据集仅包含正常视频，而测试数据集包含正常视频和异常视频。

Avenue. The dataset consisted of 16 training and 21 testing videos filmed with a single camera at the same angle. It included anomaly events such as throwing paper, running on the road, and walking in the wrong direction.
大道。 该数据集包括 16 个训练视频和 21 个测试视频，这些视频是用单个相机以相同角度拍摄的。它包括异常事件，例如扔纸、在路上奔跑和走错方向。

SHTech. The dataset consisted of 330 training videos and 107 testing videos, capturing 13 scenes from different angles. It included various anomaly events such as the appearance of anomaly objects (e.g., cars, bicycles) and action anomalies (e.g., fighting, jumping).
SHTech 的该数据集由 330 个训练视频和 107 个测试视频组成，从不同角度捕捉了 13 个场景。它包括各种异常事件，例如异常物体（例如，汽车、自行车）的出现和动作异常（例如，打斗、跳跃）。

Corridor. The dataset consisted of 208 training and 150 testing videos filmed with a single camera at the same angle. It included previously unseen anomaly events such as protesting, hiding, and playing with the ball.
走廊。该数据集包括 208 个训练视频和 150 个测试视频，这些视频是用单个相机以相同角度拍摄的。它包括以前看不见的异常事件，例如抗议、躲藏和玩球。

Object Detection. The pretrained YOLOv5 was utilized for object detection in the videos. The bounding boxes of the objects were generated based on the last or middle frames among the $d$ input frames provided, and these bounding boxes were shared across other frames. Specifically, for the Avenue, SHTech, and Corridor datasets, we used 10, 4, 4 input frames, respectively. For the Avenue dataset, objects were detected based on the middle frame. Furthermore, to minimize the loss of temporal information, margins were added to the width and height of the bounding boxes, ensuring that they remained equal in size. Finally, the object images were resized to (224, 224) before being input into the pre-trained network.
对象检测。预训练的 YOLOv5 用于视频中的对象检测。对象的边界框是根据提供的 $d$ 输入帧中的最后一个帧或中间帧生成的，并且这些边界框在其他帧之间共享。具体来说，对于 Avenue、SHTech 和 Corridor 数据集，我们分别使用了 10、4、4 个输入帧。对于 Avenue 数据集，根据中间帧检测对象。此外，为了最大限度地减少时间信息的损失，在边界框的宽度和高度上添加了边距，以确保它们的大小保持相等。最后，在输入到预训练网络之前，将对象图像的大小调整为 （224， 224）。

Memorizing details. We utilized CLIP’s ResNet-101 as the vision encoder. The value $c^{\prime}$ for split pooling in the local stream was determined by the dataset size. Specifically, it was set to 32 for the smaller Avenue dataset and 64 for the larger SHTech and Corridor datasets. For $lf$ and $gf$ generation, average pooling with a kernel size of $(3,3)$ and stride of 1 was used. In the spatial partition, average pooling was performed with a kernel size of $(4,4)$. In the high-level partition, $L$ in TPP was set to 2, and during max pooling, a kernel size of $(2)$ with a stride of 1 was used. During memory optimization, coreset subsampling rates were set to 1, 25, and 10% for the Avenue, SHTech, and Corridor datasets, respectively.
记住细节。我们使用 CLIP 的 ResNet-101 作为视觉编码器。本地流中拆分池的值 $c^{\prime}$ 由数据集大小决定。具体来说，对于较小的 Avenue 数据集，它设置为 32，对于较大的 SHTech 和 Corridor 数据集，它设置为 64。对于 $lf$ 和 $gf$ 代，使用了内核大小和 $(3,3)$ stride 为 1 的平均池化。在空间分区中，使用核大小 . $(4,4)$ 在高级分区中， $L$ TPP 设置为 2，在最大池化期间，使用步幅为 1 的内核大小 $(2)$ 。在内存优化期间，Avenue、SHTech 和 Corridor 数据集的核心集子采样率分别设置为 1% 、 25% 和 10% 。

Inference details. The anomaly score was smoothed using a 1D Gaussian filter during testing. The parameters $\delta_{1}$ and $\delta_{2}$ used in LAS computation were set to $(0.7,0.3)$ for Avenue and $(0.5,0.5)$ for the SHTech and Corridor datasets. The parameters $\gamma_{1}$ and $\gamma_{2}$ used in the final anomaly score computation were set to $(0.7,0.3)$ for Avenue and $(0.9,0.1)$ for the SHTech and Corridor datasets. The experiments were conducted using an NVIDIA GeForce RTX 3090 graphics card.
推理详细信息。在测试期间，使用 1D Gaussian 滤波器对异常分数进行平滑处理。LAS 计算中使用的参数 $\delta_{1}$ $\delta_{2}$ 设置为 $(0.7,0.3)$ Avenue 以及 $(0.5,0.5)$ SHTech 和 Corridor 数据集。最终异常分数计算中使用的参数 $\gamma_{1}$ 和 $\gamma_{2}$ 设置为 $(0.7,0.3)$ Avenue 以及 $(0.9,0.1)$ SHTech 和 Corridor 数据集。实验是使用 NVIDIA GeForce RTX 3090 显卡进行的。

Consistent with prior research, the frame-level area under the curve (AUC) of Receiver Operation Characteristic (ROC) was used to evaluate the performance of the proposed model. This evaluation was calculated as the area under the ROC curve by varying the threshold. Specifically, we aggregated anomaly scores across all frames in the dataset to calculate the micro-averaged AUROC.
与之前的研究一致，使用接收者操作特性 （ROC） 的帧级曲线下面积 （AUC） 来评估所提出的模型的性能。该评估是通过改变阈值计算为 ROC 曲线下的面积。具体来说，我们汇总了数据集中所有帧的异常分数，以计算微平均 AUROC。

Tab. 1 presents the results of the proposed model’s performance on three benchmark datasets. It demonstrates the competitive performance compared to other state-of-the-art (SOTA) methods. Specifically, when compared to HSC[32], which utilizes appearance and motion memory, our approach outperforms by 0.4% and 2.1% on the Avenue and SHTech datasets, respectively, leveraging three memory components effectively. And, compared to FPDM[35], which also employs the CLIP, our model achieves better performance by accurately detecting anomaly events at the local stream. Furthermore, compared to the other methods using memory, a superior performance is achieved. This demonstrates that a method focused on memory optimization is more effective than traditional methods using memory. In addition, individuals without deep learning expertise can easily use the proposed approach because it does not require training.
 表 1 显示了所提出的模型在三个基准数据集上的性能结果。它展示了与其他最先进的 （SOTA） 方法相比的竞争性能。具体来说，与利用外观和运动记忆的 HSC[32] 相比，我们的方法在 Avenue 和 SHTech 数据集上分别高出 0.4% 和 2.1%，有效地利用了三个内存组件。而且，与同样采用 CLIP 的 FPDM[35] 相比，我们的模型通过准确检测本地流中的异常事件实现了更好的性能。此外，与其他使用内存的方法相比，实现了卓越的性能。这表明，专注于内存优化的方法比使用内存的传统方法更有效。此外，没有深度学习专业知识的个人可以轻松使用所提出的方法，因为它不需要培训。

To intuitively understand the use of multiple memories, for the Avenue dataset, we compared PatchCore (PC), which utilizes appearance information, with
为了直观地了解多个记忆的用途，对于 Avenue 数据集，我们将利用外观信息的 PatchCore （PC） 与
VideoPatchCore (VPC), which incorporates not only appearance information, but also motion and high-level information. Here, PC refers to the use of spatial memory only in VPC. Fig. 4(a) depicts an anomalous frame showing the action of throwing papers, necessitating consideration of motion information. Therefore, temporal memory plays a crucial role in this scenario. Fig. 4(b) depicts an anomalous frame showing the action of walking in the incorrect direction, necessitating consideration of both object and scene contexts. Therefore, high-level semantic memory plays a crucial role in this scenario. It is known that VPC, which utilizes temporal and high-level semantic memory, detects anomalies well, whereas PC, which does not utilize them, does not detect anomaly. Additional experiments on the SHTech and Corridor dataset are presented in the supplementary materials.
VideoPatchCore （VPC），它不仅包含外观信息，还包含运动和高级信息。此处，PC 仅指在 VPC 中使用空间内存。无花果。 图4（a）描绘了一个异常的画面，显示了投掷纸张的动作，需要考虑运动信息。因此，时态记忆在这种情况下起着至关重要的作用。无花果。 4（b） 描绘了一个异常帧，显示了向错误方向行走的动作，需要考虑物体和场景上下文。因此，高级语义记忆在这种情况下起着至关重要的作用。众所周知，利用时间和高级语义内存的 VPC 可以很好地检测异常，而不利用它们的 PC 不会检测到异常。补充材料中介绍了 SHTech 和 Corridor 数据集的其他实验。

Spatial and Temporal memory. Tab. 2 presents a comparison of the VAD performance affected by the spatial and temporal memory. The spatial memory effectively detects the appearance of the anomaly object, while temporal memory effectively detects the action anomaly event. As it is important to consider both the motion and appearance information for accurate anomaly detection, using both types of memory helps achieve high performance on the Avenue, SHTech, and Corridor datasets.
空间和时间记忆。 表 2 显示了受空间和时间记忆影响的 VAD 性能的比较。空间记忆有效地检测异常对象的外观，而时态记忆有效地检测动作异常事件。由于要同时考虑运动和外观信息以进行准确的异常检测非常重要，因此使用这两种类型的内存有助于在 Avenue、SHTech 和 Corridor 数据集上实现高性能。

Split pooling. Tab. 3 presents the AUROC scores based on information compression techniques in the local stream. Head compresses $lf$ by selecting $c^{\prime}$ channels from the front. Random compresses $lf$ by selecting random $c^{\prime}$ channels. Split pool divides $lf$ into $c^{\prime}$ groups and compresses by averaging the channels of each group. The common compression methods lose performance as they fail to preserve the original information, whereas the proposed compression method achieves high performance by effectively compressing the original information.
拆分池。 表 3 显示了基于本地流中信息压缩技术的 AUROC 分数。通过从前面选择 $c^{\prime}$ 通道来压缩 $lf$ 头部。通过选择随机 $c^{\prime}$ 通道进行随机压缩 $lf$ 。Split pool （拆分池） $lf$ 划分为 $c^{\prime}$ 多个组，并通过平均每组的通道进行压缩。常见的压缩方法由于无法保留原始信息而失去性能，而所提出的压缩方法通过有效压缩原始信息来实现高性能。

Mismatched pooling. Tab. 4 presents the AUROC scores based on feature pooling methods in the temporal partition. The avg and max pool perform GAP and GMP during the memorization and inference stage, respectively. The common pooling methods fail to effectively detect action anomalies, whereas the proposed method achieves effective detection. This demonstrates that differentiating between normal and abnormal temporal patches by varying the pooling methods at each stage is effective for VAD.
池不匹配。 表 4 显示了基于时间分区中特征池方法的 AUROC 分数。avg 和 max 池分别在记忆和推理阶段执行 GAP 和 GMP。常见的池化方法无法有效检测动作异常，而所提方法实现了有效检测。这表明，通过改变每个阶段的池化方法来区分正常和异常的颞部斑块对 VAD 是有效的。

Local and Global stream. Tab. 5 presents a comparison of the anomaly detection performance affected by the local and global stream. When using only the local stream, it is difficult to detect anomalies based on scenes and abnormal behaviors among multiple objects. To address this issue, we incorporated the global stream, which utilizes a wide range of spatial-temporal information. When adding the global stream to Avenue, there is a significant performance improvement, whereas adding the global stream to SHTech and Corridor results in relatively slight performance improvements. In the latter datasets, there are many situations where objects are adjacent to each other, allowing the local stream alone to partially fulfill the role of the global stream.
本地和全球流。 表 5 显示了受本地和全局流影响的异常检测性能的比较。当仅使用本地流时，很难根据场景和多个对象之间的异常行为进行检测。为了解决这个问题，我们合并了 global stream，它利用了广泛的时空信息。将全局流添加到 Avenue 时，性能会得到显著提升，而将全局流添加到 SHTech 和 Corridor 后，性能提升相对较小。在后一个数据集中，有很多情况是对象彼此相邻，允许单独的本地流部分履行全局流的角色。

Coreset subsampling ratio. Tab. 6 presents the AUROC scores based on the subsampling ratio. Generally, increasing the sampling ratio in large datasets stores a diverse range of normal features in memory, leading to improved performance, but it can also result in slower processing speed. However, using Coreset subsampling, the performance difference in memory usage between 1% and 99% is very minimal, less than 0.5%, and the frames per second (FPS) can be up to 7.3 times faster. This demonstrates the effectiveness of coreset subsampling in large datasets such as SHTech and Corridor.
核心集子采样率。 表 6 显示了基于子采样率的 AUROC 分数。 通常，在大型数据集中增加采样率会在内存中存储各种正常特征，从而提高性能，但也可能导致处理速度变慢。但是，使用 Coreset 子采样时，1% 和 99% 之间的内存使用性能差异非常小，小于 0.5%，并且每秒帧数 （FPS） 可以提高 7.3 倍。这证明了 SHTech 和 Corridor 等大型数据集中核心集子采样的有效性。

Memorizing Technique. Tab. 7 presents the AUROC scores based on the subsampling order. The former method constructs memory from each video and concatenates them, whereas the latter method constructs memory from all videos collectively. We expected that the former method, which optimizes memory for each video, would be less constrained by hardware limitations but might suffer from a lower performance due to the uneven distribution of features. However, in fact, the performance difference between the two methods is almost nonexistent. This suggests that it is possible to sufficiently store a diverse range of normal features regardless of the order of subsampling.
记忆技巧。 表 7 显示了基于子采样顺序的 AUROC 分数。前一种方法从每个视频中构建内存并将它们连接起来，而后一种方法从所有视频中集体构建内存。我们预计前一种方法（优化每个视频的内存）受硬件限制的限制较小，但由于功能分布不均匀，性能可能会降低。但实际上，两种方法之间的性能差异几乎不存在。这表明，无论子采样的顺序如何，都可以充分存储各种法向特征。

To intuitively understand the use of multiple memories, we compared PatchCore (PC), which utilizes appearance information, with VideoPatchCore (VPC), which incorporates not only appearance information, but also motion and high-level information, on the SHTech and Corridor datasets.
为了直观地了解多个记忆的使用，我们在 SHTech 和 Corridor 数据集上比较了利用外观信息的 PatchCore （PC） 和 VideoPatchCore （VPC），后者不仅包含外观信息，还包含运动和高级信息。

Fig. 1(a) shows anomaly frames depicting the action of skateboarding, necessitating consideration of motion information. Therefore, temporal memory plays a crucial role in this scenario. Fig. 1(b) shows anomaly frames depicting the action of fighting, necessitating consideration of interactions between two peoples. Therefore, high-level semantic memory plays a crucial role in this scenario. It is known that VPC, which utilizes temporal and high-level semantic memory, detects anomalies well, whereas PC, which does not utilize them, does not detect anomalies.
无花果。 1（a） 显示了描绘滑板动作的异常帧，需要考虑运动信息。因此，时态记忆在这种情况下起着至关重要的作用。无花果。 1（b） 显示了描绘战斗动作的异常帧，需要考虑两个民族之间的互动。因此，高级语义记忆在这种情况下起着至关重要的作用。众所周知，利用时态和高级语义内存的 VPC 可以很好地检测异常，而不利用它们的 PC 不会检测到异常。

Fig. 2(a) shows anomaly frames depicting the action of sudden running, necessitating consideration of motion information. Therefore, temporal memory plays a crucial role in this scenario. Fig. 2(b) shows anomaly frames depicting the action of moving a suspicious object, necessitating consideration of relationship between the person and object. Therefore, high-level semantic memory plays a crucial role in this scenario. VPC, which employs high-level and temporal memory, effectively distinguishes between abnormal and normal frames. In contrast, PC tends to produce false positives.
无花果。 图2（a）显示了描述突然奔跑动作的异常帧，需要考虑运动信息。因此，时态记忆在这种情况下起着至关重要的作用。无花果。 图2（b）显示了异常帧，描绘了移动可疑物体的动作，需要考虑人与物体之间的关系。因此，高级语义记忆在这种情况下起着至关重要的作用。VPC 采用高级和临时内存，可有效区分异常帧和正常帧。相比之下，PC 往往会产生误报。

For a deeper understanding of memory effectiveness, we computed anomaly scores using each memory module for two specific objects shown in Fig. S3. One is the most anomalous object, and the other is the most normal object, as determined by the proposed model. Each object is characterized by the spatial (S) and temporal anomaly scores (T), while frames containing these objects are assigned a high-level anomaly score (H). The experimental results demonstrate precise prediction of anomalous objects within frames by the proposed model, with each memory module effectively fulfilling its role in various scenarios. For instance, in the bicycle scenario involving abnormal appearance, S increases due to spatial memory, while in the running scenario with abnormal behavior, T increases due to temporal memory. Finally, challenging anomalies such as "wrong direction" in the local stream are detected by H increasing in high-level memory. This validates the effectiveness of the three memory modules in VAD.
为了更深入地了解内存有效性，我们使用每个内存模块计算了图 1 中所示的两个特定对象的异常分数。 S3 中。一个是最异常的物体，另一个是最正常的物体，由提议的模型确定。每个对象都由空间 （S） 和时间异常分数 （T） 表征，而包含这些对象的帧则被分配一个高级异常分数 （H）。实验结果表明，所提模型对帧内异常目标进行了精确预测，每个内存模块在各种场景中都有效地发挥了其作用。例如，在涉及异常外观的自行车场景中，S 因空间记忆而增加，而在行为异常的跑步场景中，T 因时态记忆而增加。最后，通过高级内存中的 H 增加来检测本地流中的 “错误方向” 等具有挑战性的异常。这验证了 VAD 中三个内存模块的有效性。

Fig. S4 depicts the t-SNE plots of normal patches stored in memory (denoted by green ’*’), along with normal (blue ’o’) and anomalous patches (red ’o’) from the test data. The results show that normal patches are clustered closely together compared to anomalous patches, aligning closely with the distribution of memory patches. In contrast, anomalous patches exhibit a wider dispersion and tend to be farther away from the memory patches. These findings suggest that the proposed memory effectively stores the normalcy of videos, making it suitable for VAD.
无花果。 S4 描述了存储在内存中的正常补丁（用绿色“*”表示）以及测试数据中的正常（蓝色“o”）和异常补丁（红色“o”）的 t-SNE 图。结果表明，与异常补丁相比，正常补丁紧密聚集在一起，与内存补丁的分布密切相关。相比之下，异常补丁表现出更广泛的分散性，并且往往离记忆补丁更远。这些发现表明，所提出的内存有效地存储了视频的正常性，使其适用于VAD。

We conducted a detailed analysis of performance changes based on subsampling ratios in the Avenue, Shanghai, and Corridor datasets as shown in Tab. S1. In the SHTech and Corridor datasets, the performance was better when using more memory, whereas in the Avenue dataset, the performance tended to be higher with less memory.
我们根据 Avenue、Shanghai 和 Corridor 数据集中的子采样率对性能变化进行了详细分析， 如表 S1 所示。在 SHTech 和 Corridor 数据集中，使用更多内存时性能更好，而在 Avenue 数据集中，内存较少时性能往往更高。

These results can be explained with two reasons. First, due to the diversity of normal data in the SHTech and Corridor datasets, the performance improves as more normal patches are stored in memory. Second, the Avenue dataset mainly consists of action anomalies, making it challenging to distinguish them from normal instances without using temporal information. Therefore, from a spatial memory perspective, filtering out normal patches that resemble anomalies enhances the performance. Meanwhile, other memories that utilize temporal information exhibit robust performance regardless of their size. Consequently, using only 1% of memory overall yields the best performance.
这些结果可以用两个原因来解释。首先，由于 SHTech 和 Corridor 数据集中法线数据的多样性，随着内存中存储的法线图块越多，性能也会提高。其次，Avenue 数据集主要由动作异常组成，这使得在不使用时间信息的情况下将它们与正常实例区分开来具有挑战性。因此，从空间内存的角度来看，过滤掉类似于异常的正常补丁可以提高性能。同时，其他利用时间信息的内存无论大小如何，都表现出强大的性能。因此，总体上仅使用 1% 的内存即可产生最佳性能。

However, as evidenced by the experimental results, the performance difference between using 10% and 99% of memory is very small. Therefore, in practical use, sufficiently good performance can be maintained even with memory usage set at 10% or lower.
然而，实验结果证明，使用 10% 和 99% 的内存之间的性能差异非常小。因此，在实际使用中，即使内存使用率设置为 10% 或更低，也可以保持足够好的性能。