AI Research

Strictly Causal Streaming Video Anomaly Detection w... | AI Research

Key Takeaways

Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core introduces a method for detecting video anomalies in real-ti...
We introduce a strictly causal streaming anomaly detector whose fixed size state is updated in O(1) time and memory per incoming frame, with no lookahead and no clip buffering.
Its temporal core is a diagonal linear state space recurrence with an input and state dependent decay gate, trained self supervised through causal next embedding prediction on a frozen visual backbone.
We derive a closed form relationship between the recurrence decay spectrum and both detection delay and the shortest anomaly it can reliably capture, then validate empirically on UCSD Ped2 and CUHK Avenue.
We further report end to end latency and throughput measured directly on Apple M3 Pro hardware, 0.74 ms and 0.77 ms per frame (over 1300 FPS), rather than simulated GPU numbers.

Paper AbstractExpand

Recent work has applied Mamba style state space models (SSMs) to video anomaly detection, yet existing approaches still rely on buffering clips or windows internally, lack a theoretical account of how temporal memory relates to detection latency, and benchmark efficiency only through GPU throughput rather than the edge hardware these methods are intended to target. We introduce a strictly causal streaming anomaly detector whose fixed size state is updated in O(1) time and memory per incoming frame, with no lookahead and no clip buffering. Its temporal core is a diagonal linear state space recurrence with an input and state dependent decay gate, trained self supervised through causal next embedding prediction on a frozen visual backbone. We derive a closed form relationship between the recurrence decay spectrum and both detection delay and the shortest anomaly it can reliably capture, then validate empirically on UCSD Ped2 and CUHK Avenue. The settling delay bound predicted from the learned base decay (57 to 59 frames) sits far above the measured detection delay (1.6 and 18.4 frames), showing that the event boundary gate, not the base decay, governs responsiveness. We further report end to end latency and throughput measured directly on Apple M3 Pro hardware, 0.74 ms and 0.77 ms per frame (over 1300 FPS), rather than simulated GPU numbers. With an untuned initial configuration the method reaches 67.9 percent and 70.2 percent frame level AUC on Ped2 and Avenue, trailing prior non causal SSM baselines in accuracy. Ablations over decay rate, state size, and gating reveal that the gate contribution is dataset size dependent, hurting accuracy on the smaller Ped2 training set but helping on the larger Avenue one. Closing this accuracy gap and extending evaluation to a third, larger benchmark are immediate next steps.

Strictly Causal Streaming Video Anomaly Detection with a Theoretically-Grounded State-Space Core introduces a method for detecting video anomalies in real-time on edge hardware. Unlike existing state-space models (SSMs) that rely on buffering video clips or windows, this approach processes frames one by one, ensuring that predictions depend only on past data. The research, authored by Yogesh Kumar, provides a theoretical framework to explain how a model’s memory relates to its detection speed and validates performance on consumer-grade hardware.

A Strictly Causal Approach

The model uses a diagonal linear state-space recurrence to maintain a fixed-size state that updates in constant time and memory per frame. By using a frozen visual backbone (ResNet-18) to extract embeddings, the system avoids the need for heavy, per-frame training. The core innovation is an "event-boundary gate"—an input- and state-dependent mechanism that allows the model to reset its state quickly when it detects a mismatch between the current input and its memory. This design enables the system to operate with strict causality, meaning it makes no lookahead and requires no clip buffering.

Theoretical Latency Analysis

The paper derives a closed-form relationship between the recurrence’s decay spectrum and its responsiveness. The author defines a "settling-delay bound," which predicts how long a model takes to react to an anomaly based on its base decay rate. Empirical testing on the UCSD Ped2 and CUHK Avenue datasets revealed that the actual detection delay (1.6 to 18.4 frames) is significantly faster than the theoretical bound predicted by the base decay (57 to 59 frames). This suggests that the event-boundary gate, rather than the base decay, is the primary driver of the model's responsiveness to anomalies.

Performance and Hardware Efficiency

The method was evaluated on an Apple M3 Pro, achieving processing speeds of over 1,300 frames per second with a latency of approximately 0.74 to 0.77 ms per frame. In terms of accuracy, the model reached 67.9% frame-level AUC on Ped2 and 70.2% on Avenue. The author notes that these accuracy figures trail prior non-causal SSM baselines, attributing the gap to the use of an untuned, initial configuration rather than a fundamental flaw in the streaming architecture.

Limitations and Future Directions

The research highlights a dependency between the gating mechanism and dataset size. Ablation studies showed that the event-boundary gate improved performance on the larger CUHK Avenue dataset but hindered accuracy on the smaller UCSD Ped2 dataset, suggesting the gate may be prone to overfitting when training data is limited. The author identifies the need to close this accuracy gap and extend evaluations to larger benchmarks, such as ShanghaiTech, as immediate next steps. Additionally, the current implementation uses a simplified frame-overlap criterion for localization, which the author notes should be treated as illustrative until fully aligned with official region- and track-based evaluation protocols.

Comments (0)

No comments yet

Be the first to share your thoughts!