Abstract:Recent advances in vision-language models (VLMs) have made impressive strides in understanding spatio-temporal relationships in videos. However, when spatial information is obscured, these models struggle to capture purely temporal patterns. We introduce $\textbf{SpookyBench}$, a benchmark where information is encoded solely in temporal sequences of noise-like frames, mirroring natural phenomena from biological signaling to covert communication. Interestingly, while humans can recognize shapes, text, and patterns in these sequences with over 98% accuracy, state-of-the-art VLMs achieve 0% accuracy. This performance gap highlights a critical limitation: an over-reliance on frame-level spatial features and an inability to extract meaning from temporal cues. Furthermore, when trained in data sets with low spatial signal-to-noise ratios (SNR), temporal understanding of models degrades more rapidly than human perception, especially in tasks requiring fine-grained temporal reasoning. Overcoming this limitation will require novel architectures or training paradigms that decouple spatial dependencies from temporal processing. Our systematic analysis shows that this issue persists across model scales and architectures. We release SpookyBench to catalyze research in temporal pattern recognition and bridge the gap between human and machine video understanding. Dataset and code has been made available on our project website: this https URL.
| Comments: | Accepted at IEEE/CVF Conference on Computer Vision and Pattern Recognition 2026 Project page at this https URL |
| Subjects: | Computer Vision and Pattern Recognition (cs.CV); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2505.24867 [cs.CV] |
| (or arXiv:2505.24867v2 [cs.CV] for this version) | |
| https://doi.org/10.48550/arXiv.2505.24867 arXiv-issued DOI via DataCite |
Submission history
From: Ujjwal Upadhyay [view email]
[v1]
Fri, 30 May 2025 17:59:12 UTC (32,523 KB)
[v2]
Tue, 28 Apr 2026 21:51:08 UTC (26,774 KB)