Open Access

ARTICLE

SAFE: A Semantic Audio-Visual Fusion Engine for Interpretable and Real-Time Crowd Anomaly Detection

Ravi Saharan, Akrisht Singh, Prakash Choudhary^*

Department of Computer Science and Engineering, Central University of Rajasthan, Ajmer, India

* Corresponding Author: Prakash Choudhary. Email:

Computer Systems Science and Engineering 2026, 50, 1-20. https://doi.org/10.32604/csse.2026.081278

Received 27 February 2026; Accepted 03 September 2026; Issue published 21 September 2026

Abstract

The task of monitoring crowds for safety is a critical challenge, yet traditional surveillance systems are often visual only, error-prone, and lack interpretability. This paper presents SAFE (Semantic Audio-Visual Fusion Engine), a real-time, multi-modal framework that delivers interpretable, operator-facing alerts by fusing complementary audio-visual cues. The visual pipeline couples SSD–ResNet face detection and Hungarian tracking with facial emotion recognition and DBSCAN-based clustering of negative affect to compute a semantic visual anomaly score that includes an explicit overcrowding signal. In parallel, the audio pipeline extracts MFCCs (Mel-Frequency Cepstral Coefficients) and employs a lightweight one-dimensional convolutional neural network (CNN) to estimate the probability of an audio anomaly. A weighted late-fusion mechanism yields a unified alert score and an annotated display that includes person IDs, emotions, and alert banners. Component-wise, the audio model attains an area under the ROC curve (AUC) of 0.99 on UrbanSound8K and 0.82 on ESC-50 public datasets. End-to-end, the system runs on a standard CPU with real-time throughput of 19.37–28.40 FPS across scenarios with 32.93–48.78 ms latency across sparse, dense, and high-motion scenes, consistently triggering fused, overcrowding, and emotion cluster alerts. Compared with single modality baselines, the proposed framework demonstrates higher robustness under low light and noisy acoustic conditions while preserving person-level interpretability, supporting practical deployment in public spaces.

---

Keywords

---