How does the intermediate fusion of acoustic and visual data work in the proposed model, and what advantages does it offer over early or late fusion strategies?
In our work, we show that unimodal approaches suffer from important limitations: acoustic-only methods are highly sensitive to ambient noise, while vision-only methods (even RGB+IR) experience significant performance degradation in low-light, nighttime, or occluded conditions. To address these issues, we adopt an intermediate audio–visual fusion strategy in which high-level features are extracted separately from acoustic spectrograms using a lightweight CNN and from RGB+IR images using a lightweight ResNet-18, and then concatenated before the final classification stage. This approach preserves the complementarity of both modalities and exploits their joint representations more effectively than early fusion, which is often affected by heterogeneous raw data, or late fusion, which combines independent decisions, resulting in higher robustness, improved accuracy, and compatibility with Edge AI constraints
