Publication

Abstract : Audio-Visual Question Answering (AVQA) requires reasoning across the interdependencies of the audio and visual modalities, therefore, making the task more challenging than the traditional Visual Question Answering (VQA). Current architectures such as the Collective QuestionGuided Network (CQGN) can perform well on benchmark tasks such as MUSICAVQA; however, these systems are hindered by redundancy in question generation and large computation costs. In response to the short-comings, we propose an augmented CQGN architecture with an Adaptive Question Generation and Selection Module (AQGSM). It is a module that applies an ensemble of transformer encoders, namely, DeBERTa-v3-base, RoBERTa-base, ELECTRA-base, and BERT-base, with the weight of 0.28, 0.25, 0.24, and 0.23 respectively, to make use of context-sensitive question generation and selection.Image representations are acquired through the CLIP ViT-B/32 encoder, whereas audio representations are acquired through Wav2Vec 2.0, and the two modalities are then combined in a multimodal integration component.With a systematic hyperparameter search on a 45,000 instance AVQA corpus, the weighted ensemble accuracy is 93.24 percentage on the validation split, and the expected performance can be improved to 94.7 percent with additional optimization. Additionally, the architecture has lowered inference time to 0.65s/batch.These findings imply an improved efficiency, scalability, and usability over baseline systems like CQGN (77 per cent accuracy) and the MAV-LLM/LAVA pipelines.