Publication

Abstract : In the realm of clinical healthcare, medical visual question answering systems emerge as a pivotal innovation that plays a crucial role in clinical decision-making and patient care. They are designed to interpret medical images in response to free-form natural language questions, thereby facilitating a more nuanced and comprehensive understanding of patient diagnostics. This approach seeks to bridge the gap between complex medical imagery and the clinical questions they prompt, offering a novel avenue for enhancing diagnostic accuracy and efficiency. We introduce the Dual Stage Attention-based Fusion framework, which leverages an advanced feature fusion strategy, along with image encoding techniques and state-of-the-art biomedical language models, to process and analyze medical images and textual queries concurrently. By employing a dual-stage attention mechanism, the proposed model enhances feature extraction and integration, enabling a more accurate and refined analysis of both visual and textual data. The model specifically addresses the challenges commonly associated with medical visual question answering, including the intricate nature of medical imagery and the scarcity of high-quality, annotated medical datasets. The proposed multimodal augmentation technique, involving an autoencoder and generative language model, significantly enriches the data for model training which mitigates the data scarcity issue and enhances the model’s ability to generalize across different medical queries. Empirical validation of our model on the VQA-RAD and SLAKE datasets demonstrates its exceptional performance, achieving state-of-the-art accuracies of 80.8% and 86.2% respectively. These results underscore the model’s potential in AI-enhanced medical diagnostics and decision support systems to offer unprecedented diagnostic insights and patient care solutions.