Vision-Language Representation Learning for Image Captioning with Attention-Based Interpretability

Recent advances in vision–language models have significantly improved image captioning by enabling more effective learning of visual and textual representations. Despite these improvements, understanding how transformer-based models generate captions remains challenging because their decision-making process is often difficult to interpret. This paper presents a vision–language representation learning framework for image captioning that integrates a Vision Transformer (ViT) encoder with a BERT-based decoder to generate descriptive image captions while incorporating attention-based interpretability. The proposed framework employs attention visualization and head-wise attention analysis to identify the image regions that contribute to caption generation, providing greater insight into the model’s internal reasoning process. Experimental evaluation on the COCO Image Caption dataset achieved BLEU-1, BLEU-2, BLEU-3, and BLEU-4 scores of 0.6384, 0.4636, 0.3383, and 0.2516, respectively. In addition, a Gradio-based interactive web application was developed to enable real-time caption generation together with attention map visualization. The experimental results demonstrate that the proposed framework generates semantically meaningful captions while improving the transparency and interpretability of transformer-based image captioning systems.