Improve and Debias Image Caption Model by Learning Better Visual Features
Open AccessRecent technology in deep learning has advanced state-of-the-art image captioning, the task of generating appropriate textual descriptions for an image. Image captioning is not an easy task and is highly inspired by human cognition -- mainly image perception (understanding image contents including objects and their relationships) and sentence planning and generation (describing the image with a natural language). To model image perception, current state-of-the-art image captioning models heavily depend on CNN and R-CNN to extract the visual features as an input to their model. Even though these methods achieved acceptable results in generating human-like descriptions, they overlooked two main issues; the image is not perceived for the caption purpose and gender bias in the generated captions. This thesis proposes visual features based on psycholinguistic studies to explicitly model human gazing behavior. The proposed visual features utilize the captions as additional sources to learn humans' perceptions, which improves the image perception model's quality and, subsequently, the quality of generated captions. To this end, we develop a model based on the pointer network that learns the gazing mechanism from the entities in the captions to generate the visual features. We then introduce a gazing sub-network to integrate the learned visual features into the current state-of-the-art image caption models. Our experiments show a significant increase in the performance of the image captioning models when the sequence of the gazed objects is utilized as additional visual features.To address the second issue, we investigate gender bias in the current state-of-the-art image caption models. For this purpose, we propose an evaluation method that better captures the generated captions' bias. We also attempt to understand the cause of the gender bias in the caption models by analyzing their visual features. We then propose approaches to explicitly control the gender of the captions.
- All rights reserved
Notice to Authors
If you are the author of this work and you have any questions about the information on this page, please use the Contact form to get in touch with us.
| Thumbnail | Title | Date Uploaded | Visibility | Actions |
|---|---|---|---|---|
|
|
Alahmadi_gwu_0075A_15949.pdf | 2022-04-25 | Open Access |
|