Authors: Obeida ElJundi, Mohamad Dhaybi, Kotaiba Mokadam, Hazem Hajj, and Daniel Asmar

{oae15, mad45, kam38, hh63, da20}@aub.edu.lb

Introduction

Image Captioning refers to the art of describing the content of an image by computers. It is a well-known problem in CV and NLP. It has many applications, such as improved information retrieval, early childhood education, for visually impaired persons, for social media, and so on. Although remarkable work has been accomplished recently for English, and due to the lack of large and publicly available datasets, the progress on Arabic Image Captioning is still lagging. Therefore, we developed our own dataset based on Flickr8K. It can be found under data/Flickr8k_text/ folder, named Flickr8k.arabic.full.txt. It has the exact same format as the original Flickr8K.

Paper: here
Dataset: ./data/Flickr8k_text/Flickr8k.arabic.full.txt
Presentation: here

Model

Inspired by recent advances in neural machine translation, the sequence-to-sequence encoder-decoder approach was adopted here.  For more details, please check the paper.

Results

Good example: 

Bad example: 

Results show that developing language specific datasets and end-to-end models outperforms translating English generated captions to a destination language as the later may accumulate both models errors: image captioning and translation.

Citation

Please cite this paper:

@conference{visapp20,
author={Obeida ElJundi. and Mohamad Dhaybi. and Kotaiba Mokadam. and Hazem Hajj. and Daniel Asmar.},
title={Resources and End-to-End Neural Network Models for Arabic Image Captioning},
booktitle={Proceedings of the 15th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 5: VISAPP,},
year={2020},
pages={233-241},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0008881202330241},
isbn={978-989-758-402-2},
}

Contact information

For help, issues, or personal communication related to this work, please contact Obeida ElJundi: oae15@mail.aub.edu