ICDAR 2019 Competition on Scene Text Visual Question Answering

• Published in: 2019 International Conference on Document Analysis and Recognition (ICDAR) pp. 1563 - 1570
• Main Authors: Furkan Biten, Ali, Tito, Ruben, Mafla, Andres, Gomez, Lluis, Rusinol, Marcal, Mathew, Minesh, Jawahar, C.V., Valveny, Ernest, Karatzas, Dimosthenis
• Format: Conference Proceeding
• Language: English
• Published: IEEE 01-09-2019
• Subjects: Dictionaries; Knowledge discovery; Measurement; scene text; scene understanding; Task analysis; Text recognition; vision and language; visual question answering; Visualization
• ISSN: 2379-2140
• DOI: 10.1109/ICDAR.2019.00251

This paper presents final results of ICDAR 2019 Scene Text Visual Question Answering competition (ST-VQA). ST-VQA introduces an important aspect that is not addressed by any Visual Question Answering system up to date, namely the incorporation of scene text to answer questions asked about an image. The competition introduces a new dataset comprising 23,038 images annotated with 31,791 question / answer pairs where the answer is always grounded on text instances present in the image. The images are taken from 7 different public computer vision datasets, covering a wide range of scenarios. The competition was structured in three tasks of increasing difficulty, that require reading the text in a scene and understanding it in the context of the scene, to correctly answer a given question. A novel evaluation metric is presented, which elegantly assesses both key capabilities expected from an optimal model: text recognition and image understanding. A detailed analysis of results from different participants is showcased, which provides insight into the current capabilities of VQA systems that can read. We firmly believe the dataset proposed in this challenge will be an important milestone to consider towards a path of more robust and general models that can exploit scene text to achieve holistic image understanding.