New ERC Proof of Concept project aims to interpret images automatically

Professor of Computational Semantics and AI Yoad Winter has been awarded an ERC Proof of Concept project. In the new project, principal investigator Winter and co-researchers Tejaswini Deoskar and Denis Paperno will develop an automatic captioning system for the fields of geography, medicine and art history. The project builds on the results of the ongoing doctoral research of Sofia Nikiforova in the ROCKY ERC AdG project.

ROCKY ERC AdG project

In the ROCKY ERC AdG project, images from the image bank Geograph are provided with automatically generated captions. This is possible because images are linked to an external knowledge base. Both the images and the external information are then ‘read’ and ‘interpreted’ by a convolutional neural network (CNN) for image understanding and a Transformer network for language generation.

The new ERC Proof of Concept project

In the future, this automatic captioning system can also be used more generally. This offers opportunities for, among other things, making automated help for people with a visual impairment easier and retrieving information from visual data. To achieve this, the researchers in the new Proof of Concept project are working on making the ROCKY captioning system scalable, so that it can be applied to large data sets and for more users.

The goal is to be able to use the system in non-geographical fields, which also make extensive use of imagery that requires knowledge-intensive interpretation. The fields of medicine and art history will be the first. For this purpose the Proof of Concept project creates a multi-purpose open source platform that can be cost-effectively adapted to the needs of various new fields.