Migration related semantic concepts (MRSCs) are a set of concepts related to the issue of migration, such as: law enforcement, political instability, war, natural disasters etc. They are used in the MIRROR system for searching and filtering content. Since there are no image training corpora with MRSC annotations to train a classifier, a zero-shot learning approach based on a dual encoding neural network is employed. The dual encoding network comprises 2 different networks; the first encodes videos and the second text. These 2 networks are jointly trained on large datasets that contain video and caption pairs, with the objective to minimize the distance between these 2 encodings. As a result, the dual encoding network learns to map semantically related visual and text content close together, so we can apply it to predict the similarity between an image and each MRSC. Additionally, if you augment each MRSC with a set of more descriptive sentences, that improves the accuracy of the method by a lot. This methodology can be easily extended to annotate images with visual concepts from any domain. A similar methodology can be employed in case of choosing the correct thumbnail for each news article from a set of images associated with it. The news articles are crawled from web pages. Crawling software is sometimes incapable of identifying the main image of each webpage, so they crawl every image in it. An automated solution to this issue can be found in the methodology that was followed by the MRSC, by using the dual encoding network to extract the encoding of the title of the news article as well as the encodings of each associated image and compared their similarities. As a result, the image with the highest similarity was chosen as the thumbnail that best represents the news article. Discussed methodology is used in the MIRROR system.
