Rich Image Captioning in the Wild

Kenneth Tran, Xiaodong He, Lei Zhang, Jian Sun, Cornelia Carapcea, Chris Thrasher, Chris Buehler, Chris Sienkiewicz; The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2016, pp. 49-56

Abstract


We present an image caption system that addresses new challenges of automatically describing images in the wild. The challenges include generating high quality caption with respect to human judgments, out-of-domain data handling, and low latency required in many applications. Built on top of a state-of-the-art framework, we developed a deep vision model that detects a broad range of visual concepts, an entity recognition model that identifies celebrities and landmarks, and a confidence model for the caption output. Experimental results show that our caption engine outperforms previous state-of-the-art systems significantly on both in-domain dataset (i.e. MS COCO) and out-of-domain datasets. We also make the system publicly accessible as a part of the Microsoft Cognitive Services.

Related Material


[pdf]
[bibtex]
@InProceedings{Tran_2016_CVPR_Workshops,
author = {Tran, Kenneth and He, Xiaodong and Zhang, Lei and Sun, Jian and Carapcea, Cornelia and Thrasher, Chris and Buehler, Chris and Sienkiewicz, Chris},
title = {Rich Image Captioning in the Wild},
booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
month = {June},
year = {2016}
}