Characterization and classification of semantic imagetext relations The beneficial, complementary nature of visual and textual information to convey information is widely known, for example, in entertainment, news, advertisements, science, or education. While the complex interplay of image and text to form semantic meaning has been thoroughly studied in linguistics and communication sciences for several decades, computer vision and multimedia research remained on the surface of the problem more or less. An exception is previous work that introduced the two metrics CrossModal Mutual Information and Semantic Correlation in order to model complex imagetext relations. In this paper, we motivate the necessity of an additional metric called Status in order to cover complex imagetext relations more completely. This set of metrics enables us to derive a novel categorization of eight semantic imagetext classes based on three dimensions. In addition, we demonstrate how to automatically gather and augment a dataset for these classes from the Web. Further, we present a deep learning system to automatically predict either of the three metrics, as well as a system to directly predict the eight imagetext classes. Experimental results show the feasibility of the approach, whereby the predictall approach outperforms the cascaded approach of the metric classifiers.

Semantic and Interaction: When Document Image Analysis Meets Computer Vision and Machine Learning Researchers in the DIA field have proposed different interesting techniques for the extraction of elements of contents inside images at different levels: lexical, syntactical, semantical. Besides, different knowledge representation techniques have been involved for the analysis of relations between them. From these observations, it seems very interesting to propose a comparative study showing the increasing similarity between CV and DIA problems and discussing the recent trends that have been or can be proposed in a short future to try to deal more efficiently with semantic and user interaction in these fields (computer vision and robust reading). By selecting and explaining interesting recent works from the CV and DIA communities, this paper tries to bring answers to the following questions: Can we do more than automatic features selection with CNN based architectures? Can we reach a more semantical point of view on the data? What are the solutions to define adaptable semantic models that can be easily learned or userdefined? How can we get more generic systems by combining machine learning u0026 user interaction?