On Device Information Extraction from Screenshots in form of tags. We propose a method to make mobile screenshots easily searchable. In this paper, we present the workflow in which we: 1) preprocessed a collection of screenshots, 2) identified script presentin image, 3) extracted unstructured text from images, 4) identifiedlanguage of the extracted text, 5) extracted keywords from the text, 6) identified tags based on image features, 7) expanded tag set by identifying related keywords, 8) inserted image tags with relevant images after ranking and indexed them to make it searchable on device. We made the pipeline which supports multiple languages and executed it ondevice, which addressed privacy concerns. We developed novel architectures for components in the pipeline, optimized performance and memory for ondevice computation. We observed from experimentation that the solution developed can reduce overall user effort and improve end user experience while searching, whose results are published.

EATEN: EntityAware Attention for Single Shot Visual Text Extraction Extracting Text of Interest (ToI) from images is a crucial part of many OCR applications, such as entity recognition of cards, invoices, and receipts. Most of the existing works employ complicated engineering pipeline, which contains OCR and structure information extraction, to fulfill this task. This paper proposes an Entityaware Attention Text Extraction Network called EATEN, which is an endtoend trainable system to extract the ToIs without any postprocessing. In the proposed framework, each entity is parsed by its corresponding entityaware decoder, respectively. Moreover, we innovatively introduce a state transition mechanism which further improves the robustness of visual ToI extraction. In consideration of the absence of public benchmarks, we construct a dataset of almost 0.6 million images in three realworld scenarios (train ticket, passport and business card), which is publicly available at https:github.combeacandlerEATEN. To the best of our knowledge, EATEN is the first single shot method to extract entities from images. Extensive experiments on these benchmarks demonstrate the stateoftheart performance of EATEN.