AI S ensing for Robotics using Deep Learning based Visual and Language Modeling (2020)

Abstract: An artificial intelligence(AI) system should be capable of processing the sensory inputs to extract both task-specific and general information about its environment. However, most of the existing algorithms extract only task specific information. In this work, an innovative approach to address the problem of processing visual sensory data is presented by utilizing convolutional neural network (CNN). It recognizes and represents the physical and semantic nature of the surrounding in both human readable and machine processable format. This work utilizes the image captioning model to capture the semantics of the input image and a modular design to generate a probability distribution for semantic topics. It gives any autonomous system the ability to process visual information in a human-like way and generates more insights which are hardly possible with a conventional algorithm. Here a model and data collection method are proposed.

## introduction in a world gifted with visible light facilitating information sharing, the living creatures have developed organs for sensing the light to understand their surrounding. in an autonomous system, this information is captured in ir, uv, and visible spectrum involving sophisticated sensors and is processed using complex algorithms. at consumer electronics show (ces) 2020, samsung presented ballie which is a personalized robot with ideas to make it self-aware of its surroundings and control iot devices around home to make the environment better. with companies targeting to launch smart home robots with capabilities of following voice commands, there is a need to develop a system that can automatically understand the semantics of the environment and take appropriate decisions on its own. the latest work in scene understanding involved construction of knowledge graph for visual semantic understanding @xcite . the authors used ontology graph in combination with visual captioning to describe the scene. another approach for functional scene understanding was introduced using semantic segmentation @xcite . all these scene understanding approaches make a system specialized in certain tasks and working environment while failing to generalize across various types of situations and capture the human emotions. it is efficient to make decision, based on a structured description of the scene instead of working on raw pixel information. fig. 1 shows scenarios where a human can easily interpret the meaning of the scene. it is easy to tell from the fig. 1b that firefighters are trying to put out the fire from building. this is also true for all the fig. 1a , 1b where a human can understand and explain the scene easily through a language representation. in this work we recommend an ai sensing system that can semantically interpret the environmental conditions, objects, relations and activity carried out from the visual feed. these interpretations are converted into text for human understanding and probability distribution for the control system to process and take decisions. the main intention of this work is to have a neural network based sen-sor processing unit capable of extracting semantic context while deployed on low powered compute hardware. this paper is divided as follows: ## proposed approach in this work, a modular approach is proposed to represent the semantic content of the outside world through vision sensor. a detailed flow diagram of the proposed method is shown in fig. 2 . it consist of three sub-modules namely, cnn feature extractor, language module, and environment context probability detector module. it combines visual, language, and context detection modules to assist the control unit to make decisions based on non-task specific environment details. ## dataset and considerations the cnn feature extractor is a pre-trained model trained on imagenet dataset @xcite for classifying 1000 objects. the language module is trained using coco image captioning dataset @xcite which consist of image and captions in target language. a bleu-1 score of 70.7 is achieved for the language module. the dataset for the environment context module is similar to the text sentiment classification dataset. the input will be a sentence and the labels are one-hot vector of target class. a dataset is created from a portion of coco caption where the semantic context topics are environment, situation, mood, presence of human, and objects in the scene as shown in fig. 4 . there are several logical considerations to be take while adopting this method. few of them are, ## conclusion the main objective of the work is to use neural networks to understand and represent the physical environment around the system. this work serve as an add-on to the existing control system by providing additional set of inputs capturing the semantic meaning. an image captioning based approach is used to obtain semantic content of the surrounding and it is represented in a probability distribution.