An Element-wise Visual-enhanced B i LSTM - CRF Model for Location Name Recognition (2020)

Abstract: In recent years, previous studies have used visual information in named entity recognition (NER) for social media posts with attached images. However, these methods can only be applied to documents with attached images. In this paper, we propose a NER method that can use element-wise visual information for any documents by using image data corresponding to each word in the document. The proposed method obtains element-wise image data using an image retrieval engine, to be used as extra features in the neural NER model. Experimental results on the standard Japanese NER dataset show that the proposed method achieves a higher F1 value (89.67%) than a baseline method, demonstrating the effectiveness of using element-wise visual information.

## introduction since the 1990s, information extraction, in which computers are used to extract structured data from unstructured documents, has been extensively studied @xcite . among the entities to be extracted, location information (where) is one of the essential components (5w1h) of event information to be extracted, and the process has evolved to include various tasks, such as location name disambiguation and mapping of location names to real-world geographic locations @xcite . location name recognition has been typically conducted as a named entity recognition (ner) task @xcite . in this field, deep learning models using visual information have been actively studied in recent years, especially in the extraction of named entities (nes) from posts in social networking services (snss) such as twitter and snapchat @xcite ; @xcite . these methods use images attached to a post as multimodal features to disambiguate word meanings in the post. for example, the word washington can be used to refer to washington, d.c. (location) or the presidency of george washington @xcite . looking at the attached image, washington could be further disambiguated. as mentioned above, visual information is considered capable of explaining word meanings and provide useful information for location name recognition. for example, figure 1 shows images of two different modernized cities, shenzhen in china and dubai in the uae, and figure 2 shows images of rural villages, manali in india and hakone in japan. one can easily recognize common objects from these images: skyscrapers in figure 1 , and townscapes surrounded by mountains and rivers in figure 2 . similarities like these would provide sufficient information to consider that words like shenzhen and dubai in documents have the same ne aspect. in this paper, we propose a method for location name recognition that utilizes images more effectively. specifically, image data are obtained for each word in a document through an image retrieval engine, using the words in the document as a search query, and used as an extra multimodal feature in a neural ner model. the proposed model has two advantages. first, it is robust to unseen words that do not appear in the training data; standard ner models tend to be vulnerable to unseen words. image data corresponding to each word in the document would provide additional information to clarify word meanings, as shown in the examples of figure 1 and figure 2 . second, our method can be applied to any documents to obtain element-wise image data corresponding to each word in the document; those of previous studies can only be applied to documents with images attached to them. in addition, in the proposed method, we introduce a gate mechanism to control the extent to which the visual features from images are input to the neural ner model. polysemous words, abbreviations, and misspellings in a document could result in inappropriate instances in the image data obtained by the image retrieval engine. the gate's function is to remove the harmful effects derived from these instances by increasing or decreasing the degree of effect of a visual feature in the model when an image is appropriate or inappropriate for the document's context, respectively. we evaluate the model's performance for location name recognition using a standard bilstm-crf model as our baseline and then show the effectiveness of element-wise visual information and gate mechanism, through our experimental results. ## bilstm-crf model this section describes the details of the bilstm-crf model as a basis for our baseline model. as mentioned in the previous section, the bilstm-crf model is one of the most common models for ner. the input is a word or character sequence in a document and the output is a sequence of labels representing ne information. in this study, we use a character-based model because the dataset used in the experiments is japanese and errors caused by word segmentation can be ignored. characterbased models have been confirmed to outperform word-based models when japanese documents are used @xcite . let @xmath0, @xmath1, and @xmath2. here, x is created by concatenating three types of vector (embedding) sequences x c , x w , and x f . the t-th element x t of x is given by equation ( 1 ). the sequence x @xmath3,t } m @xmath4. each element of x c , x c,t corresponds to a glove embedding @xcite for the corresponding character in c. in addition to x c , we also use x w and x f , which are sequences of word embeddings to integrate the word meanings into the input. x w is the characterbased word sequence, which is a word sequence whose length is the same as that of the character sequence c. let @xmath5, where m denotes the number of characters of a word sequence. here, let @xmath6, (@xmath7. w is a variation of s, which is created by repeating each word |s i | times where |s i | denotes the number of characters in the word s i . note that w t and w t+1 in w are the same value if they come from the same word s i . the sequence x @xmath8,t } m @xmath9. each element of {x w,t } is also trained by glove @xcite . the sequence x f is the alternative version of x w , using the flair training scheme @xcite instead of glove. the input x is given to the lstm network layer. in this layer, each unit of the lstm updates the state of the t-th element x t on the basis of the previous lstm (c t-1 ) and the hidden state (h t-1 ), and outputs the updated state as h t and c t . in a bilstm network, the output -→ h of the forward lstm and the output ←h of the backward lstm are combined to compute the total output ← → h . next, the output of the lstm network layer, ← → h , is sent to the next crf layer. in this layer, the labeling scheme that takes into account the transition probability between labels is carried out, and the output sequence y is calculated against x. the output is selected for the optimal sequence on the basis of the equation ( 4 ) where ϕ is the feature function and w crf is a weight coefficient learned in this layer. the element y t of the output label sequence y represents the entity label for each character c t . in general, named entity may be composed of multiple characters. so, we use the bio scheme to represent the chunks of the named entity. ## proposed method the proposed method is a variation of the visualenhanced bilstm-crf models that enable to integrate visual features into the basic bilstm-crf model described in the previous section. the proposed method utilizes element-wise visual features by obtaining image data for each word in the input document through an image retrieval engine where each word in the document is used as a search query. by retrieving images associated to words, the proposed method can be applied to any documents, while the previous visualenhanced models mentioned in section 2 can only be applied to documents with images attached to them. figure 3 shows an overview of the proposed method. the left-hand side shows the basic bilstm-crf model. the right-hand side shows the proposed module to create element-wise visual features. in this section, we describe the proposed method step by step. first, we explain the procedure for constructing queries from the input document (section 4.1). next, we explain how to obtain visual embeddings (section 4.2) and integrate the visual features to the original text features (section 4.3). we then update the input vector sequence shown in equation (1) to carry the visual features to the bilstm-crf (section 4.4). ## conclusion in this study, we proposed a ner model that uses images corresponding to all nouns in a document as features and a gate mechanism that controls the extent to which visual features are provided as input to the neural ner model. we conducted experiments to confirm its performance in location name recognition. experimental results show that the proposed method achieved a higher f1value performance than the baseline model in the ene corpus dataset, with a significant difference of @xmath10.01. in future research, we will investigate whether the proposed model is effective for cases other than location names. we also aim to improve our model to be more effective by conducting elaborate query investigations that are motivated by the error analysis. the hyper-parameter k, which means the number of images per word, would be critical for obtaining valuable visual embeddings. therefore, we will also investigate whether the larger the k, the better the location name recognition performance. the experimental results showed that the proposed method has little contributions when query words are polysemous. we would like to attempt word sequence queries with nouns and adjectives/verbs instead of single noun queries.