Increasing accuracy of a semantic word labelling tool based on a small lexicon (2020)

Abstract: Semantic annotation has become an important piece of information within corpus linguistics. This information is usually included for every lexical unit of the corpus providing a more exhaustive analysis of language. There are some resources such as lexicons or ontologies that allow this type of annotation. However, expanding these resources is a time-consuming task. This paper describes a simple NLP baseline for increasing accuracy of the existing semantic resources of the UCREL Semantic Analysis System (USAS). In our experiments, Spanish token accuracy is improved by up to 30% using this method.

## introduction apart from raw texts, a corpus can include extra linguistic information by way of annotation. most common types of annotation are grammatical, semantic, prosodic and historical. the semantic one has become an important piece of information within the corpus linguistics research field. a corpus with this information is a useful resource to extract knowledge from a real context: as @xcite state, it can be considered as a semistructured database that offers deep information about human knowledge, concepts and relations among them. semantic annotation in corpus linguistics tends to recognise semantic categories and concepts at different syntactic levels, such as word level, phrase level or sentence level @xcite . for this purpose, the information about grammatical tags and ner (named-entity recognition) classes contribute to determine lexical semantics to some extent, but they are not sufficiently informative @xcite . semantic annotation tries to overcome these barriers by adding new categories. in this paper, we describe an nlp baseline that increases accuracy of a semantic role labelling tool that makes use of a small semantic lexicon in spanish language @xcite based on the usas tagset @xcite . we are able to increase accuracy by means of a very simple strategy that makes use of freely-available nlp toolkits such as nltk @xcite and spacy @xcite . a novel approach using wordnet similarity based on the information content theory @xcite is also employed in order to search synonyms of unknown words and, therefore, increase lexical accuracy. as a proof of concept, we carried out different experiments with texts from the finance domain. the article is organised as follows: section 2 explains our approach together with the different processes that are executed. implementation is described in section 3. we show different experiments in section 4. last, section 5 outlines conclusions and future directions. ## overview of our approach the usas lexicon is based on the longman lexicon of contemporary english @xcite , which ensures, up to a certain point, the linguistic validity and motivation of this resource. there are 21 major discourse fields, expanding into 232 category labels foot_0 . usas employs a group of labels in an attempt to include most meanings of the lexicalised unit. to employ usas lexicon, we need to extract lemma and grammatical annotation for each word. regarding the spanish version, it contains around 10,000 words and 5,000 multiword expressions, and most of them are spanish named entities such as places or locations. as a consequence of its lemma pos semantic tags business noun i2.1 a1.1.1 a5.1+++ reduced size, accuracy of the spanish usas lexicon is limited if it is used in specific text domains. for instance, only 3.30% of the lexicon entries belong to the finance domain. if this is the only resource employed for tagging, there will be many words that will not have any tag, and thus, many words will be incorrectly tagged as unknown because they do not appear in the lexicon (e.g. índice -index). a more in-depth analysis reveals that some of these words are lemmatised incorrectly (e.g. véase (note) is lemmatised as véase instead of ver). in addition, some words appear in the lexicon with only one grammatical category when they can belong to different categories (e.g. mucho (many/much) can be an adverb and adjective in spanish). to solve these problems, the simplest solution is to add new entries to the lexicon, however this is a very time-consuming task. another solution is trying to improve results of the operations required by usas such as lemmatisation or grammatical annotation. we can also try to incorporate other techniques such as stemming in order to match the stem of the word with another stem in the lexicon. in order to achieve that, we can simply employ available spanish resources from nlp toolkits such as spacy and nltk. in the rest of this section, we will describe how lexicon accuracy may be increased using some preprocessing techniques in a specific domain such as the finance one. ## implementation and deployment we develop all the components of the tagger following the specifications proposed in the previous section. fig. 1 shows a simplified workflow of this process that can be described as follows: ## experiments in the absence of resources for validating our tool we needed to build a custom-made gold standard. this is a consequence of the usas tagset, a very specific classification system, and the spanish language, which has less lexical coverage than english language using this resource. we extracted some sample texts that were not included in the corpus together with some texts from independent sources. the size of the gold standard is 20,395 words. this size is a consequence of the laborious task of manual annotation. in order to evaluate the accuracy, we followed the same metrics as @xcite . a first metric refers to those instances where the first candidate tag is correct, and a second metric makes reference to the cases where the other tags in the list are correct or closely related to the word sense. these results are shown in last, the confusion matrix of the semantic tagger according to the 21 major discourse fields of the usas taxonomy can be found in the supplementary material. confusion matrix of all the 232 subcategories would be a more detailed option but the representation of all the subcategories may be slightly confusing. the most remarkable issues are the following: ## conclusions and further work the main contribution of this paper is a strategy that utilises existing nlp toolkits such as nltk and spacy to preprocess texts in order to obtain better results using only a small lexicon as source of semantic information. this strategy is implemented following a simple and straightforward approach. empirical results are reported and compared across an ad hoc gold standard based on texts from the finance domain. this study also introduced a novel approach for extending lexical accuracy of semantic lexicons by means of synsets similarity of wordnet which did not provide the expected results, maybe as a consequence of the limited lexicon. we hope that this approach could be easily extended to other domains and even with underresourced languages. therefore, expected future work includes reproducing the good results obtained in other text domains and employing languages different from spanish or english. we also need to investigate how to take advantage of the semantic similarity provided by wordnet or even word embeddings using taxonomies like usas.