Neural Language Models vs W ordnet-based Semantically Enriched Representation in CST Relation Recognition (2021)

Abstract: Neural language models, including transformer-based models, that are pre-trained on very large corpora became a common way to represent text in various tasks, including recognition of textual semantic relations, e.g. Cross-document Structure Theory. Pre-trained models are usually fine tuned to downstream tasks and the obtained vectors are used as an input for deep neural classifiers. No linguistic knowledge obtained from resources and tools is utilised. In this paper we compare such universal approaches with a combination of rich graph-based linguistically motivated sentence representation and a typical neural network classifier applied to a task of recognition of CST relation in Polish. The representation describes selected levels of the sentence structure including description of lexical meanings on the basis of the wordnet (plWordNet) synsets and connected SUMO concepts. The obtained results show that in the case of difficult relations and medium size training corpus semantically enriched text representation leads to significantly better results.

## introduction recognition of semantic relations linking text fragments may provide insight into the semanticpragmatic structure of text or be a basis for humanlike reasoning. the cross-document structure theory (cst) @xcite defines a system of semantic relations connecting topically related texts. however, due to the large number of relations and often subtle differences between them, cst relation recognition is known to be much harder than textual entailment (te) recognition. te depends on a binary decision whether one piece of text semantically entails another one due to their content, while cst is a model of more general use, but more difficult to achieve good results, especially when a classifier is trained on a domain different than the domain of its application. cst relations are based on relations between semantic content of the text fragments, like subsumption or background. such semantic oppositions are not trivial in the case of several relation types. for instance, differences in the definitions of description, follow-up or elaboration indicate some potential difficulties that may arise when we want to recognize certain types of relations. in case of description, the new additional information is about the current, non-historical nature of an event, e.g. the first sentence describes an object or entity appearing in the second sentence. elaboration provides some additional details regarding the event, but generally the sentences convey the same core information. follow-up provides some unrevealed facts about the event but appearing after occurrence of this event, thus it may be some kind of description for related events. @xcite showed that enriched graph based representation of sentences that combines elements from the levels of words, syntactic structures and also semantic structures results in significant improvement of the recognition performance in comparison to less informed approaches of simpler representation models. the semantic parts of graphs included wordnet synsets, sumo @xcite concepts, proper names and selected semantic relations from noun phrases, where the wordnet and sumo based graph elements dominates. the recent rapid development of approaches based on word embeddings, neural language models and deep neural classifiers shows that novel end-to-end methods can be very successful when applied to downstream tasks. in our work we want to verify this common claim by comparing approaches util-ising more complex text representation with those based on versatile neural language models or word embeddings. the goal of this paper is to compare two approaches: a typical 'neural' approach and the elaborated method of @xcite , as well as their combination, as the first step into this domain. the approach was presented on the basis of a well built, but medium size corpus. this may be an exemplar of a practical problem: in practice many tasks are sparingly illustrated by annotated data, and, thus, poses challenges to 'neural' methods as they require large resources to fine tune representations based on pre-trained neural models (contextual embeddings) to the problem. for the comparison we used the same annotated corpus and exactly the same representation as in @xcite and neural language models pretrained on very large corpora, and fine tuned on the same annotated corpus. our aim is to verify a claim that knowledge-based representation, especially wordnet-based, may be still useful in such cases. ## related work in @xcite cst relations were recognised by a supervised approach with boosting on the basis of lexical, syntactic and semantic features extracted from sentence pairs. the evaluation was performed in two steps: binary classification for relationship detection, and multi-class classification for relationship recognition. @xcite , in addition to labelled data, exploited also unlabelled instances that improved the performance. boosting technique was used in combination with the same set of features to classify the data in cstbank @xcite . relation detection was significantly improved to @xmath0.8839. however, recognition of the relation type was still unsatisfactory. @xcite is one of few works that address the problem of cst relations recognition for languages other than english. they utilised cst in search for topically related portuguese documents. they applied a supervised approach based on similarity measures calculated for sentence pairs from different documents: cosine similarity and a variant of the jaccard index. cut-off thresholds for the similarity were studied in combination with the performance of classifiers. @xcite constructed a cst corpus for portuguese and used it to conduct their study. @xcite applied the supervised learning to recognise a a subset of cst relations: identity, paraphrase, subsumption, elaboration and partial overlap. svm algorithm was used and examples from cstbank. the features of @xcite were expanded with: cosine similarity of word vectors, jaccard index to measure intersection of common words, longer sentence indicator, and uni-directional word coverage ratio. kumar et al. (2012a) followed zahri and fukumoto (2011 ), but restricted the set of relations to four and used only four features: tf-idf based cosine sentence similarity, words coverage ratio, sentence length difference, and longer sentence flag. the performance of svm in relation recognition was between (f1): 0.54 and 0.91 for the same relations @xcite presented results obtained with svm, a feed-forward neural network and cbr (case-based reasoning). the features of zahri and fukumoto (2011) were extended with the jaccard based similarity of noun phrases and verb phrases from the compared sentences. the best result was achieved with cbr based on the cosine similarity measure: from 0.722 to 0.966. due to the ambiguity in the interpretation of certain cst relationships, @xcite proposed several refinements to cst in order to reduce the ambiguity. they improved definitions by introducing several additional constraints on the co-occurrence of different relations in texts. the cst taxonomy was amended by adding a division based on the form and information content of relations. the improved model was used in evaluation of supervised cst relation recognition the applied features included: sentence length difference, ratio of shared words, sentence position in text, differences in word numbers across poss, and the number of shared synonyms between sentences. the j48-based classifier achieved the best average score of 0.403. in similar task of implicit discourse relation recognition @xcite used encoder-decoder (rnn) trained directly on character-level data from a large training corpus of annotated relations (reported f1 between 0.3 and 0.8, depending on the relation type). (bai and zhao, 2018) used elmo @xcite and subword-level encoding as an input to a stack of a convolutional encoder, and a recurrent encoder and a multiple layer perceptron with softmax layer as the classifier -f1 between 0.36 and 0.51 was obtained. @xcite represented input data by pre-trained word embeddings and next trained a neural tensor network on a large corpus of annotated sentences obtaining f1: 0.38 -0.72. however, @xcite used topic model word vectors as representation, but also enriched it with features extracted by dependency parser to recognise causal relations between events -a similar task to ours, but narrower. ## dataset for comparison, we utilised exactly the same dataset as in @xcite , i.e. of sentence pairs annotated with cst relations from the kpwr corpus @xcite , henceforth wut cst. the underlying corpus used to build the dataset contained 11 949 complete documents that were clustered and split into groups of 3 news, each including the most similar and potentially related documents. a set of bundles for manual annotation process was prepared -every one with 10 triples {d 1 , d 2 , d 3 } of most similar documents, that were randomly assigned to the annotators. finally, 96 bundles covering more than 2800 documents were analysed in order to discover new instances of cst relations. the imposed similarity structure facilitated searching for sentence pairs linked by a cst relation. manually annotated pairs of sentences (by at least by 3 annotators each) representing new instances of cst relations formed the gold reference subcorpus introduced for the first time by @xcite . each annotator was exploring the corpus independently. the annotators followed the guidelines used for the construction of @xcite adapted to polish. however, for the final corpus wut cst 1 we have rejected uncertain cst instances with inconsistent annotations. this means that our wut cst corpus contains only cst instances with almost homogenous annotations assigned by at least n -1, @xmath1. the final distribution of collected cst instances in our wut cst corpus is presented in figure 2 . a corpus, with similar distribution of discourse relations linking multiple documents (texts from journals in brazilian portuguese), was also introduced in @xcite . we updated the original dataset to eliminate data 1 https://clarin-pl.eu/dspace/handle/11321/305 redundancy and improve its quality by removing noisy sentence pairs. to deal with highly imbalanced class distribution we decided to completely remove specific minor classes as their sample size was too small to prepare a robust and effective supervised model in a supervised setting. the updated dataset is available at https:// clarin-pl.eu/dspace/handle/11321/781 . ## neural representation successful applications of transformer-based language models in many nlp tasks seem to be grounded in transfer learning methods and intensive model pre-training on large textual corpora. as pre-trained neural language models became very successful pushing the limits in many different natural language tasks we decided to start off with the most popular transformer-based language models and prepare baseline solutions for cst task. to prepare our baseline solutions we decided to choose elmo and bert @xcite pre-trained language models as it has been shown that they express good performance in natural language inference (nli) tasks e.g. textual entailment (te). this choice was motivated by the fact that nli tasks and cst theory are strongly interconnected. ## complex representation we started from the representations proposed by @xcite . in the original work the best result were reported for the combination of manually engineered features and complex graph-based similarities. we preserve the original setting and we shortly recollect features inspired by the literature in sec. 5.1 and the graph-based features in sec. 5.2. finally, both groups of features are concatenated into one single vector as a sentence representation in the experiments discussed in sec. 8. as graph-based features are those making the differences, cf @xcite , the combined representation will be referred as graph-based representation (or features). ## cst relation recognition in this section we describe the architecture of the baseline solutions as well as the architecture of their extensions. the architectures are generally based on contextual word embeddings computed by applying pre-trained language models to given sentence pairs. our main aim was to evaluate existing modern language models and compare them with various wordnet-based features in the task of the recognition of discourse relations. as the task itself is closely related to other nli tasks, we had assumed that the applied neural language models should bring very good results. the first architecture uses contextual word embeddings to generate sentence embeddings of sentence pairs from the wut cst corpus. given a sentence pair (s 1 , s 2 ) we generate an input vector space representation of this pair v @xmath2, e(s 2 ) ∈ r d lm computed by a given language model lm . the concatenated vector v @xmath3, e(s 2 )] is then passed through a multi-layer dense classification network with dropout and relu activations on its hidden layers, and softmax in the output layer. the baseline architecture is presented in figure 4 . since the architecture itself is very simple we are easily able to extend it and incorporate supplementary features by concatenating precomputed vector space representations of input sentences v lm with a vector v gf of additional features coming from the graph-based representation (including similarity values calculated for various graphs) v @xmath4, v gf ]. as a result the baseline architecture is expanded with pre-computed graph-based features mentioned in section 5.2. ## experimental setting to conduct the experiments we used the updated version of wut cst dataset as it was mentioned in sec. 3. we divided the dataset into three distinct parts to train, tune and evaluate selected neural models and their extensions. to prepare and test the models we applied popular transformer library called hugging face 6 most of the language models used in this work were fine-tuned to the task to obtain the best possible results. we found that fine-tuning the models slightly increases their performance. the elmo appeared to be difficult to tune, thus, we decided to test only the pre-trained version of this model (elmo nf t ). for each pair of sentences we compute their vectors using given language model and classify them with the same baseline architecture presented in figure 4 . the extended models used additional graph-based features as an input to classification network (see sec. 6). as a baseline approach we used logistic model tree (lmt) @xcite in @xcite . we selected the default parameters offered by weka framework @xcite . ## results the overall results are presented in ## conclusions neural language models (word and sentence embeddings) are capable to express enormous amounts of knowledge about possible language contexts, if pre-trained on a corpus that is large enough and representative. we applied models which have been built on very large corpora and showed very good performance when used as a basis in many applications. however, the complexity of such pre-trained models causes that machine learning algorithm must cope with it, unless they are fine tuned to a given problem on a dataset large enough. in order to do this, one requires appropriate data, both in terms of the good representation of the problem, and, very important, substantial size. extraction of elements of linguistic structures introduces generalisation, highlighting most important markers and a kind of mapping to an abstract space. we showed that such enriched representation may help in problems where we do not have enough training data. a future challenge is to find a way of balancing and combining the two approaches.