Data-Driven Detection of General Chiasmi Using Lexical and Semantic Features (2021)

Abstract: Automatic detection of stylistic devices is an important tool for literary studies, e.g., for stylometric analysis or argument mining. A particularly striking device is the rhetorical figure called chiasmus, which involves the inversion of semantically or syntactically related words. Existing works focus on a special case of chiasmi that involve identical words in an A B B A pattern, so-called antimetaboles. In contrast, we propose an approach targeting the more general and challenging case A B B’ A’, where the words A, A’ and B, B’ constituting the chiasmus do not need to be identical but just related in meaning. To this end, we generalize the established candidate phrase mining strategy from antimetaboles to general chiasmi and propose novel features based on word embeddings and lemmata for capturing both semantic and syntactic information. These features serve as input for a logistic regression classifier, which learns to distinguish between rhetorical chiasmi and coincidental chiastic word orders without special meaning. We evaluate our approach on two datasets consisting of classical German dramas, four texts with annotated chiasmi and 500 unannotated texts. Compared to previous methods for chiasmus detection, our novel features improve the average precision from 17% to 28% and the precision among the top 100 results from 13% to 35%.

## introduction knowledge about the use and distribution of stylistic devices, such as the chiasmus, can be used for stylometric purposes such as attribution to a certain author or genre @xcite . current computational methods for stylometry focus on analyzing word distributions and do not take more complex stylistic devices into account @xcite @xcite . however, since styles may also differ in their use of stylistic devices, this is an important factor to consider. different stylistic devices can also be an indicator of the structure of arguments @xcite . automatic detection of stylistic devices is an important prerequisite for enabling such analyses at scale. the chiasmus is a stylistic device that is found in texts since the antiquity. chiasmus is defined as an inversion of semantically or syntactically related words, phrases, or sentences in an a b b' a' pattern @xcite . it can be used, for example, to emphasize contrasts. one example for it is: eng ist die welt, und das gehirn ist weit (narrow is the world, and the brain is wide) wallensteins tod @xcite the semantically related words are narrow and wide, as well as world and brain. existing works for automatic chiasmus detection focus on a special case called antimetabole, which consists of an inversion of identical lemmata (i.e. @xmath0, @xmath1. an example for this is: un pour tous, tous pour un (one for all, all for one) les trois mousquetaires @xcite here, the inverted words are "all" and "one". in contrast to this, we are interested in finding general chiasmi, where the lemmata can be different. in this work we present the first machine learning based method for finding general chiasmi. existing methods find candidates by searching for inversions of lemmata. however, such an approach cannot find general chiasmi like the one by schiller mentioned above, which does not consist of an inversion of lemmata but of semantic concepts. we propose searching for chiasmus candidates by finding inversions of part-of-speech (pos) tags in an a b b' a' pattern instead of inversions of lemmata. this requires more sophisticated filtering of the candidates, which include much more false positives without rhetorical intention. to this end, we use a linear classifier and propose a novel set of lexical and semantic features. our semantic features are based on word embeddings @xcite to include information about the semantic relationship between the words constituting the chiastic pattern. compared with existing methods, our novel features improve the average precision from 17% to 28% in an experiment on four fully annotated dramas by friedrich schiller. further experiments on 493 unannotated texts confirm the usefulness of our automatic general chiasmus detector, which improves the precision among the top 100 rated chiasmus candidates from 13% to 35%. ## related work only few works address the detection of chiasmi, and the majority of those focus on antimetaboles and not the general case of chiasmus. while it is easy to capture all occurences of antimetabole in a text by just gathering all cross-wise repeating lemmata, this approach also yields a disproportionately high number of false positives @xcite . thus the main challenge is to remove the false positives from the set of chiasmus candidates. the approach of @xcite introduces a filtering step for the chiasmus candidates. here, the authors use hand-crafted features based on punctuation and word repetitions together with a maually tuned classifier. they expanded on their work with more features and machine learning @xcite , and trained a kernel svm and a linear classifier to detect antimetaboles, which constitutes the current state of the art. in contrast to their method, we search for candidates not based on lemmata but on pos tags. we furthermore include novel features incorporating lemma information and semantic information about the conceptual similarity of the words constituting the chiasmus. a first approach for general chiasmus detection was presented by @xcite by searching for inversions in syntax trees. however, this misses many true positives and they do not filter their results, leading to numerous false positives as well. in our work, we use inversions of pos tags with an additional learning-based filtering step to find more candidates and discard more false positives. ## method our method consists of two steps; first the detection of suitable chiasmus candidates, second filtering these candidates using machine learning. candidate extraction first we extract candidates from the text. in contrast to @xcite , we do not search for inversions of lemmata but of pos tags. like them, we limit the candidate phrase length to 30 tokens. with this stragety we find candidates for general chiasmi as well as antimetaboles. however, the candidates do not include lemma information a priori, in contrast to lemma-based candidate mining. we also find more false positive candidates, which raises the performance requirements for the subsequent filtering step. thus, in addition to the dubremetz features we include two new types of features in our work. ## experiments we perform two types of experiments 1 . first we evaluate the average precision (ap) of different feature combinations using 5-fold cross-validation on an annotated dataset to conduct an ablation study regarding the different feature types. ap describes the area under the precision-recall curve and is common in information retrieval. additionally, we conduct this experiment on antimetabole candidates used by @xcite for a comparison with their approach on their task. in the second experiment, we evaluate how well our model generalizes to texts from different authors not included in the training data. to this end, we extract pos tag inversions from the gerdra-cor corpus @xcite and retrieve the top-scoring 100 candidates using our pre-trained classifier. these results are evaluated manually to assess the precision among the top 100 results. for pos tagging, dependencies, and word embeddings we use the de_core_news_lg model from spacy @xcite . for the english data from @xcite , we use the spacy en_core_web_lg model. to create syntax trees, we used corenlp @xcite . feature combinations our dataset 2 consists of four annotated texts by friedrich schiller, "die piccolomini", "wallensteins lager", "wallensteins tod", and "wilhelm tell". we annotated the whole texts, finding 45 general chiasmi and 9 antimetaboles. to obtain negative examples, we randomly chose 4000 pos tag inversions from the 1,006,487 inversions in these texts. when we only search for chiasmi and exclude the antimetabole examples from the data, the combination of dubremetz and embedding features yields the best results. both the combinations of either embedding or all new features with the dubremetz features improve over the baseline. however, in contrast to the antimetaboles-only experiment, the embedding features bring a stronger improvement than the rest. the combination with lexical features even slightly decreases the ap. this can also be seen when using only the novel features. for the experiment with antimetaboles and chiasmi both considered as positive, the combination of the baseline and both novel feature sets or of the baseline and embedding features improves ap the most. the baseline features with the embedding features yield a better performance than the combination with the lexical features, both improving over the baseline. lexical features are more important for antimetabole detection, while embedding features help with the general chiasmus detection. in addition, we tested our approach on english data used by @xcite . this data consist of annotated candidates based on lemmata instead of pos tags, so only the antimetabole detection can be tested. since the novel features do not introduce additional information for candidates based on lemma inversions, the new features do not improve the detection. we also tested the approach by java (2015) on the combined candidates, yielding a precision of 0.2% and a recall of 5.6%, compared to 10% and 69% with our dle approach. this shows that in their approach most positive examples are discarded, while due to the missing filtering step many false positives occur. chiasmi in unseen texts since we have shown that both the novel lexical and embedding features and their combination can help with the task of antimetabole and general chiasmus detection, we evaluate the generalization performance of our chiasmus classifier trained on the four annotated schiller dramas to other texts. the first set of texts comprises seven other dramas by friedrich schiller, which contain a total of 2,822,313 pos tag inversions and 80,606 lemma inversions. to see how well our method generalizes to different authors, we tested it on the remaining 493 documents from gerdracor. for evaluation, we annotated the topscoring 100 candidates manually. we compare the baseline dubremetz method based on lemma inversions with the full combination of the dubremetz, lexical, and embedding features based on pos tag inversions. the number of chiasmi compared to the number of pos tag inversions in a text is extremely small; in our annotated dataset from the previous experiment, only 0.005% of the inversions were chiasmi or antimetaboles. thus, our approach is suited to find these stylistic devices. since our candidate mining approach also allows all four pos tags to be identical (i.e., a a a a instead of a b b a), we found that many of the false positives among the top 100 displayed another stylistic device: the parallelism. ## conclusion we present a novel machine learning approach for a problem seldom addressed: the detection of general chiasmi. first, we extend the candidate mining scheme used by methods for the special case of antimetabole detection to general chiasmi by searching for inversions in pos tags instead of lemmata. we then propose two new sets of lexical and embedding features, which encode information about the lemmata and word semantics. our experiments show that our approach outperforms the state of the art. especially the semantic embedding features proved useful for detecting chiasmi and antimetaboles. our approach still falls short of human performance, which is expected, as the number of chiasmi in a text is extremely low compared to the number of potential chiasmus candidates. an interesting next research step is to test the model on different languages without retraining. also, applying the method to search for parallelisms using an a b a' b' pattern seems promising.