Analysing Syntactic and Semantic Features in Pre-trained Language Models in a Fully Unsupervised Setting (2022)

Abstract: Transformer-based pre-trained language models (PLMs) have been used in all NLP tasks and resulted in a great success. This has led to the question of whether we can transfer this knowledge to syntactic or semantic parsing in a completely unsupervised setting. In this study, we leverage PLMs as a source of external knowledge to perform a fully unsupervised parser model for semantic, constituency and dependency parsing. We analyse the results for English, German, French, and Turkish to understand the impact of the PLMs on different languages for syntactic and semantic parsing. We visualize the attention layers and heads in PLMs for parsing to understand the information that can be learned throughout the layers and the attention heads in the PLMs both for different levels of parsing tasks. The results obtained from dependency, constituency, and semantic parsing are similar to each other, and the middle layers and the ones closer to the final layers have more syntactic and semantic information.

## introduction transformer-based pre-trained language models (plms) such as bert @xcite , gpt-2 @xcite , xlnet @xcite , roberta @xcite , distilbert @xcite have shown state-of-art performance in many down-stream nlp tasks. the performance of such large plms has also begged the question of what type of information that these models can naturally acquire through self-supervised learning. this has been investigated especially through probing tasks to analyse the linguistic information that is learned during pre-training of such large models @xcite @xcite @xcite . one type of linguistic information that has been affluently analysed is syntactic information, and most of the recent probing studies have been based on this question: "can transformerbased large language models learn syntactic structures during pre-training?". recent studies address this question and propose unsupervised models that use syntactic knowledge obtained from plms for nlp tasks such as constituency @xcite and dependency parsing @xcite . there are two aims in this study: 1. we aim to analyse the linguistic information that is learned by plms in different syntactic levels (dependency, constituency and semantic parsing) which deviates from the previous work, and provide a comparison with different languages. 2. we aim to demonstrate whether it is possible to use the linguistic information learned from plms in a fully unsupervised model for dependency, constituency and semantic parsing. existing approaches that use pre-trained language models are evaluated mainly on constituency parsing @xcite and dependency parsing @xcite . however, there is not any study that evaluates various parsing levels including semantic parsing using the same parsing model and compares the parsing results to understand the behaviour of plms for different levels of parsing from syntax to semantics. in this paper, we evaluate a fully unsupervised model for three parsing tasks. we adopt the chart-based zero-shot parsing model @xcite ) that is based on the syntactic distance concept @xcite . to our knowledge, this will be the first study that combines syntactic distance with plms to apply to semantic parsing with zero-shot learning. in this study, we particularly use ucca graph-based semantic representation for semantic parsing, which has been tackled as a constituency parsing problem in previous studies @xcite . in addition to the well-studied languages such as english, german, and french, we also evaluate the models for turkish with a comparably smaller dataset. we obtain the best results with multilingual plms. the results show that the zero-shot parsing model performs better with shorter sentences. it also shows that plms performs the best with middle layers and the ones closer to the final layers interestingly for all of the three parsing tasks, which are in line with the previous studies. ## related work a recent research direction has been towards analysing plms without fine-tuning on a particular down-stream task to explore the type of information that is learned during pre-training, which is called probing. for that purpose, plms have been investigated in various tasks such as language modeling @xcite , dependency parsing @xcite , constituency parsing @xcite @xcite , discourse parsing @xcite , commonsense reasoning @xcite , and grammar induction @xcite . some of the studies have also questioned if the syntax is encoded in plms @xcite @xcite and some of them analysed how large language models encode other types of linguistic information such as coference, entity information, parsing, and semantic roles, and ner @xcite . in line with our work, @xcite use syntactic distance for character-level and word-level language modeling, and unsupervised constituency parsing. @xcite use plms for unsupervised constituency parsing focusing on attention heads by ranking and ensembling them. @xcite propose a model with chart-based decoder for the same problem, which also solves the greedy search problem of @xcite . all of these studies are based on the idea that the syntactic structure of sentences are naturally learned along with language modeling. some of those works @xcite also combine syntactic distance with plms to induce syntactic structure in an unsupervised setting. a recent work by @xcite also introduces joint learning of constituency parsing with dependency parsing in an unsupervised framework. our work is similar to these and we also follow the chart-based zero-shot parsing introduced by @xcite . however, this is the first time that several parsing tasks are tackled using the same unsupervised model and this is the first time ucca-based semantic parsing is performed in an unsupervised setting. ## chart-based zero-shot parsing we utilise the syntactic distance concept @xcite which was particularly explored for constituency parsing @xcite ) by directly using the plms without fine-tuning. here, we adopt chart-based zero-shot parsing based on syntactic distance for three different parsing problems that are semantic, constituency and dependency parsing to explore usability of plms in zero-shot setting. the method calculates scores for spans where an input sentence @xmath0, • • • , w n } is made up of a set of labeled spans as follows: here, s span (i, j) denotes per-span scores that are calculated recursively by splitting spans into smaller spans as defined below: for each distance function with score functions , we obtain the weights of the i th layer and j th attention head of that layer. then we calculate the span scores using the distance functions. we select the tree with the lowest score for each distance function, which leads to 4 trees in i th layer and j th attention head. therefore, we finally obtain 4 × l × a trees, where l is the number of layers and a is the number of attention heads. the final f1 scores are calculated for each tree and the highest f1 score is reported in the results. ## three levels of parsing with a single model we use the chart-based zero-shot parsing model for three types of parsing ranging in different semantic and syntactic levels with different granularities and structures of a given text: dependency parsing dependency parsing is concerned with the syntactic relations between words in a sentence. those syntactic relations are discovered in terms of dependencies of words on each other. in order to apply the zero-shot parsing model for dependency parsing, we compute the scores for each tree and then we apply eisner (eisner, 1996) decoding algorithm (rather than cyk) to produce dependency trees using the tree scores. constituency parsing constituency parsing is concerned with extracting the syntactic structure of a given text through phrasal constituents. therefore, it is more concerned with the syntactic structure of an entire sentence rather than the relations between words as opposed to dependency parsing. we apply zero-shot parsing without adding any additional step for constituency parsing. semantic parsing semantic parsing is concerned with extracting the semantic structure of a given text using a formal representation. we particularly use ucca @xcite graph-based semantic representation to extract semantic relations within the text. in order to perform ucca-based parsing, we first convert ucca graphs into constituent trees by removing discontinuities and remote edges @xcite . then we perform zero-shot learning to tackle semantic parsing as a constituency parsing problem. after finding the tree with the lowest score, we convert constituency trees back into the ucca-based graphs, and restore discontinuity units. we disregard the remote edges and implicit edges. ## experiments and results we conducted experiments to evaluate the unsupervised parser on dependency, constituency, and semantic parsing for english, german, french, and turkish since ucca-annotated datasets are only available for these languages. ## conclusion we analyse the syntactic information learned by transformer-based plms for various parsing problems (namely dependency, constituency, and semantic parsing) using a fully unsupervised zeroshot parser. to the best of our knowledge, this is the first study that compares an unsupervised model for three different parsing problems in a fully unsupervised setting and analyses the linguistic information learned from plms during pre-training for three different parsing tasks from syntax to semantics. the results show that plms provide information from mostly middle and towards the final layers for all parsing tasks, which is also in line with the previous work on constituency and dependency parsing. however, interestingly, the study shows that when it comes to structure learning, syntax and semantics are both encoded in middle and towards the final layers.