On the Branching Bias of Syntax Extracted from Pre-trained Language Models (2020)

Abstract: Many efforts have been devoted to extracting constituency trees from pre-trained language models, often proceeding in two stages: feature definition and parsing. However, this kind of methods may suffer from the branching bias issue, which will inflate the performances on languages with the same branch it biases to. In this work, we propose quantitatively measuring the branching bias by comparing the performance gap on a language and its reversed language, which is agnostic to both language models and extracting methods. Furthermore, we analyze the impacts of three factors on the branching bias, namely feature definitions, parsing algorithms, and language models. Experiments show that several existing works exhibit branching biases, and some implementations of these three factors can introduce the branching bias.

## introduction neural language models such as lstm @xcite , gpt2 @xcite , and bert @xcite have achieved state-of-the-art performance in various downstream nlp tasks. many recent works try to interpret their success by revealing the linguistic properties captured by these language models @xcite @xcite . one interesting line of these works tries to extract discrete constituency trees from pre-trained language models @xcite @xcite . the core of these works is to extract syntax in two stages. firstly, it defines the feature scores based on a language model, namely, the feature definition stage. secondly, it leverages the feature scores to build a constituency tree, namely, the parsing stage. however, the degree to which the extracted constituency trees match gold constituency annotations may imprecisely reflect the model's competence of capturing syntax, since their final performance may benefit from the branching bias. for example, as pointed out by @xcite , the syntax extracted from the ordered neuron based language model @xcite is biased to rightbranching languages foot_0 (e.g., english). nevertheless, the approach to measuring the bias in @xcite is highly dependent on the architecture of ordered neuron and its parsing algorithm. therefore, it is far from trivial to be applied to general pre-trained language models. this paper proposes a new approach to reveal the branching bias of syntax from pre-trained language models, which is agnostic to model architectures and parsing algorithms. the key idea of our approach is based on the following observation: we can construct a left-branching language by reversing a right-branching language and vice versa. an illustration is given in figure 1 . if a syntax extracting method has no branching bias, the parsing performances on the original language and the re-versed language should have little or no difference. therefore, the performance gap can be used as an indicator of branching bias. using our approach, we find that some recent works on pre-trained language models suffer from the branching bias @xcite @xcite . we further investigate on an in-depth question: does the bias come from language models? or the extraction methods (feature definition and parsing algorithm)? we propose a simple approach to quantitatively analyze the bias in them, which tries to control the impacts of other factors while studying a specific part in the pipeline. ## conclusion in this paper, we propose an approach to quantitatively analyze the branching bias in extracting syntax from pre-trained language models. unlike previous work, our approach is more general to be applied to any pre-trained language models and syntax extracting methods. furthermore, we systematically analyze three factors in depth that may affect the branching bias: the language model, feature definition, and parsing algorithm. our experiments show that branching biases are in many recent works. in addition, these biases can be brought by each of the three factors. we appeal that researchers should carefully design their syntax extracting method to reveal the real competence of syntax from a pre-trained language model.