R 2 D 2: Recursive Transformer based on Differentiable Tree for Interpretable Hierarchical Language Modeling (2021)

Abstract: Human language understanding operates at multiple levels of granularity (e.g., words, phrases, and sentences) with increasing levels of abstraction that can be hierarchically combined. However, existing deep models with stacked layers do not explicitly model any sort of hierarchical process. In this paper, we propose a recursive Transformer model based on differentiable CKY style binary trees to emulate this composition process, and we extend the bidirectional language model pre-training objective to this architecture, attempting to predict each word given its left and right abstraction nodes. To scale up our approach, we also introduce an efficient pruning and growing algorithm to reduce the time complexity and enable encoding in linear time. Experimental results on language modeling and unsupervised parsing show the effectiveness of our approach.

## introduction the idea of devising a structural model of language capable of learning both representations and meaningful syntactic structure without any humanannotated trees has been a long-standing but challenging goal. across a diverse range of linguistic theories, human language is assumed to possess a recursive hierarchical structure @xcite @xcite such that lowerlevel meaning is combined to infer higher-level semantics. humans possess notions of characters, words, phrases, and sentences, which children naturally learn to segment and combine. pretrained language models such as bert @xcite have achieved substantial gains across a range of tasks. however, they simply apply layer-stacking with a fixed depth to increase the modeling power @xcite . moreover, as the core transformer component @xcite does not capture positional information, one also needs to incorporate additional positional embeddings. thus, pretrained language models do not explicitly reflect the hierarchical structure of linguistic understanding. inspired by @xcite , @xcite proposed a fully differentiable cky parser to model the hierarchical process explicitly. to make their parser differentiable, they primarily introduce an energy function to combine all possible derivations when constructing each cell representation. however, their model is based on tree-lstms @xcite and requires o(n 3 ) time complexity. hence, it is hard to scale up to large training data. in this paper, we revisit these ideas, and propose a model applying recursive transformers along differentiable trees (r2d2). to obtain differentiability, we adopt gumbel-softmax estimation @xcite as an elegant solution. our encoder parser operates in a bottom-up fashion akin to cky parsing, yet runs in linear time with regard to the number of composition steps, thanks to a novel pruned tree induction algorithm. as a training objective, the model seeks to recover each word in a sentence given its left and right syntax nodes. thus, our model does not require any positional embedding and does not need to mask any words during training. figure 1 presents an example binary tree induced by our method: without any syntactic supervision, it acquires a model of hierarchical construction from the word-piece level to words, phrases, and finally the sentence level. ## experiments as our approach (r2d2) is able to learn both representations and intermediate structure, we evaluate its representation learning ability on bidirectional language modeling and evaluate the intermediate structures on unsupervised parsing. ## conclusion and outlook in this paper, we have proposed an efficient ckybased recursive transformer to directly model hierarchical structure in linguistic utterances. we have ascertained the effectiveness of our approach on language modeling and unsupervised parsing. with the help of our efficient linear pruned tree induction algorithm, our model quickly learns interpretable tree structures without any syntactic supervision, which yet prove highly compatible with human-annotated trees. as future work, we are investigating pre-training our model on billion word corpora as done for bert, and fine-tuning our model on downstream tasks.