Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation (2023)

Abstract: Syntactic structures used to play a vital role in natural language processing (NLP), but since the deep learning revolution, NLP has been gradually dominated by neural models that do not consider syntactic structures in their design. One vastly successful class of neural models is transformers. When used as an encoder, a transformer produces contextual representation of words in the input sentence. In this work, we propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. Specifically, we design a conditional random field that models discrete latent representations of all words in a sentence as well as dependency arcs between them; and we use mean field variational inference for approximate inference. Strikingly, we find that the computation graph of our model resembles transformers, with correspondences between dependencies and self-attention and between distributions over latent representations and contextual embeddings of words. Experiments show that our model performs competitively to transformers on small to medium sized datasets. We hope that our work could help bridge the gap between traditional syntactic and probabilistic approaches and cutting-edge neural approaches to NLP, and inspire more linguistically-principled neural approaches in the future.

## introduction once upon a time, syntactic structures were deemed essential in natural language processing (nlp). modeling and inference about syntactic structures was an indispensable component in many nlp systems. that has all changed since the deep learning revolution started a decade ago. modern nlp predominantly employs various neural models, most of which do not consider syntactic structures in their design. one type of neural models that are particularly successful is transformers @xcite . given an input text, a transformer produces a vector representation for each word that captures the meaning as well as other properties of the word in its context. such contextual word representations can then be served into downstream neural networks for solving various nlp tasks. the power of transformers in producing high-quality contextual word representations is further unleashed with large-scale pretraining @xcite . nowadays, a vast majority of nlp models and systems are built on top of contextual word representations produced by some variants of pretrained transformers. like most other neural models, transformers were developed based on human insight and trial and error, without explicit design for incorporating syntactic structures. nevertheless, there is evidence that contextual word representations produced by pretrained transformers encode certain syntactic structures @xcite and attention heads in pretrained transformers may reflect syntactic dependencies @xcite @xcite . because of the heuristic nature of the transformer model design, exactly how transformers acquire such syntactic capability remains unclear. in this paper, we propose probabilistic transformers, a very different approach to deriving contextual word representations that is based on classic nonneural probabilistic modeling with innate syntactic components. specifically, we design a conditional random field that models discrete latent representations of all words as well as a syntactic dependency structure of the input sentence, and we define a potential function which evaluates the compatibility of the latent representations of any pair of words connected by a dependency arc. we use mean field variational inference for approximate inference, producing a marginal distribution for each latent word representation, the probability vector of which can then be used as a contextual vector representation of the word. while we propose our model from a purely syntactic and probabilistic perspective that is unrelated to transformers, we show that there is a striking resemblance between the computation graph of the inference procedure of our model and that of a transformer, with our intermediate distributions over dependency heads corresponding to self-attention scores and our intermediate distributions over latent word representations corresponding to intermediate word embeddings in a transformer. in short, we start with a probabilistic syntactic model but reach the transformer! we empirically compare our model with transformers when trained with either masked language modeling or downstream tasks. our experimental results show that our model performs competitively to transformers on small to medium sized datasets. we hope that probabilistic transformers, instead of being a replacement of transformers, could benefit the analysis of the syntactic capability of transformers and at the same time inspire novel extensions of transformers. furthermore, we hope our work would promote future research of neural models that are linguistically more principled, theoretically more well-founded, and empirically no less powerful than existing models. ## probabilistic transformers we will first introduce the basic model, a conditional random field (crf) as illustrated in figure 1 , then show the inference procedure, and finally introduce some variants to the basic model. ## comparison with transformers although our probabilistic transformers are derived as a probabilistic model of dependency structures over latent word labels, we find that its computational process has lots of similarities to that of transformers. below, we first re-formulate a probabilistic transformer in a tensor form to facilitate its comparison with a transformer, and then discuss the similarities between the two models at three levels. ## experiments we empirically compare probabilistic transformers with transformers on three tasks: masked language modeling, sequence labeling, and text classification. for each task, we use two different datasets. we also perform a syntactic test to evaluate the compositional generalization ability of our model. ## related work there have been several studies trying to incorporate syntactic structures to transformers. @xcite force one attention head to attend to predicted syntactic governors of input tokens. @xcite ; @xcite try to integrate constituency or dependency structures into transformers. @xcite propose a dependency-constrained self-attention mechanism to induce dependency and constituency structures. our work deviates from all these previous studies in that we start from scratch with probabilistic modeling of word representations and dependencies, but obtain a model that is strikingly similar to transformers. ## discussion it is worth noting that in this work, our primary goal is not to propose and promote a new model to compete with transformers. instead, it is our hope that our work could benefit the analysis and extension of transformers, as well as inspire future research of transformer-style models that are linguistically more principled, theoretically more well-founded, and empirically no less powerful than existing models. in the long run, we aim to bridge the gap between traditional statistical nlp and modern neural nlp, so that valuable ideas, techniques and insights developed over the past three decades in statistical nlp could find their place in modern nlp research and engineering. the datasets used in our experiments have small to medium sizes (around 10k to 60k training sentences). our preliminary experiments with mlm on larger data show that our models significantly underperform transformers, which suggests that our model may not be as scalable as transformers. one possible cause is the absence of a feed-forward structure in our model. recent researches show that the feed-forward layers might serve as an important part of transformers @xcite . further research is needed to analyze this problem. our model can be extended in a few directions. instead of discrete labels, we may assume z variables representing discrete vectors or even continuous vectors, which may lead to more complicated inference. we may model dependency labels by pairing every h variable with a dependency label variable. while we focus on contextual word representation (i.e., encoding) in this paper, we may extend our probabilistic model to include a decoder. considering the similarity between our model and transformers, we speculate that some of these extensions may be used to inspire extensions of transformers as well. ## conclusion we present probabilistic transformers, a type of syntactic-aware probabilistic models for contextual word representation. a probabilistic transformer acquires discrete latent representations of all words in the input sentence by modeling a syntactic dependency structure of the input sentence. we use mfvi for approximate inference and find a striking resemblance between the computation graph of the inference procedure of our model and that of a transformer. our experimental results demonstrate that our model performs competitively to transformers on small to medium sized datasets.