Syntactic Control of Language Models by Posterior Inference (2025)

Abstract: Controlling the syntactic structure of text generated by language models is valuable for applications requiring clarity, stylistic consistency, or interpretability, yet it remains a challenging task. In this paper, we argue that sampling algorithms based on the posterior inference can effectively enforce a target constituency structure during generation. Our approach combines sequential Monte Carlo, which estimates the posterior distribution by sampling from a proposal distribution, with a syntactic tagger that ensures that each generated token aligns with the desired syntactic structure. Our experiments with GPT2 and Llama3-8B models show that with an appropriate proposal distribution, we can improve syntactic accuracy, increasing the F1 score from 12.31 (GPT2-large) and 35.33 (Llama3-8B) to about 93 in both cases without compromising the language model’s fluency. These results underscore both the complexity of syntactic control and the effectiveness of sampling algorithms, offering a promising approach for applications where precise control over syntax is essential.

## introduction syntactic control of generated text is crucial for many domain-specific applications of language models, where structural constraints, such as formality, grammatical correctness, or adherence to a given template, can significantly affect usability and readability. despite recent advances, achieving fine-grained syntactic control remains a challenge for large-scale language models. more broadly, this challenge falls under the area of controlled generation, which studies how to guide language models to produce text with desired properties. controlling the syntactic structure of sentences is crucial in some domains. for instance, maintaining stylistic consistency is important in fields such as legal, technical, and formal writing, as well as in creative writing, where sentence structure significantly contributes to tone and clarity. in educational settings, controlling syntactic complexity figure 1 : example user-system interaction. the user specifies a target syntactic structure as a penn treebank syntax tree with ? in the place of words. the system probabilistically fills in the missing words, such that the probability of the generated sentence is high under the lm (p) and the likelihood of the targeted syntactic structure is high under the syntactic analyzer (φ). gpt4 fails to generate a valid string, as it generates an additional adjective (jj big), making it impossible for the sequence of words to have the target syntax (i.e., φ = 0). and ambiguity enables intelligent grammar tutoring in both native and foreign languages (renduchintala et al., 2016). syntactic control is also valuable for generating psycholinguistic stimuli, allowing studies on how syntax impacts cognitive load, memory, or reading time (britton et al., 1982; vos et al., 2001). more generally, it can support the generation of text that is easier to comprehend, for example, by avoiding deeply nested structures or long, ambiguous sentences (kauchak, 2013; xu et al., 2015), and reduce semantic ambiguity (e.g., prepositional attachment ambiguities), which can leave readers confused or increase cognitive processing effort. finally, from an interpretability perspective, syntactic control provides a framework for analyzing how syntax affects the behavior of language models (linzen et al., 2016; futrell et al., 2019). there are many techniques in the literature that perform controlled generation. one line of work in controlled generation 1 has focused on developing inference-time algorithms that control for qualities of generated text, such as topic, sentiment, and toxicity. perhaps, the most relevant work focuses on generating code subject to the syntactic constraints of a programming language. 2 our work, in contrast, considers controlled generation under the constraints of natural language syntax. our setting is challenging because natural language does not have simple context-free rules and, thus, cannot be easily checked for violations during generation (e.g., to rule out poor choices from being sampled). moreover, unlike programming languages, natural language syntax is much richer, inherently ambiguous, and harder to analyze, making it an interesting new challenge for controlled generation methods. an alternative to controlled generation is prompting instruction-tuned models. recent studies (sun et al., 2023; ashok and poczos, 2024) have shown the promise of instruction-tuned lms for following specific control targets, such as generating text on a particular topic or with a positive sentiment. a previously noted limitation of prompting instructiontuned models is their difficulty with syntactic control, where prompting fails to guide models to produce text with the desired syntactic structure (sun et al., 2023; ashok and poczos, 2024). it is perhaps unsurprising that prompting alone is insufficient for reliably controlling syntax. enforcing a global structure upon the generated string is inherently challenging. to improve syntactically controlled generation at inference time, we propose a sampling method that approximates the posterior distribution over strings generated by a language model under a target syntactic structure. our approach is based on sequential monte carlo, an algorithm that estimates the posterior by drawing samples from a proposal distribution and weighting them by the likelihood that the given string follows a specific attribute, in our case, a syntax tree. in this paper, we use parsers-as-taggers 3 to further guide the generation towards samples with higher likelihood. we argue that taggers can provide effective guidance since they, by design, factorize the syntactic structure and align it with each word in the sequence. specifically, we use tetratagger (kitaev and klein, 2020), which produces a constituency analysis through a clever linear encoding of the constituency tree that assigns a pair of tags to each word. while the original is trained on a masked language model, we also train tetratagger with an autoregressive language model as its backbone to guide sampling. our experiments with gpt4 (openai, 2024) align with previous findings (sun et al., 2023), showing that the gpt4 model has difficulty generating sentences with the desired constituency trees in zero-or few-shot settings. we then apply our method to gpt2-large (radford et al., 2019), as well as to llama3-8b models (llama team, 2024). sentences generated using our method adhere much more closely to the target syntax than those produced without it. for instance, with a well-chosen proposal distribution, our method improves the f1 score of gpt2-large generations from 12.31 to 93.69 without degrading the model's fluency. moreover, our method can be applied to instruction-tuned models like llama3-8b, achieving a similar f1 score with gpt2-large. these results demonstrate that controlled generation by posterior inference can make smaller models competitive with larger ones, like gpt4. ## controlled generation by posterior inference a language model (lm) p is a probability distribution over strings @xmath0, where σ is the set of tokens in the vocabulary. most state-of-the-art language models are factored into per-token conditional distributions. the probability of a string y under a language model is given by p(y) language models contain a vast amount of prior knowledge about what constitutes well-formed and fluent natural language. however, they have limited inherent understanding of external constraints or specific control targets. in the context of this work, the language model serves as a prior probability p(y) that a string y is desirable. @xcite in order to con-trol generation, we define an additional factor, the likelihood φ(t | y) that y possesses the desired attribute t. @xcite together, the prior and the likelihood define a posterior distribution p (y | t) that accounts for the inherent uncertainties in each distribution: note that z(t) is the probability that any string generated by p has the attribute t. unfortunately, the exact computation of z(t) and exact sampling from p (y | t) are generally intractable. however, we will present principled, practical approximations in § §2.1 and 2.2. likelihood. note that φ(t | y) is defined by a neural model that has been trained to approximate the true association between the target attribute tand strings y. in the case of categorical labels, foot_2 φ(t | y) may be a probabilistic classifier trained to predict a label tfor each string y (e.g., tcould be a positive or negative sentiment judgment, and φ(t | y) can be learned from a sentiment classification corpus). however, in this paper, t is a user-specified syntax tree where the internal nodes of a syntax tree are provided, but the words are left for the lm to provide (see fig. 1 for an illustration). thus, φ(t | y) is a rich probabilistic model that assigns conditional probabilities to syntactic analyses t for each string y; foot_3 see fig. 1 for an illustration. we describe our syntactic likelihood model in §3. in general, the likelihood doesn't necessarily need to be a conditional probability distribution. we can view our abstract problem as posterior ## tetratagger likelihood in this section, we describe the syntactic likelihood model φ(t | y), based on tetratagger. a tetratagger is a neural constituency parser that works by assigning two tags to each word in the sentence, except for the last word, which is assigned one tag. the tags assigned to a word represent how this word and its parent (which is an intermediate node) are situated in the constituency tree. we denote the set of tags by t , which consists of four tag types, foot_13 hence the name tetratagger. given a string of l words @xmath1, foot_14 its corresponding binarized constituency contains 2l-1 nodes. @xcite tetratagger encodes the syntactic structure by assigning two tags to each word in the sequence. therefore, φ models two conditional probability distributions over tags t per word, except the last word to which we only assign one tag. for notational convenience, we assign an extra dummy tag to the last word with probability 1, such that now each word is assigned exactly two tags. given y, the probability of a tag sequence t under φ is modeling. the tetratagger's conditional distributions φ(t 2ℓ-1 | y) and φ(t 2ℓ | y) are defined by a pair of linear transformations on top of the existing transformer-based lm. more specifically, we first pass y to a language model and extract the ddimensional latent representation h(x ℓ | y) ∈ r d of x ℓ from the last layer of the language model. next, we apply each transformation u, @xmath2. finally, we apply softmax to predict the distribution over the tags: given a dataset of strings and their corresponding ground-truth tetratags, i.e., a dataset of (y, t) pairs, we learn u and v by maximizing the conditional log-likelihood of the dataset. tokens vs. words. while the language model in eq. ( 8a ) is assumed to represent words in a string, language models operate over tokens and not words. therefore, following kitaev and klein (2020), we set h(x ℓ | y) to be the representation of the last token of x ℓ extracted from the last layer of the language model. ## autoregressive tetratagger shaping we now introduce an autoregressive tetratagger, which we use as a shaping function (ψ). we first transform the desired constituency tree to a sequence of tags @xmath3... , t 2l-1 , where l is the number of words in the tree's yield y. for simplicity, we will assume for now that each word consists of a single token (i.e., @xmath4; we will explain at the end of this section how to support multi-token words. we assume that the tetratagger is using an autoregressive lm backbone by factorizing eq. ( 7 ). thus, we define the autoregressive tetratagger probability as the crucial difference from the likelihood φ (eq. ( 7 )) is that when we predict the two tags at position n, we only have access to the string up to that point (@xmath5, rather than the complete string (y). this makes ψ an effective shaping function as it can be efficiently evaluated as the words of the sentence are being generated left to right. @xcite while one might argue that bi-directional information may be critical for accurately predicting syntactic structure, we found that our autoregressive tetratagger is a decent parser (see §5.2). note that at the final step where we generate the eos token (line 9), we calculate the weights by using the likelihood φ(t | y), as it is now a @xcite the ratio ψ(y complete sentence. @xcite lastly, we mention that ψ is parameterized in the same way as φ (see eq. ( 8a )); thus, it can be trained in precisely the same way. we provide additional training details in §5.2. multi-token words. we again highlight the fact that tetratagger assigns tag probabilities to the last token of each word. therefore, in cases where the sampled token from the proposal lm is not the last token of the word, we keep sampling from the lm until we hit the last token. to detect word boundaries, we use a heuristic algorithm, where we decode the next token and observe whether the next token starts a new word. prompt formulation. for instruction-tuned models, we first experimented with various prompt formulations and ultimately selected the bestperforming one shown in app. a.4. to include the constituency trees in this prompt for few-shot settings, we need to linearize the tree. this is done using the bracketing representation of the trees and by replacing the leaf nodes with question marks as mentioned above. note that for non-instruction models, the prompt is empty. evaluation. for evaluation of the generated sentences, we parse them to constituency trees and tetratags. we compute the following metrics: ## conclusion in this paper, we introduced a sampling algorithm designed to control the syntactic structure of text generated by language models. our method, based on sequential monte carlo, leverages a parsing-astagging framework, guiding the generation process by incorporating syntactic taggers. at each step of text generation, the algorithm samples m particles from a proposal distribution and assigns weights to them using a shaping function that assesses the syntactic structure of the partial sequences generated. our sampling algorithm ensures that both linguistic content and syntactic correctness are taken into account. our experiments with gpt2 and llama3-8b models demonstrate that our approach achieves large improvements in generating text that aligns with target syntactic structures. these results highlight the potential of combining sampling algorithms with tagging frameworks to enhance the syntactic controllability of language models. amini is supported by the eth ai center doctoral fellowship.