Language Generation via Combinatorial Constraint Satisfaction: A Tree Search Enhanced Monte-Carlo Approach
Abstract
Generating natural language under complex constraints is a principal formulation towards controllable text generation.
We present a framework to allow the speciﬁcation of combinatorial constraints for sentence generation.
We propose TSMH, an efﬁcient method to generate high likelihood sentences with respect to a pre-trained language model while satisfying the constraints.
Our approach is highly ﬂexible, requires no task-speciﬁc training, and leverages efﬁcient constraint satisfaction solving techniques.
To better handle the combinatorial constraints, a tree search algorithm is embedded into the proposal process of the Markov chain Monte Carlo (MCMC) to explore candidates that satisfy more constraints.
Compared to existing
MCMC approaches, our sampling approach has a better mixing performance.
Experiments show that TSMH achieves consistent and signiﬁcant improvement on multiple language generation tasks.
Introduction
Supervised techniques still dominate in natural language generation tasks.
Despite its success, supervised approaches need to be trained with massive datasets of input-output pairs, which is non-trivial to acquire.
In addition, it is hard to guarantee that the output sentences satisfy constraints.
Recent approaches ﬁrst pre-train a language model on a general-purpose dataset, then ﬁne-tune the neural net on a task-speciﬁc dataset [1, 2].
These approaches partially mitigate data hunger in training large neural networks.
Nevertheless, they still require carefully crafted datasets for ﬁne-tuning.
We present a combinatorial constraint satisfaction approach for language generation.
In particular, we sample sentences that attain high likelihoods from a language model and satisfy task-speciﬁc constraints.
Sampling sentences that attain high likelihoods in the language model ensures the sentence quality.
Constraints guarantee that the sentences ﬁt the speciﬁc language task.
The constraints can be hard ones such as the grammar rules or soft ones such as attaining positive sentiment scores.
Our method harnesses constraint satisfaction, rather than learning, to guide language generation.
In fact, there is no task-speciﬁc training in our approach.
Our approach is highly ﬂexible since constraints can be switched quickly to be adapted to a different task, even faster than ﬁne-tuning.
It also allows us to leverage the latest developments of automated reasoning for language generation.
Although the ﬁeld of language generation is dominated by learning, reasoning should play an equally important role.
Human beings can write beautiful words from reasoning over what is needed in the speciﬁc writing task, without learning from previous examples.
To better handle the combinatorial constraints, a tree search is embedded into the proposal process of the Markov chain Monte Carlo (MCMC) for constrained language generation, which suggests
Hard/Soft Constraints
[Figure] [Figure]
Tree
Search
MCMCSampling [Figure]
New Input
OutputSentence MassiveDataset
OutputSentence SupervisedTraining
Pretrained
LM
Supervised
Constraint Satisfaction
Paris is located in France.
Paris located in France.
R
I
D
Paris is located in France.
Is Paris located in France?
R
I
D
R
I
D
What is loctated in France?
R
I D ...
A single proposal in TSMH A single proposal in CGMH ...
Accept rate=10eAccept rate=100%
Is is located in France.
Accept rate=10e- ... candidate proposals that satisfy more constraints.
Our approach is motivated by Sample-Search [3, 4, 5], which integrates backtrack search into importance sampling.
Making multiple word-level changes within one proposal step of MCMC allows the direct transition between legitimate sentences, while previous approaches must go through infeasible intermediate states.
Such moves are typically rejected by MCMC and therefore result in a slow mixing rate (See Figure 1(b)).
In literature, constrained language generation has been attacked in a supervised way in [6, 7, 8, 9, 10].
There are also various works which model language rules as decomposed tree structures [11] or sentiment tags [12].
Markov Logic network [13, 14] is also used to formulate grammar rules.
The
Euclidean distance in semantic space is considered as soft constraints in [15, 16, 17].
To summarize, our contributions are: 1) We deﬁne the problem of constraint satisfaction driven natural language generation, and propose a sampling-based approach to tackle the problem with combinatorial constraints.
We propose a T ree S earch enhanced M etropolis- H astings (TSMH) framework, which mixes faster than standard MCMC in the presence of combinatorial constraints. 3)
Experiment results on generating interrogative, imperative sentences with keywords, and sentences with given sentiments demonstrate that our TSMH is able to generate sentences that satisfy more hard and soft constraints as well as retain good ﬂuency.
Language Generation via Combinatorial Constraint Satisfaction
We provide a general framework for the constrained natural language generation.
In this framework, sentences are generated by sampling from a probability distribution that is proportional to the score of a pre-trained language model times the constraint score.
Formally, let π ( x ) be the probability that sentence x is sampled, it should be propositional to:
[Equation]
Here, P LM ( x ) is the score of a language model [2, 18], which measures the quality of sentence x .
Higher P LM ( x ) means the sentence x is better in quality.
Constraint( x ) is a task-speciﬁc penalty term, which are composed of hard and soft constraint terms:
[Equation]
Both the hard constraint score Φ hard ( x ) and the soft constraint score Φ soft ( x ) are ﬂoat values ranging from 0 to 1 .
The closer to 1 , the more satisﬁed the constraints are.
Unlike supervised methods which require training with massive data, our framework can solve language generation tasks with no task-speciﬁc training.
P LM ( x ) comes from a general language model, only trained on general-purpose language tasks.
There is no ﬁne-tuning of P LM ( x ) on the speciﬁc task.
Φ hard ( x ) is based on crafted hard constraints.
Φ soft ( x ) comes from either user-deﬁned functions, or pre-trained neural networks, which again is not ﬁne-tuned on the speciﬁc task.
The overall formulation is composed of the language model and the task-speciﬁc constraints.
It allows us to sample sentences which are close to natural language while satisfying constraints.
2.1 Constraint Formulation
Hard Constraints.
The hard constraint score of sentence x is computed as: Φ hard ( x ) =
β M − (cid:80)
i c i ( x ) , where β ∈ [0 , 1] .
c i ( x ) is an indicator variable which takes 1 if the sentence x satisﬁes the i -th constraint, and M is the total number of hard constraints.
We use propositional logic to deﬁne c
i ( x ) .
Given a sentence x with length m , let w Vi ∈ { 1 , 0 } be an indicator variable that the i -th word in the sentence is in category V .
For example, given a keyword K , we can enforce its existence in the sentence by: c ( x ) =
w [K]∨ w [K]· · · ∨ w [K] m .
Here [K] is a set containing the keyword K .
Furthermore, we enforce the sentence type to be imperative by: c ( x ) =
w [VERB]∨ ( w [ADV]∧ w [VERB]) where the ﬁrst word in the sentence should be either a verb: w [VERB]or an adverb followed by a verb: w [ADV]∧ w [VERB].
The [VERB] and [ADV] represent the set of verbs and adverbs accordingly.
After deﬁning every hard constraint, to efﬁciently evaluate if the sentence preserve all the constraints, we use template to represent a set of sentences where each word is either given or speciﬁed by a word category.
We use the number of hard constraints a sentence satisﬁes at the template level to reduce the search tree size.
For example, a template [[K] , [AUX] , [OTH] , [OTH]] represent a series of sentences that the ﬁrst word is the keyword K , the second word is an auxiliary verb and the last two words are the other words.
Soft Constraints.
A soft constraint assigns a ﬂoat value between 0 and 1 to indicate the constraint satisfaction degree.
Soft constraint Φ soft ( x ) can be derived quite ﬂexibly, either from a user-deﬁned function or a pre-trained neural network.
For example, to ensure two sentences are semantically similar, the soft constraint can be the cosine similarity of their sentence vectors.
Furthermore, to ensure the sentences generated with speciﬁc sentiment, the soft constraint can be the score of a sentiment analysis neural network, representing whether the sentence has the requested sentiment.
2.2 Tree Search Enhanced MCMC
Once the probability distribution π ( x ) is deﬁned, we use Markov chain Monte Carlo (MCMC) to sample sentences.
Starting from one sentence x , MCMC moves to the next sentence x ∗ by ﬁrst generating a sample x ∗ from the proposal distribution Q ( x ∗ | x ) and then accept x ∗ with the acceptance rate: A ( x ∗ | x ) = min (cid:110) 1 , π ( x ∗ ) Q ( x | x ∗ ) π ( x ) Q ( x ∗ | x ) (cid:111) . Previous work [19] proposes to use MCMC for constrained sentence generation, namely CGMH algorithm.
Their proposal distribution only suggests sentences with one-word modiﬁcation.
Under the combinatorial constraints settings, the CGMH will run into the low acceptance rate problem , which is caused by the locality of the proposal distribution.
Our Tree Search enhanced Metropolis-Hastings (TSMH) still follows the classical MCMC procedure.
The only difference is a new proposal distribution Q ( x ∗ | x ) generated from a tree search process.
The tree search deﬁnes a probability distribution over templates of sentence moves.
Each template deﬁnes a subset of possible moves.
The sentences within the same template satisfy the same hard constraints.
The proposal probability distribution induced by the tree search algorithm biases towards templates that have high Constraint( x ) scores.
The detailed steps are illustrated below:
1) Given a sentence, our algorithm will randomly select several word positions for editing.
For all the selected word positions, we use Tree Search to efﬁciently enumerates all possible edit operations: to insert, delete, or replace the selected positions and what are the word category in the case of insert and replace .
Every leaf branch of the search tree will be our sentence template.
We extract all the sentence templates and count the number of constraints satisﬁed for each template.
We randomly sample several template with respect to a probability distribution that favors templates satisfying more constraints.
4) we ﬁll in the sampled template with words suggested by a language model.
According to the language model score times the soft constraint score P LM (ˆ x ) · Φ soft (ˆ x ) , we select one ﬁlled sentence ˆ x as proposal.
In summary, our approach alleviates the rejection problem of CGMH by enumerating all possibilities in the space of multiple word change at the template level.
This process enables us to handle combinatorial constraints and the Tree search allows us to prune branches of low quality sentences.
[Table]
Experiments
We evaluate our method on three tasks: interrogative, imperative, and ﬁxed sentiment sentences generation
[Table]
and summarize the results in Table 1.
In each task, we construct the speciﬁed type of sentences by sampling starting from keywords and enforcing task-speciﬁc constraints.
We select the sentence with the highest π ( x ) value among the sentences generated by each algorithm as the output.
In general, our method TSMH outperforms baselines and generates sentences that satisfy more constraints, are of good quality and are likely to be close to the natural language.
For the metrics in Table 1, Valid% denotes the percentage of generated sentences that satisfy all constraints.
π ( x ) is the the stationary probability value.
P GPT − 2 ( x ) is the pre-trained GPT-2 language model score, which measures the quality of the sentences.
Accept% means the acceptance rate of MCMC.
Interrogative Sentence Generation.
We enforce that sentences with a high probability to be sampled must satisfy grammar constraints of being interrogative and contain a few given keywords.
According to the results, in the experiment with keywords, 92.67% of the output sentences of our TSMH algorithm satisfy all the constraints, while merely 18.33% satisfy constraints for the baseline.
This demonstrates that our TSMH generates sentences with more constraints satisﬁed.
In addition, our method has a higher π ( x ) (stationary probability value) and acceptance rate, suggesting that the tree search embedded help MCMC to mix faster.
Imperative Sentence Generation.
We enforce grammar constraints of being an imperative sentence: the starting word should be either a verb w [VERB]or an adverb followed by a verb w [ADV]∧ w [VERB].
We also enforce keyword constraints in this task.
As shown in Table 1, our method has a higher valid percentage of 97.75% compared to 91.32% of the baseline, showing that the sentences generated by our method can satisfy more constraints.
Our method has a higher π ( x ) (stationary probability value) and acceptance rate, suggesting our approach has a better mixing behavior.
Sentence Generation with Given Sentiment.
In this task, we require the sentences to contain the speciﬁed keywords and have positive sentiments [20].
We enforce the sentences to attain high scores from a sentiment analysis neural network.
We also enforce keyword constraints as hard constraints.
We need to emphasize that, our method uses a model pre-trained on a separate dataset for sentiment analysis, which is kept intact in our experiment.
No additional ﬁne-tuning to the sentiment analysis model was performed.
Our method has a higher sentiment score, suggesting that our method generates sentences with more positive sentiments (better aligned with the target of this experiment).
Our model also leads in terms of language model scores, suggesting the language quality is better.
Conclusions
We propose a framework for constrained language generation via sampling and combinatorial constraint satisfaction.
Our strategy is to sample sentences from the constrained space with probability proportional to the language model scores.
To handle the combinatorial constraints, a tree search is embedded into the proposal process of MCMC.
Experiments demonstrate that our approach generates sentences that satisfy more constraints are likely to be close in quality to the natural language. Acknowledgements
This research was supported by the National Science Foundation (Award number IIS-1850243 and
CCF-1918327).
The computing infrastructure was partially supported by the Microsoft AI for Earth computing award.
The authors would like to thank Mr. Ning Miao for valuable suggestions.