Relation-Constrained Decoding for Text Generation
Anonymous Author(s)
AffiliationAddress email
Abstract
The dominant paradigm for neural text generation nowadays is seq2seq learningwith large-scale pretrained language models.
However, it is usually difficult tomanually constrain the generation process of these models.
Prior studies haveintroduced Lexically Constrained Decoding (LCD) to ensure the presence of prespecified words or phrases in the output.
However, simply applying lexical constraints has no guarantee of the grammatical or semantic relations between words.
Thus, more elaborate constraints are needed.
To this end, we first propose anew constrained decoding scenario named Relation-Constrained Decoding (RCD) ,which requires the model’s output to contain several given word pairs with respectto the given relations between them.
For this scenario, we present a novel plug-andplay decoding algorithm named RE lation-guided probability S urgery and b E am
AL location (RESEAL), which can handle different categories of relations, e.g., syntactical relations or factual relations.
Moreover, RESEAL can adaptively “reseal”the relations to form a high-quality sentence, which can be applied to the inferencestage of any autoregressive text generation model.
To evaluate our method, we firstconstruct an RCD benchmark based on dependency relations from treebanks withannotated dependencies.
Experimental results demonstrate that our approach canachieve better preservation of the input dependency relations compared to previousmethods.
To further illustrate the effectiveness of RESEAL, we apply our methodto three downstream tasks: sentence summarization, data-to-text generation, andfact-based text editing.
We observe an improvement in generation quality 1 .
Introduction
Incorporating complex manual constraints into neural text generation is a challenging research topic.
One of the most important manual constraints is the relation constraint , i.e. to guarantee that twopre-specified words must appear in the generated text and keep the given relation between them.
Such relation constraints have various applications.
For instance, data-to-text generation [9, 17] andfact-based text editing [14] aim to ensure the presence of given facts (entities and relations betweenthem) in the output.
Moreover, in sentence summarization task [31], there are some key semanticrelations that must be preserved to ensure the fluency and factual constituency of the summaries.
The most prominent paradigm for text generation is seq2seq learning by finetuning the large-scalepretrained models [18, 29] and obtaining the outputs by beam search in an autoregressive manner.
However, this paradigm often fails to satisfy the complex constraints because the objective of beamsearch process is the log-likelihood of candidate sentences, not the number of constraints they havemet.
In other words, there is no explicit mechanism to enforce these constraints.
To tackle thisproblem, previous works [12, 13, 26] propose Lexically Constrained Decoding (LCD) to specify somekeywords that should be contained in the output.
However, simply utilizing these lexical constraints
What a fancy phone fancy amod
Beams amod phone
This phone is fancy
Source Input (optional)
Decoding Time Step. . . . . .
R e l a t i on
I d e n t i f i e r 0.
p rel p vocab phone . . .
p vocab fancy nsubj
What a fancy phone !
Target Output:
Decoding…
Continue Decoding… . . .
What a fancy phone !
[Dependency Placement] phone fancy amod Relation Constraints:
Output: amod
Elliot See was born in Dallas .
[Data-to-Text Generation]
Elliot See
Dallas birth place
Relation Constraints:
Output: birth place
Relation Constraints: k eep r edu c e
Bionico is a dessert from Mexico .
[Fact-Based Text Editing]
Bionico
Mexico Relation Constraints:
Output: country
Draft (Input):
Bionico is a dessert.
country still struggles to ensure the word relation constraints.
Therefore, we need to explore another form ofconstrained decoding to handle the relations between words.
In this paper, we propose a new constrained decoding scenario named Relation-Constrained Decoding(RCD) .
Specifically, we adopt the triplets ( head, relation, tail ) as the relation constraints .
At thedecoding stage, we aim to force the model output to include these relation constraints.
The right sideof Figure 1 shows two instances for RCD with different relation types.
In an evident way, satisfyingrelation constraints requires satisfying corresponding lexical constraints.
A straightforward solutionto this problem is to generate a set of candidate sentences using any LCD algorithm to ensure thekeyword preservation, and then rerank them by the number of relation constraints they have met.
However, this approach requires to first generate a number of whole sentences, which is inefficientand inflexible.
To this end, we propose RESEAL ( RE lation-guided probability S urgery and b E am
AL location), a relation-guided decoding algorithm for RCD that can dynamically adjust the choiceof words during decoding.
As illustrated in Figure 1, RESEAL modifies the standard beam searchprocess and incorporates an external relation identifier to help the model to identify the presence ofrelation constraints.
Based on the relation identifier, RESEAL dynamically adjusts the probability ofthe candidate constrained words in the generation process.
Thereafter, we further extend the dynamicbeam allocation (DBA) algorithm [26] to satisfy the relation constraints.
Among all categories of word relations, the dependency relation is most basic, standard and representative which has various public available datasets for evaluation.
Therefore, in this paper, we mainlyfocus on the dependency relation scenario of RCD.
To illustrate the effectiveness of our proposed
RESEAL, we construct a benchmark from publicly available treebanks [34] that contain sentencesand their dependency trees annotated by human.
We randomly sample a subset of dependency tripletsas the input constraints and regard the original sentences as reference outputs.
We call this task“Dependency Placement”.
Experiment results show that our method outperforms baselines and LCDmethods on dependency coverage (the ratio of satisfied relation constraints for dependency).
Afterthat, to showcase the applicability of this work, we further explore some potential applications of
RCD.
We conduct extensive experiments on three downstream tasks: sentence summarization (withdependency relations), fact-based text editing (with relations between two entities in the knowledgegraph), and data-to-text generation (with relations extracted from knowledge bases).
Across differenttasks, we observe a consistent improvement over the strong baselines.
To sum up, the contributions of our work are three-fold: (i)
We propose Relation-Constrained
Decoding (RCD), a scenario for constrained text generation, and construct its benchmarks.
Tothe best of our knowledge, we are the first to study this problem.
(ii) We design RESEAL, adecoding algorithm that
Problem Formulation
In this section, we first formulate the Relation-Constrained Decoding (RCD) problem.
For the textgeneration tasks, given an input sequence X
=
( x 1 , x 2 , ..., x N ) , where N is the input sequence length,x i ∈ V S and V S is the source vocabulary, the model generates a sequence Y =
( y 1 , y 2 , ..., y M ) ,where M is the output sequence length, y i ∈ V T and V T is the target vocabulary.
The conditionalprobability of Y given X and model parameter θ can be calculated as follows:
[Equation]
Eq.
1 usually acts as an objective for beam search.
In this paper, we denote each relation constraintas a triplet ( h, r, τ ) , where h is the head , τ is the tail , and r is the relation between them.
Given anunordered relation constraints set C =
{ ( h l , r l , τ l ) } Ll =1 , where L is the number of constraints andh l , τ l ∈ V T , we aim to make the output Y satisfy the constraints in C as much as possible.
For themodel’s output Y , we denote C
( Y ) =
{ ( h ′ l , r ′ l , τ ′ l ) } Ml =1 be the relation triplets of Y , and then wepropose to jointly optimize Eq.
1 and | C ∩ C ′ ( Y ) | as a novel objective for RCD.
Methodology
[Table] [Table]
: if RESEAL aims to enforce the relation constraintsduring standard beam search.
Algorithm 1 givesan overview of RESEAL.
To start the decoding, the decoder input is initialized with a single⟨ BOS ⟩ token.
At each time step t , the modelmaintains the k -best candidate sentences, wherek is the beam size.
The decoder produces thedistribution p vocab ( w | y <t , X ; θ ) for each tokenw in the target vocabulary V T (line 3).
Eachcandidate has different p vocab respectively to produce k |V T | candidates, then we can select topk candidates from them by the cumulative logprobability (line 5).
The decoding ends whencandidates contain k finished sentences.(line 67).
Different from standard beam search, RESEAL follows a two-step approach as follows:
Step 1: Probability Surgery (line 4)
RESEAL operates the produced probability distributionsaccording to the result of a relation identifier , which serves as an explicit signal to guarantee thepresence of relation constraints.
Step 2: Relation-Guided Top-K (line 5)
To satisfy lexical constraints, we replace the standard
Top- K operation by the one used in DBA [26].
Furthermore, to satisfy relation constraints, RESEALdynamically allocates beam by the results of relation identifier instead of the number of satisfiedconstraints used in DBA.
If removing line 4 and replacing line 5 with a normal Top- K , RESEAL is equivalent to the beamsearch.
We will describe the aforementioned two steps in detail in the rest of this section.
3.1 Probability Surgery
Algorithm 2 (line 1-7) shows the process of probability surgery.
At time step t during decoding,the model predicts the next token probability p vocab ( w | y <t , X ; θ ) .
In the scenario of RCD, whenpredicting next token w , we need to consider whether w is a head or tail in a relation constraint,which can be modeled with p rel ( w | y <t , C ) , the probability that w satisfies the relation constraints.
The core idea of our proposed probability surgery is to combine p vocab and p rel together to produce anaugmented distribution ˜ p , and then use ˜ p instead of p vocab to choose the words.
We first give a formal definition of p rel ( w | y <t , C ) .
The p rel depends on two factors: (1) the transitionprobability p trans ( y j , y i ) , the probability that y j is the head of y i , (2) the relation type probabilityp type ( y j , r, y i ) , the probability that the relation between y i and y j falls in type r .
Both p trans andp type can be obtained during decoding by a relation identifier . For dependency relations, the relation
Calculate and normalize
7: function
C
← do 11:
Allocate do 15: identifier can be a left-to-right dependency parser [7] to better fit the left-to-right manner of autoregressive decoding.
For relations between entities, the relation identifier can be any relation extractionmodel.
The function of relation identifier is to predict the head of y i , which produces a series ofp trans ( y j , y i ) .
Additionally, the relation identifier also predict the relation types, which produces aseries of p type ( y j , r, y i ) .
Note that if y j is not the head of y i , p type ( y j , r, y i )
=
for all types r .
Given the incomplete output sequence y <t
= ( y 1 , y 2 , ..., y t − 1 ) at time step t and the next token w ,we choose the relation constraints related to w in C .
Let C head ( w, t ) , C tail ( w, t ) denote the subset ofrelation constraints C at time step t where the w serves as the head or the tail, respectively:
[Equation]
We can calculate p rel ( w | y <t , C ) as follows 2 :
[Equation]
where Z w,t = 2[ | C head ( w, t ) | + | C tail ( w, t ) | ] is a normalizing factor.
Consequently, the augmenteddistribution can be calculated as follows:
[Equation]
where g : [0 , 1] → (0 , 1] is a gate function to transform p rel to a weight of p vocab .
We aim to increasethe weight when the p rel increases, so g cannot be a monotonically decreasing function.
Moreover,the output of this function should not be exactly zero, because assigning zero probability will makethe log-likelihood of the whole sentence be negative infinite.
Based on these requirements, inspiredby Schick et al.
[32], we adopt this parameterized form of g in this paper:
[Equation]
where λ, ρ are the hyperparameters and λ > 0 , ρ ∈ (0 , 1] .
λ controls the decay of output value.
ρ isa threshold for the probability.
We fix ρ = 0 .
5 in this paper.
If the p rel of a word w is greater thanor equal to this pre-specified threshold ρ , it is confident enough to consider w satisfies the relationconstraints.
Thus we set g ( p rel )
= 1 for this situation to preserve the p vocab of w .
On the contrary,when p rel is close to 0 , g ( p rel ) will be close to a relatively small value e − λ , which indicates that themodel will be less likely to choose the words violating relation constraints.
3.2 Relation-Guided Top-K
To ensure the presence of lexical constraints, we adopt the Top- K operation in DBA [26].
DBA firstlygenerates a set of candidates and then selects k of them through beam allocation.
The candidate setgenerated by DBA (line 8) is the union of three sets: (1) the normal Top-
K tokens, (2) all unsatisfiedlexical constraints, and (3) the single-best token for each hypothesis in the beam.
After that, DBAgroups together the candidates with the same number of satisfied lexical constraints into some banksand selects a different number of candidates from different banks.
The candidates with fewer lexicalconstraints will have more chances to be selected.
However, the original DBA is not aware ofthe relations between words.
Since we have already processed the candidate sentences by relationidentifier in probability surgery, we can now use the processed result to guide the bank allocation.
As illustrated in Algorithm 2, we propose to use the number of correct relation constraints of i -thcandidates (line 10-12) to divide the banks, rather than the number of satisfied lexical constraintsused by DBA.
This modification can jointly consider both lexical and relation constraints, becauseone relation constraint is equivalent to two lexical constraints and their relation.
Experiments on Dependency Placement
There are many kinds of word relations in natural language, so it is necessary to showcase theperformance of our proposed RESEAL on different relations.
Since syntactic dependency structuresserve as the principle of how words are combined to form sentences, dependency relations can bethe most basic and important for text generation.
Thus, in this section, we mainly focus on thedependency relation scenario of RCD, and evaluate RESEAL on the Dependency Placement task.
Besides, we will conduct extensive experiments on three downstream tasks in Section 5.
4.1 Task and Dataset
We first define Dependency Placement task: given the constraints C of dependency relations, output afluent sentence Y which appropriately places these constraints.
The model input X is optional, whichcan be a single ⟨ BOS ⟩ token, or a sequential transformation of C to provide necessary information.
We then construct the dataset for dependency placement task from the English-EWT [34] corpus 3 ,which contains 16,621 sentences with dependency annotations and standard train/dev/test set split.
For each sentence with m words, we randomly sample n dependency triplets { ( h i , r i , τ i ) } ni =1 as thegiven constrains C , where n < m .
The original sentences serve as references.
We refer to this datasetas English-EWT-Dep.
More details about English-EWT-Dep can be found in Appendix B.
4.2 Evaluation Metrics
In this section, we discuss the appropriate metrics for the dependency placement task to evaluate an
RCD algorithm (including but not limited to our proposed RESEAL).
Simply using the automaticevaluation to compare the system outputs with the references is not suitable, because there are toomany sentences that can be the correct answer given several dependencies.
We mainly focus onwhether the relation constraints are satisfied when examining an RCD algorithm.
To this end, theoutput sentences should be processed again by an accurate external parser 4 .
This parser shouldpreferably be different from the one used in an RCD algorithm, which can better examine thegeneralization ability across the parsers.
In this paper, we use two widely-used dependency parsersprovided by Stanza [27] and spaCy 5 for evaluation.
Let C ( i ) denote the relation constraints of
i -thoutput Y ( i ) in test set.
Let C ′
( Y ( i ) ) denote the dependency relation triplets obtained by the externalparser.
Let W ( i ) and W ′ ( Y ( i ) ) denote the sets if we omit the dependency relation r of C ( i ) and
C ′ ( Y ( i ) ) .
Similar to the unlabeled/labeled attachment score (UAS/LAS) used in dependency parsing,we can define the unlabeled/labeled coverage (UC/LC) as follows:
[Equation]
i
[Equation]
( i
[Equation]
[Table]
Moreover, we also report the BLEU [25] metric, the GPT-2 [28] perplexity (PPL) and word coverage(the proportion of lexical constraints that are satisfied) [Table] 4.3 Baselines
Since there are no existing work about dependency placement [Table] we design some straightforward baselines to compare with our method:
• Base :
Use BART large [18] as the backbone, and then funetune it English-EWT-Dep.
The inputis the concatenation of the triplets of C separated by special token # .
For example, if C ={ ( h 1 , r 1 , τ 1 ) , ( h 2 , r 2 , τ 2 ) } , the input sequence X = h 1 # r 1 # τ 1 , h 2 # r 2 # τ 2 .
The target outputis the reference.
During decoding, we use standard beam search with beam size k = 20 .
• CGMH [23]: Use MCMC sampling to generate a sentence by modifying it.
We use BERT large [4]as its replacement probability, and GPT-2 large [28] as its language model.• X-MCMC [10]: Improve CGMH by using XLNet [41].
X-MCMC-C adds a classifier to instructthe X-MCMC models where and how to modify the candidate sentences.• DBA [26]: Use DBA algorithm to decode on Base model.
Beam size k is set to 20 .•
DDBA [22]: A denoised variant of DBA by filtering noisy constraints.• NEUROLOGIC [21]: A LCD algorithm which support complex lexical constraints in Conjunctive Normal Form (CNF).
• Rerank : Preserve all k sentences generated by DBA , and select the sentence that satisfies themost relation constraints using left-to-right parser [7].
We report the results when applying RESEAL to the Base .
More details can be found in Appendix C.1.
4.4 Discussion
Results
Table 1 shows the results for dependency placement.
We find that the sampling-basedmethod [10, 23] can achieve better word coverage, but the low UC/LC scores demonstrate thatthey fail to enforce the relation constraints.
RESEAL achieves best UC/LC among all the methodswith significant decline of the PPL and competitive word coverage.
Specifically, RESEAL gainsan improvement of 1 . 15 on BLEU and 11 .
19% ( Stanza / spaCy ) on LC compared to DBA.
These results illustrate the weakness of existing LCD algorithms to correctly place the multiplerelation constraints.
In addition, though Rerank requires to generate many completed sentences, andthus is much more inefficient than RESEAL, our proposed RESEAL still outperforms Rerank onalmost all the metrics.
An intrinsic reason for this may be that RESEAL can dynamically adjust theword selection according to the parsing result during decoding, but Rerank can not make a choiceuntil finishing all the sentences.
[Table]
Ablation Study
Table 2 shows the results of ablation study.
We observe a decrease of 0 .
79 on
BLEU and 8 .
45% ( Stanza / spaCy ) on LC by removing probability surgery.
We also observea decrease of 0 .
73 on BLEU and 4 .
17% ( Stanza / spaCy ) on LC by removing RG-Top- K .
Probability surgery enables the tokens with correct dependencies to enter the candidate set.
RG-TopK dynamically allocates the beam according to the parsing result to satisfy more relation constraints.
These two components are both crucial.
Furthermore, we find that adopting another beam allocationstrategy (“word+rel” in Table 2) also hurt the performance.
Impact of Hyperparameters
For the dependency placement task, the performance would benefitfrom a larger beam size, which is consistent with the observations of Post and Vilar [26].
The left of
Figure 2 shows the labeled coverage (LC) as a function of beam size by different methods on the testset.
We observe that RESEAL can achieve better LC scores with smaller beam sizes when comparedwith Rerank .
Apart from that, the decay factor λ introduced in Section 3.1 is another importanthyperparameter of RESEAL.
We investigate its influence on the model performance.
Figure 2 showsthe results when varying λ from 0 to 10 based on the validation set.
If we set λ as a relatively largevalue ( > 4 ), the performance of RESEAL tends to be stable.
Specifically, λ = 0 is equivalent toremoving probability surgery (from Eq. 5), which will result in worse performance.
More Discussions
In Appendix D.1, we discuss the impact of the dependency parser.
In Appendix D.2, we discuss the time complexity of RESEAL.
In Appendix D.3, we show some generatedtexts for dependency placement task.
Experiments on Downstream Tasks
To further explore the effectiveness of our proposed RESEAL, we conduct experiments on the threedownstream tasks: sentence summarization, fact-based text editing, and data-to-text generation.
5.1 Sentence SummarizationDataset
We conduct experiments on English Gigaword dataset [31], which contains about 3.8Mtraining sentence pairs.
We use the validation and test set provided by Zhou et al.
[43] with 8,000 and
[Table]
2,000 sentence pairs, respectively.
Following previous work, we also evaluate our model on the testset of DUC2004 [24] (with 500 input sentences) and MSR-ATC [37] (with 785 input sentences).
Dependency Prediction
To apply RESEAL to sentence summarization, we first need to obtainreasonable dependency triples to construct relation constraints.
In this paper, we train a vanilla BERTbase-uncased [4] model to predict which dependencies should be present in the target output.
Firstly,we parse the sentences in the dataset by the left-to-right parser [7].
Then we use the intersection ofdependencies of source and target sentences as the ground truth to train the dependency predictor.
For each triplet ( h, r, τ ) , we concatenate the contextual embedding of h , τ and label embedding of rto perform binary classification.
More details can be found in Appendix C.2.
Results
Following previous work [15, 43], we report ROUGE F1 [20] on Gigaword and MSRATC, and ROUGE recall on DUC2004.
Table 3 show the results.
BART consistently outperformsthe previous models without pretraining across different datasets.
Over the strong BART baseline,
RESEAL can achieve better ROUGE scores on all datasets with predicted relation constraints.
Wealso investigate the upper bound of BART+RESEAL by using the gold relation constraints, whichshows that there is room for improvement with more accurate dependency predictors.
5.2 Fact-Based Text EditingDataset
Fact-based Text Editing is a novel task proposed by Iso et al.
Given some triplets(facts) from knowledge graphs and a draft text, this task aims to revise the draft text to contain thesefacts.
We adopt the WebEdit dataset provided by Iso et al.
[14], which contains 181K/23K/29Kinstances as train/valid/test set.
Note that this dataset can be viewed as a natural scenario for
RCD because both the relation constraints (facts) and model input (draft text) are provided.
Moreimportantly, based on the results of error analysis, the models trained on this dataset still suffer frommissing facts and incorrect relations (see “Qualitative evaluation” section in [14]).
RESEAL mayalleviate these problems by explicitly enforcing facts and relations.
Models and Results
For the relation identifier, we train a simple BiLSTM [11] encoder withbiaffine attention [6] on the training set of WebEdit (See Appendix C.3.1 for more details).
We use
EncDecEditor and FactEditor reported by Iso et al.
[14] as our baseline models.
EncDecEditor isan encoder-decoder model based on LSTM, with two separate encoders for facts and drafts and adecoder for generating revised texts.
FactEditor shares the same encoders with EncDecEditor buthas a novel decoder that doesn’t follow the conventional autoregressive decoding manner.
Thuswe can only apply RESEAL to the EncDecEditor.
For a fair comparison, we do not use pretrainedmodels and keep the same architecture setting as EncDecEditor.
However, the source code and thetraining details of EncDecEditor are unreleased, thus we reimplement EncDecEditor using our owntraining configuration (See Appendix C.3.2).
We denote this model as Seq2Seq.
Table 4 showsthe experiment results of WebEdit.
Following Iso et al.
[14], we report the BLEU and SARI [40]score (the average F1-score for keep, add and delete operations).
Owing to the difference of trainingsettings, Seq2seq can achieve significant improvement on BLEU (+7.28) and SARI (+1.54) comparedto FactEditor.
Based on this result, we further adopt RESEAL on Seq2Seq and then observe asubstantial improvement on BLEU (+1.16) and SARI (+4.59).
The above results demonstrate that
RESEAL can achieve better facts and relations preservation over the strong Seq2Seq baseline.
denote the metrics of “KEEP”, “ADD”, “DELETE”, respectively.
[Table]
[Table] [Table]
[Table]
5.3 Data-to-Text GenerationDataset
Data-to-text generation is another direct application for RESEAL.
In this paper, weadopt WebNLG dataset [9] which provides facts as inputs and sentences containing these facts asoutputs.
We use the data provided by Ribeiro et al.
[30] which contains 18,102/872/1,862 instancesas train/valid/test set.
Each test instance has 1-3 references.
Models and Results
For WebNLG dataset, the setting of the relation identifier is the same as thatfor WebEdit.
Table 5 shows the experiment results of WebNLG dataset.
We adopt our RESEAL on
T5 [29] and report the BLEU score for evaluation.
We observe an improvement of 0.53 BLEU on T5small and 0.42 BLEU on T5-base.
The detailed experimental settings can be found in Appendix C.4.
Related Work
Lexically Constrained Decoding (LCD)
Prior explorations for LCD can be summarized into fourcategories.
The first line of studies has proposed some model-agnostic methods which only modifythe decoding process.
They are independent from the training.
Hokamp and Liu [12] propose thegrid beam search (GBS) algorithm, a modification to beam search to impose the lexical constraints.
Post and Vilar [26] propose the dynamic beam allocation (DBA) algorithm with less time complexity.
Vectorized DBA [13] and Denoised DBA [22] are two different DBA variants.
The second line ofstudies requires some modifications to the training process.
Augmenting the training data with thelexical constraints is a general approach [3, 5, 35].
Another branch of previous works focuses onadding additional structure to the model [36, 38, 39].
The fourth line of studies applies Markov Chain
Monte Carlo (MCMC) to constrained text generation in a refinement manner [10, 23, 33].
Differentfrom these methods, we do not only focus on the isolated lexical constraints.
We propose to adoptrelation constraints to consider the relationship between words.
Dependency-Guided Generation
Dependency is a natural way to represent the syntactic orsemantic relations between words, so it can be used to guide the text generation.
There are few worksexploring this.
Filippova and Strube [8] propose to compress the dependency graph to guide thesentence fusion.
Akoury et al.
[1] propose to predict a chunked syntactic parse tree and then generatetokens conditioned on the parse.
Jin et al.
[15] encode the dependency relations by a graph encoderto improve sentence summarization.
Casas et al.
[2] propose a language model where the generationis driven by the expansion over the dependency parse tree.
Yang and Wan [42] propose a dependencymodeling objective to incorporate dependency knowledge.
However, one drawback of these methodsis the limitation of interpretability and controllability since they only use the dependency as a latentvariable during training, and cannot explicitly control the generation at the inference stage.
Conclusion
In this paper, we explore the Relation-Constrained Decoding (RCD), a new decoding scenario with amore complex definition of constraints.
We propose RESEAL, a novel algorithm for RCD, which canbe applied to the decoder of different models to preserve specific relation constraints.
We examine twodifferent experiment settings of RCD: dependency placement and downstream tasks.
For dependencyplacement, we construct the benchmark for dependency placement, and the experiment results showthe strength of RESEAL for satisfying relation constraints.
Furthermore, we apply RESEAL tothree downstream tasks as extended experiments for practical applications.
Extensive experimentsdemonstrate the effectiveness and universality of our method
Checklist
The checklist follows the references.
Please read the checklist guidelines carefully for information onhow to answer these questions.
For each question, change the default [TODO] to [Yes] , [No] , or[N/A] .
You are strongly encouraged to include a justification to your answer , either by referencingthe appropriate section of your paper or providing a brief inline description.
For example:
• Did you include the license to the code and datasets?
[Yes] See Section 4.
• Did you include the license to the code and datasets?
[No] The code and the data areproprietary.
• Did you include the license to the code and datasets?
[N/A]
Please do not modify the questions and only use the provided macros for your answers.
Note that the
Checklist section does not count towards the page limit.
In your paper, please delete this instructionsblock and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper’s
contributions and scope?
[Yes]
(b) Did you describe the limitations of your work?
[Yes] See Appendix D
(c) Did you discuss any potential negative societal impacts of your work?
[Yes] See Appendix E
(d) Have you read the ethics review guidelines and ensured that your paper conforms to
them?
[Yes]
If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results?
[N/A]
(b) Did you include complete proofs of all theoretical results?
[N/A]
If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi-
mental results (either in the supplemental material or as a URL)?
[Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how theywere chosen)?
[Yes] See Appendix C
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[N/A]
(d) Did you include the total amount of compute and the type of resources used (e.g., type
of GPUs, internal cluster, or cloud provider)?
[Yes] See Appendix C
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
[Yes] See Section 4 andSection
(b) Did you mention the license of the assets?
[Yes]
(c) Did you include any new assets either in the supplemental material or as a URL?
[Yes]
See Section 4 and Appendix B
(d) Did you discuss whether and how consent was obtained from people whose data you’re
using/curating?
[N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable
information or offensive content?
[Yes]
If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if
applicable?
[N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review
Board (IRB) approvals, if applicable?
[N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount
spent on participant compensation?
[N/A]
A More Explanations About RESEAL
Both C head ( w, t ) and C tail ( w, t ) can be empty set as C can be arbitrary.
Noted that in this part wesimply sum over all p trans and p type to produce p rel , which is different from the method we use whencombining p vocab and p rel .
The reason why we use this form is that if substituting Eq. 5 and 3 into
Eq. 4, we can derive a decomposed form:
[Equation]
Eq.
7 indicates that we assign high probability to w at time step t if and only if all the relationconstraints in C head ( w, t ) ∪ C tail ( w, t ) are satisfied.
Any violation of the constraints will result in asmall value of g ( p rel ( w | y <t , C )) and reduce the augmented probability ˜ p of w according to Ed. 4.
B Dataset Construction for English-EWT-Dep
In this section, we give more details about the data construction process of English-EWT-Depintroduced in Section 4.1.
We only consider the dependency relations whose modifier is an adjective,noun, or verb and randomly sample parts of them (at a ratio of 40% ).
The sentences which are shorterthan 4 words are omitted.
Some sentences containing personal information (e.g. phone number oraddress) are also discarded.
The statistics of the English-EWT-Dep dataset is as follows:
[Table]
C Implementation Details
[Table]
We provided the implementation details for the four tasks included in this work: dependencyplacement, sentence summarization, fact-based text editing and data-to-text generation.
Note that inthis paper, we conduct all the experiments on a NVIDIA A40 GPU with 40 GB memory.
C.1 Dependency Placement
For the Base model of dependency placement task, we use BART large [18] as our pretrained model.
We use label smoothing of 0 . 1 and optimize the model by Adam [16].
The learning rate is 2e-5 witha warmup rate of 0 . 1 .
The weight decay is 0 .
We train the Base model for 10 epochs with a batchsize of 32 .
C.2 Sentence Summarization
For the sentence summarization task, we also use BART large [18] as our pretrained model.
Thehyperparameter setting is similar to the dependency placement except that we only train for 5 epochswith a batch size of 100 .
The ROUGE evaluation option for GigaWord and MSR-ATC is -m -n 2 -w1.2.
The ROUGE evaluation option for DUC2004 is -m -b 75 -n 2 -w 1.2.
For the dependency predictor used in this task, we adopt a BERT-base-uncased [4] model.
Since thedependency is a word-level relation and BERT uses sub-word tokenization, we use the mean vectorof the start and the end token as the representation f ( w ) of a word w :
[Equation]
We use an additional embedding layer with the dimension of 200 to represent the dependency label r .
Let f L ( r ) denotes the representation of r , the feature of ( h, r, w ) is defined as the concatenation off ( h ) , f ( w ) and f L ( r ) .
We optimize the dependency prediction model by Adam [16].
The learning rate is 2e-5 with a warmuprate of 0 . 1 .
The weight decay is 0 .
We train this model for 10 epochs with a batch size of 200 .
Aftertraining, for each sentence in test set, we select the dependency relation constraints with predictedprobability larger than a threshold ρ = 0 .
It’s worth noting that our reported performance of sentence summarization can be further improvedby training a more accurate dependency predictor, or using a single keyword extractor and preservingthe corresponding dependencies of predicted keywords.
We leave more in-depth explorations asfuture work.
C.3 Fact-Based Text Editing
Different from sentence summarization task, fact-based text editing does not require to predict therelation constraints, because the relation triplets have been provided in the dataset.
Therefore, toadopt RESEAL, we only need to train the relation identifier and the LSTM [11] seq2seq model.
C.3.1 Relation Identifier for Fact-Based Text Editing
The structure of relation identifier for fact-based text editing is borrowed from the biaffine dependencyparser [6].
We use a single layer LSTM with single direction.
We use two separate MLPs with onehidden layer to extract the features of heads and tails, respectively.
Finally, we use a biaffine attentionmodule to classify the relation types.
Moreover, following the setting of Iso et al.
[14], we use asimple embedding lookup table to embedding the input words.
The detailed hyperparameter settingis shown as follows:
[Equation]
hidden [Equation]  MLP output size Number of relation categories
Maximum source length
Learning rate 1e
Dropout 0 .Batch size
Total
epochs
The training data for the relation identifier can be heuristically constructed from WebEdit dataset in adistant supervision manner, which is also the mainstream scheme for dataset construction of relationextraction models.
We consider the target texts (revised text) in WebEdit as the inputs, and thenuse the given facts to match the words in revised texts.
We omit the instances where the head wordand the tail word are not in the same sentence, because in this situation it is difficult to determinewhether these two words have given relations.
Furthermore, we construct many negative samplesfrom the positive samples by randomly deleting the words between the head words and the tail words.
The relations between the head words and the tail words in negative samples are annotated as “norelation”.
We add these negative samples into training set and test set to improve the robustness ofour relation identifier.
With the above mentioned model and augmented training data, we obtain a robust relation identifierfor WebEdit, which can achieve about 93 F1 score in the test set.
C.3.2 Training Configuration
We reimplement the EncDecEditor [14] and denote it as Seq2Seq.
The original paper only report thearchitecture of FactEditor [14].
Therefore, we have to make our Seq2Seq model have the same sizeas the FactEditor.
The detailed hyperparameter setting is shown as follows:
Embedding size
Table
[Table]
size
Text Encoder hidden size
Decoder hidden size
Maximum source length
Learning rate 1eTeacher forcing ratio 0 .
Dropout 0 .
Batch size
Total epochs
Note that we use the same vocabulary as the corresponding relation identifier.
Similar to EncDecEditorand FactEditor, our Seq2Seq model also employs attention and copy mechanism.
C.4 Data-to-Text Generation
In this part, we describe more details about the experiments of data-to-text generation task.
C.4.1 Entity and Relation Prediction
The settings of data-to-text generation is similar with that of fact-based text editing.
The majordifference is that the triplets given in WebNLG dataset cannot directly serve as relation constraints.
The entities given in the source inputs may not has the same form in the references.
Therefore, similarto the dependency prediction in sentence summarization task, we need to predict whether an entityshould appear in its output.
In this paper, we directly use the encoder of T5 [29] to predict how to preserve the entities.
Specifically,for all the triplets ( h, w, τ ) , we transform them into a sequential form.
Then we perform the sequencelabeling task to predict which entities should be preserved in the output.
This prediction task canbe optimized jointly with the standard seq2seq learning.
It must be pointed out that according toour experiment result, this prediction task itself does not influence the performance of data-to-textgeneration, so adding this task is not the reason for our performance gains.
We can obtain the entitiesand their corresponding relations and set them as relation constraints during the inference stage byadopting RESEAL, which achieves better relation preservation and enhance the performance.
C.4.2 Relation Identifier for Data-to-Text Generation
The relation identifier on WebNLG dataset shares the same architecture with the one used in factbased text editing.
The only difference is the vocabulary.
Due to the fact that we use T5 as ourbackbone for WebNLG, we adopt the tokenizer used in T5 to make the tokenizer consistent.
Thevocabulary size is 32,128.
C.4.3 Training Configuration
For the training settings for the WebNLG dataset, we train the T5-small and T5-base with a learningrate of 5e-5 and a batch size of 20 for 30 epochs.
We optimize the model by Adam [16].
No weightdecay and label smoothing are applied.
The maximum length for both source and target are 384.
D More Discussions about Dependency Placement
D.1 Impact of the Dependency Parser
In this work, there are two types of dependency parser.
The left-to-right parser [7] is used for thedecoding.
Stanza [27] and spaCy are used for the evaluation of algorithm.
For the parser which is used for the decoding, the result will benefit from a more accurate parser.
Forthe parser which is used for the evaluation, a more accurate one would increase the absolute values of
UC and LC.
For more precise evaluation, you can use multiple such parsers to produce an averageresult.
D.2 Time Complexity Analysis
Similar to most of decoding-only LCD algorithms, RESEAL is usually slower than the standard beamsearch, but the overall runtime is within an acceptable range.
The bottleneck of decoding speed is thetimes of decoder forward propagation.
For a sentence of length N , beam search and DBA take Ntimes forward.
Our method should be slower due to the relation identifier.
There are at most 2 | C |candidate sentences to be parsed at time step t .
Therefore, it will at most take N (1 + 2 | C | ) times offorward for RESEAL, which still has O ( N ) time complexity.
A faster implementation of RESEALis to save the outputs and hidden states obtained by the relation identifier at time step t , and reusethem at next time step t + 1 .
This implementation is left for the future work.
D.3 Case Study
Table 7 shows the sentences generated by all listed methods for dependency placement task.
Basemay omit some lexical constraints.
DBA and Rerank satisfy all the lexical constraints, but theycannot correctly handle the relations between them.
Rerank may generate sentences with repetitions(e.g.
the word “charge”).
RESEAL can produce sentences that are close to the grammatical structureof the references.
Furthermore, table 8 shows more examples for the dependency placement task.
[-The relation identifier reaches 93.
-]
[Table]
E Impact Statement
Our work has a developed generic decoding method for relation-constrained decoding.
Therefore,despite the contributes to better text generation, our proposed methods may be used to generatemore human-like fake text.
However, the impacts are more apparent when considering deployedapplications, while our proposed methods as the methodologies can not have any direct negativesocietal impacts
[Table]
[Table]
, ‘obl’]
Ref Ps I may have to come back to work for acall [‘come’, ‘call’, ‘obl’], [‘back’, ‘work’, ‘obl’]
Base
At the time , there were no clients in the facility and the staff rushed out of the building as soon as there was a problem .
[‘clients’, ‘facility’, ‘nmod’]
DBA
The clients of this facility were rushed out of the building by the time I got there and there was no one there to help me .
[‘clients’, ‘facility’, ‘nmod’], [‘rushed’, ‘time’, ‘obl’], [‘rushed’, ‘was’, ‘advcl’]
Rerank
The clients of this facility were rushed out of the building by the time I got there and there was no one there to help me .
[‘clients’, ‘facility’, ‘nmod’], [‘rushed’, ‘time’, ‘obl’], [‘rushed’, ‘was’, ‘advcl’]
RESEAL
When there was clients in facility , they rushed out of the building at the same time as the clients were leaving .
[‘was’, ‘clients’, ‘nsubj’,], [‘rushed’, ‘time’, ‘obl’, ], [‘rushed’, ‘was’, ‘advcl’], [‘clients’, ‘facility’, ‘nmod’]
Ref
Overpriced and the doctor acted arrogant and rushed at a time when there was very few clients in the facility .
[‘was’, ‘clients’, ‘nsubj’,], [‘rushed’, ‘time’, ‘obl’, ], [‘rushed’, ‘was’, ‘advcl’], [‘clients’, ‘facility’, ‘nmod’]