T emp LM : Distilling Language Models into Template-Based Generators (2023)

Abstract: While pretrained language models (PLMs) have greatly improved text generation, they have also been known to produce unfaithful or inappropriate content. In contrast, classic template-based systems provide strong guarantees of faithfulness at the cost of fluency. We propose TempLM, which achieves the best of both worlds by distilling a PLM into a template-based generator. On the E2E and SynthBio data-to-text datasets, we show that TempLM is more faithful than the original PLM and is more fluent than prior template systems. Notably, on an out-of-domain evaluation, TempLM reduces a finetuned BART model’s unfaithfulness rate from 83% to 0%. In a human study, we find that TempLM’s templates substantially improve upon human-written ones in BERTScore.

## introduction pretrained language models (plms; @xcite can generate fluent text and are data-efficient when being transferred to downstream tasks @xcite . however, plms have been known to produce unfaithful outputs @xcite and inappropriate content @xcite that can lead to disastrous outcomes in real-world deployments @xcite . these errors can be worsened when models are queried with out-ofdomain (ood) input. figure 1 shows that querying a finetuned plm with a novel entity (e.g. starbucks) not in the training data can lead to surprising failures even though the plm achieves high indomain performance. this poses a great challenge in deploying plms in real-world applications. in stark contrast, classic template-based systems @xcite @xcite employ templates consisting of words and nonterminal fields, which are robust to novel entities by design. moreover, templates are directly readable by humans, and human in domain : plm generates high-quality output name aromi food chinese near the crown plaza hotel area city centre input data output text aromi is a chinese restaurant near the crown plaza hotel in the city centre. out of domain : plm produces unfaithful output name starbucks food chinese near the crown plaza hotel input data area city centre output text the chinese restaurant, is located in the city centre. the crown plaza hotel, inspection can provide direct guarantees of faithfulness. however, templates can be too rigid and produce disfluent text with unexpected inputs. in this work, we seek to borrow the merits of classic template-based techniques to improve faithfulness and interpretability, while retaining the plm's flexibility and data efficiency. we propose templm, a novel framework that distills a plm into a template-based system for data-to-text tasks. at training time, templm extracts templates that maximally recover the induced probability distribution of the plm, similar to model distillation @xcite . at inference time, templm uses the plm to select appropriate data (content selection) and templates (surface realization). while distilling a plm into a template-based generator brings benefits, it also raises new challenges. extracting templates that match a plm's probability distribution is a challenging combinatorial optimization problem with no clear solution. our approach relies on two new ideas. first, because our goal is to recover the plm's induced probability distribution, templm initializes its search procedure by delexicalizing plm's genera-tion outputs, i.e. abstracting the value in the output with data fields. for example, we can delexicalize "aromi is a chinese restaurant" into "[name] is a [food] restaurant." second, templm leverages the plm's generation ability to refine templates, using a novel consensus beam search algorithm. unlike prior works @xcite , our approach can leverage any plm to generate templates, allowing us to take advantage of improvements in the data efficiency and fluency of plms. we evaluate templm on the e2e @xcite and the synthbio datasets @xcite . we observe that templm is the most faithful generation method (with zero faithfulness errors) on the e2e in-domain test set. furthermore, templm fixes the unreliable ood behavior of plms, reducing the unfaithful output rate from 83% to 0%. in addition, we show that templm achieves higher metric scores than classic text generation techniques and a previous hybrid neuraltemplate method (5 bleu scores higher than @xcite even when trained with 42 times less data). we further conduct a human study where we ask annotators to write templates for synthbio with a time constraint. we observe that templm produces more fluent templates than both the average template writer and an ensemble aggregating all the template writers. ## experiments we evaluate templm's ability to generate faithful and fluent text in three settings: an in-domain evaluation on standard data-to-text benchmarks, an out-of-domain evaluation that stress tests the ability to generalize to novel inputs, and a human study comparing templm's template extraction ability to that of human template writers. ## conclusion and future work we propose templm, a novel framework for distilling plms into template-based systems. templm is designed to achieve better robustness and interpretability while inheriting the fluency and data efficiency of plms. our evaluations show that templm can completely eliminate the unfaithful outputs produced by a finetuned bart model for out-of-domain inputs. on in-domain evaluation, templm is able to produce more fluent outputs compared to classic template systems, prior neuralhybrid template methods, and even human template writers. in the future, we look forward to extending the templm framework to learn compositional templates and grammars, as well as improving its coverage to diverse outputs, potentially via paraphrases of its input data.