A Pipeline for Extracting Abstract Dependency Templates for Data-to-Text Natural Language Generation (2023)

Abstract: We present work in progress that aims to address the coverage issue faced by rule-based text generators. We propose a pipeline for extracting abstract dependency template (predicate-argument structures) from Wikipedia text to be used as input for generating text from structured data with the FORGe system. The pipeline comprises three main components: (i) candidate sentence retrieval, (ii) clause extraction, ranking and selection, and (iii) conversion to predicate-argument form. We present an approach and preliminary evaluation for the ranking and selection module.

## introduction rule-based natural language generation (nlg) systems have become increasingly unpopular since the nlp field switched first to statistical systems, then to neural: rule-based systems tend to have low coverage (limited robustness to new inputs), reduced suprasentential fluency, and on the whole need to be built manually, all of which in combination means they are no longer competitive in shared task competitions and other nlp research contexts. however, their output can generally be guaranteed to have high accuracy and grammaticality, which continues to make them the system of choice in many commercial contexts. 1 moreover, they can 1 e.g. arria nlg's nlg engine. be efficient in terms of data and energy requirements, and suitable for low-resource languages. that is, on their own or in combination with, e.g., language-model-based modules, rule-based nlg potentially has an important role to play in the current nlp landscape if shortcomings such as the coverage issue addressed here can be overcome. webnlg+. the present work was prompted by the webnlg+ shared task @xcite , in which part of the test set inputs contained features not seen in the training or development data. the webnlg+ dataset is a benchmark for data-to-text nlg consisting of aligned dbpedia triple sets and texts. dbpedia triples are the building blocks of the inputs, and consist of three related elements called a property, a subject and an object in semantic web terminology. a subject (denoted by db-subj in this paper) is usually an entity that has a property and a value for this property, which is the object (db-obj). e.g. in figure 1 , the entity audi_a1 is associated with 4 properties: engine, transmission, assembly and bodystyle. the semantics of each property is defined by dbpedia editors, foot_0 but in most cases, the property of the db-subj is db-obj makes it clear (e.g., the transmission of the audi_a1 is 5-speed manual). the coverage issue. unlike their neural counterparts, rule-based generators submitted to the webnlg+ challenge such as rdfjsrealb (lapalme, 2020), dangnt-sgu (tran and nguyen, 2020) or forge @xcite are not able to cope with new (previously unseen) properties. forge, which we are aiming to extend, operates on dependency structures at several levels of representation (syntax, semantics), and needs partially lexicalised predicate-argument (predarg) structures in the propbank style @xcite to use as input for generation (see figure 2b ). in other words, if a mapping between property and predarg structure as shown in figure 2a-b does not exist, the generator cannot introduce the appropriate words and, unless a backup mechanism is in place, it will fail to generate a text. thus, the overall problem that we are tackling is the following: given (i) the rule-based forge generator that covers all properties in the webnlg+ training data, (ii) a file which contains the mappings between these properties and their respective predarg template, (iii) an input triple set that contains one or more properties not currently covered by the generator, automatically extend the mapping file in ii with new unseen property/template pairs that will enable forge to generate a text that verbalises all input properties. proposed solution. our aim is the automatic extraction of property/template pairs via a pipeline for retrieving and ranking candidate clauses from wikipedia that correspond to a given dbpedia instantiated property (i.e. a triple), and converting them to predicate-argument representations. we are at an early stage of this research: the pipeline and components have been defined and connected, and we have identified two main challenges in our approach: one is candidate clause extraction, i.e. how to find a sentence or a clause that exactly matches the input triple, the other is the identification of such candidates, i.e. if provided with a list of candidates clauses that contains a match, is it possible to identify it. in this paper, we focus on the second challenge, since if we are not able to identify target candidates, the approach cannot work. in the remainder of the paper, we present the different components and resources used in our pipeline, and provide an encouraging preliminary quantitative and qualitative evaluation of a transformer-based candidate ranking and selection component. 3 2 related work a number of papers have tackled the extraction of templates from text to be used as input for nlg. @xcite mine and prune sentence templates from wikipedia articles, but (i) extract templates given an entity (instead of a property as in our case), and (ii) manage to obtain a template for about 20% of the target entities. @xcite achieve impressive coverage with their (multi-property) sentence templates, but also suffer accuracy problems, with the text faithfully representing the input in only about half the cases. our general approach is conceptually similar to @xcite 's, who use an open information extraction (ie) tool to identify candidate sentence spans that verbalise a given property, and then acquire lexicalisation information via verbnet, resorting to default strategies when a predicate is not covered by verbnet. @xcite suggest several general approaches to align triple components and textual elements, namely string, substring, hypernym and synonym matching; for property matching, they also use a pre-trained vector model to calculate the distance between words. other recent work on this topic uses keyword matching @xcite or cosine similarity @xcite for aligning triples and text in under-resourced languages. in order to assess the strength of the alignment, @xcite apply a natural language inference (nli) model to detect (lack of) entailment between the triples and the candidate sentences. one difference between our approach and most of the related work on template mining for nlg is that we want to extract predicate-argument templates (figure 2b ), and not full-sentence templates. however, the approaches have a lot in common, since we extract the predicate-argument structures from sentences. the main issue with most of the approaches above is the lack of accuracy. recently, transformers have been shown to improve accuracy for question-answering @xcite , including for the specific task of aligning text and structured data @xcite and also for fact checking, for instance for comparing tables and text @xcite . in our approach, we therefore explore another way of aligning linguistic predicates and properties via transformer-based meaning similarity scoring. ## template extraction pipeline in this section, we describe the components that allows us to extract one or more predarg template(s) given one input dbpedia triple. 4 figure 3 shows a complete view of the pipeline (see appendix a for module outputs). the three main components of the pipeline (indicated across the bottom in figure 3 ) are: (i) candidate sentence retrieval, (ii) candidate clause ranking and selection, (iii) conversion to predicate-argument template. ## experiments and preliminary results in this paper, we provide a first evaluation of the ranking component; we believe that there are many ways of finding more candidate sentences (see section 5), but predicting which candidate is suitable (or not) is particularly crucial in our pipeline. 8 see appendix b for details on the data and fine-tuning. evaluation setup. for the evaluation, we compare two models, the off-the-shelf transformer @xcite and our fine-tuned version of it, on two datasets, (a) the webnlg+ development subset of single-property inputs (401 triples), and (b) the subset of the webnlg+ test set comprising all and only items with properties not seen in the webnlg+ training data (113 triples). the objective is to obtain performance upper and lower bounds for the fine-tuned model by examining how accurate it is at selecting the right candidate (a) for properties seen during fine-tuning, and (b) for unseen properties, which is the most realistic scenario for predarg template extraction. for each input triple, there are 1 to 3 exactly matching sentences (the corresponding reference sentences in the webnlg+ dataset), which are the target sentences that we want the model to prefer (rank highest) for the input triple. for use as the nonmatching candidates, which should be dispreferred (ranked lower) by the model, we select all other sentences that verbalise one-triple inputs, and all sentences that verbalise two-triple inputs; the total dev and test candidate pools contain 1,834 and 2,887 sentences respectively. this way, we ensure that we have candidates with a significant meaning overlap with the target sentences (one-triple inputs can share elements with one another, see section 3.2, and two-triple inputs can include elements or even full triples of the one-triple inputs). results. on the development data (top half of to assess to what extend the models capture the semantics of the properties, we repeated the experiment above but modifying the input triples in two ways: replacing the property name by another randomly selected property (avg. top p m od ), and inverting the db-subj and db-obj (avg. top p invso ). the off-the-shelf model has a harder time discriminating between correct and wrong properties than the fine-tuned model (similarity scores of 0.785 and 0.684, respectively, for the off-the-shelf model, 0.963 and 0.754 for the fine-tuned model). however, neither of the models is able to discriminate cases where the db-subj and db-obj are switched, yielding even higher scores on average than with the original triple (avg. top p invso ). we then looked for the threshold at which a model gets the best f1 score when selecting a candidate sentence. we tested all thresholds (in steps of 0.01 from 0 to 1) for each model on the dev set and obtained values of 0.73 and 0.87 for the offthe-shelf and fine-tuned models respectively, which yield a f1 of 0.798 and 0.955 respectively. on the unseen test set, these thresholds yield a significantly lower f1 score, the fine-tuned model reaching an f1 of only 0.694 and the off-the-shelf model 0.429. note that a better f1 can be achieved on these unseen triples by selecting different thresholds (both higher, at 0.93 and 0.78 respectively). 9 error analysis. we examined all the false positives and false negatives for the best threshold on the dev set (0.87), and found the following errors. 10 false positives (53 errors): (i) a sentence that corresponds to 2 triples was selected, because one or more elements of the second triple are very similar with the input triple's db-subj, db-obj or property (75% of errors); (ii) the selected sentence verbalises a triple that is almost identical to the input triple (25%). false negatives (35 errors): (i) mismatch between a db-subj, property or db-obj and their corresponding verbalisation due to an accent, a comma in a number, quotation marks, parentheses, casing (57%); (ii) a triple element is verbalised with a word judged semantically distant (29%); (iii) a reference sentence is wrong (14%). only false negatives (i) and (iii) in the stem from errors or lack of normalisation in the data; the other errors are due to the model. discussion. we were surprised by the decrease in the score between the dev and the test sets, especially for the off-the-shelf transformer, for which we would expect no difference between seen and unseen properties. we hypothesise that the test set is more challenging: (i) the reference sentences seem less similar (0.910 on test vs 0.932 on dev when running the off-the-shelf transformer on the gold sentences for triples of size 1); (ii) some problematic cases are more frequent (e.g. the db-subj or db-obj has content in parentheses in 34% of 9 fig. 6 and 7 in appendix c show the f1/threshold plots. 10 see tables 3 to 8 in appendix d for examples. the test triples, vs 12% in the dev set); (iii) there are more candidate sentences for the test set (see evaluation setup). there are likely other factors. all properties of dev. set (401 triples) off-the-shelf fine-tuned accuracy all (%) 91.02 98.50 accuracy one (%) 98.25 99.50 avg. top p ok 0.785 0.963 margin 0.170 0.346 avg. top p m od 0.684 0.754 avg. top p invso 0.803 0.971 f1 (thresh.) 0.798 (0.73) 0.955 (0.87) unseen porperties of test set (113 triples) off-the-shelf fine-tuned accuracy all (%) 56.64 73.45 accuracy one (%) 87.61 96.46 avg. top p ok 0.787 0.929 margin 0.110 0.212 avg. top p m od 0.702 0.776 avg. top p invso 0.815 0.952 f1 dev thresh. 0.429 0.694 f1 (best thresh.) 0.537 (0.78) 0.745 (0.93) ## future work we are currently developing the approach reported here further, including investigating how to increase the f1 for candidate selection on unseen data, for instance by using cross-encoders for the final ranking of the top candidates or nli to filter out bad candidates @xcite . to find more and better candidates, we will apply coreference resolution on the wikipedia pages, test open ie approaches to identify text spans @xcite , and explore the use of simple wikipedia @xcite and wexea @xcite . we will further develop our prototype clause extractor, and will apply our approach to other languages to test its portability. if the clause is above the defined threshold, it is selected for the template. see more examples of ranking and selection in appendix d.