System Description for the C ommon G en task with the POINTER model (2021)

Abstract: In a current experiment we were testing CommonGen dataset for structure-to-text task from GEM living benchmark with the constraint based POINTER model. POINTER represents a hybrid architecture, combining insertion-based and transformer paradigms, predicting the token and the insertion position at the same time. The text is therefore generated gradually in a parallel non-autoregressive manner, given the set of keywords. The pretrained model was fine-tuned on a training split of the CommonGen dataset and the generation result was compared to the validation and challenge splits. The received metrics outputs, which measure lexical equivalence, semantic similarity and diversity, are discussed in details in a present system description.

## introduction the 2021 edition of the generation evaluation and metrics (gem) challenge for the creation of living nlg benchmark leaderboard @xcite , comprised four groups of tasks -summarization, structure-to-text, simplification and dialog. the commongen dataset makes part of the structure-to-text group and was designed to measure a common sense reasoning capacities of generative models given a set of concepts @xcite . due to the nature of the constraint based text generation of the pointer model @xcite and resemblance in a generation strategy (the model takes a set of keywords as an input and generates a text, containing these keywords) the commongen dataset for hard constrained generation of the gem benchmark appears to be a good fit for testing the model performance. the pretrained pointer model was therefore fine-tuned on a training set of the commongen dataset and the inference results were compared to the validation and challenge splits of the same dataset. 2 2 data description and pre-processing the insertion-based transformer architecture leverage implies the use of the masking mechanism, the goal of which is to predict not only the likelihood of a token itself, but the likelihood of the token insertion between two given tokens, in other words, we need to predict the word and the place where a new word is inserted. in that regard, a text is preprocessed in a specific way, where the tokens are scored using a combination of three schemes of the token importance measurement (term frequency-inverse document frequency (tf-idf), part-of-speech (pos) tagging and yet-another-keyword-extractor (yake)) and the highest scored tokens are replaced with a special noinsertion token [noi] tag. this procedure is iterative and results in generation of several utterances out of the initial sentence. during the training phase, the model is initialised with the multilingual bert and its vobabulary is extended with the [noi] tag. at the inference time, the masking mechanism is used in a reverse order, allowing an iterative tokens prediction -the model will chose to either generate a token or a [noi] tag at a given generation stage and if the next stage contains @xcite tag predictions only, the generation is finished. the model was pre-trained on 12gb of wikipedia corpora, therefore the pre-training data consisted of a well written english with the correct spelling, grammar and punctuation. for the finetuning, the sentences from the training split were preprocessed with the pre-training data generation script, 3 which inserts the token position masks in a gradual manner, resulting in a data augmentation from 67.389 source entries to 160.680 processed entries. ## training details and decoding strategy the fine-tuning was done on 8 cores (16gb of ram each) of a tpu-v3 device, following the multiprocessing paradigm, and took three hours to train on 40 epochs with the batch size equal to 64 and gradient accumulation equal to 2. the finetuning hyperparameters were preserved from the original paper and included adamw optimizer, learning rate equal to 1e-5, adam epsilon equal to 1e-8, 10 warmup optimizer scheduler steps and the seed equal to 1. the inference of the finetuned model was done using the concept sets from the validation and challenge splits of the commongen dataset. the decoding strategy included two sampling methods, applied separately -greedy and sampling. the greedy decoding is based on a greedy search algorithm, which consists of choosing the highest scoring token at a given time step, along with the temperature @xcite , while sampling uses a combination of top-k @xcite , top-p @xcite and the temperature parameters to render model predictions. for the greedy decoding method, a temperature, which is a scale factor of each token's probability before going through softmax function, was set to its lower value 0.3, ensuring the most stable generations. this parameter alone draws a limit on the model's creativity, resulting in a more rigid generation. for the sampling decoding method, the parameters promoting a high creativity of the model were chosen: the top-k window of the most probable tokens was set to 10, following the strategy expressed in the original paper @xcite , the top-p cumulative probability threshold for the most probable tokens was set to its highest tested value 0.95, according to the original paper @xcite , and the temperature was set to 0.9 -this is the highest lower probability threshould for this sampling parameter, allowing the maximum tokens pass-though without giving up stability of the text generation. other parameters were common for both sampling methods and included noi decay and reduce decay, which were equal to 1, and prevent, reduce stop, lessrepeat, which were set to true. the inference for both decoding methods was done with the maximum sequence length description content keys val. olympic athlete then brings in the tennis ball straight back up down on the tennis court. sampling val. olympic athlete quickly moves toward the soccer ball about halfway way up on the clay court. target val. the boy must run from one end of the court to the other to throw the ball into the hoop. ## metrics outputs before diving in the metrics output results, let us explore a few examples of the generated text. 4 the since the goal of gem challenge is an in-depth analysis of the model performance regarding lexical, semantic similarity and language richness, we will divide the analysis in separate subsections. ## conclusions the system description depicted the experiment on application of the commongen task from the gem benchmark to a hard constraint text generation with the insertion based transformer. the use of eleven automated metrics for measuring the generative performance of the pointer model allowed to detect the issues of the model output and reveal the advantages of a specific decoding method. for the lexical equivalence, meteor metric seems to be the most relevant (since it takes stemmed forms of the words and makes the synonym comparison), when looking at the score augmentation for more creative text generations, accomplished with the sampling decoding method. the semantic similarity measured with the bertscore and bleurt neural based metrics showed that both validation and challenge splits result in a semantically equivalent text generations, with a small difference between decoding methods, while the application of nubia metric with a refined semantic relevance parameter resulted in a better score for the validation split. the entropy showed the noisiness of the generated text for both decoding methods, and the distinct score showed an unexpected boost for the greedy decoding, which means less words' repetitions than for the sampling decoding. finally, the unique score showed that sampling decoding