Transforming Multi-Conditioned Generation from Meaning Representation (2021)

Abstract: Our study focuses on language generation by considering various information representing the meaning of utterances as multiple conditions of generation. Generating an utterance from a Meaning representation (MR) usually passes two steps: sentence planning and surface realization. However, we propose a simple one-stage framework to generate utterances directly from MR. Our model is based on GPT2 and generates utterances with flat conditions on slot and value pairs, which does not need to determine the structure of the sentence. We evaluate several systems in the E2E dataset with 6 automatic metrics. Our system is a simple method, but it demonstrates comparable performance to previous systems in automated metrics. In addition, using only 10% of the dataset without any other techniques, our model achieves comparable performance, and shows the possibility of performing zero-shot generation and expanding to other datasets.

## introduction in many conversation systems, generating sentences with specific information is useful. for example, it can be used in chatbot systems or spoken dialogue systems to generate utterances that contain meaning representations (mrs) corresponding to a user's query. in order to train the nlg system that reflects this variety of information, a large amount of labeled data is required. at the 2017 e2e challenge @xcite , a large dataset was released, which consisted of pairs of mrs representing restaurant reviews and corresponding utterances. when generating sentences from an input (flat or structured mr), there are a template-based approach and a neural network-based approach. @xcite ; @xcite ; @xcite generate sentences based on template. the template method is to obtain the structural sets of sentences corresponding to mrs from training data and apply the template appropriate to the test data to generate the sentences. in @xcite ; @xcite , a template is formed based on rules, and @xcite learns the template structure of a sentence as a neural network. there is also a way to generate a natural language with a neural network without using a template. @xcite ; @xcite ; @xcite ; @xcite ; @xcite are sequence-to-sequence models of encoders and decoders, which have a one-stage framework, and @xcite is the model used by e2e challenge organizers. in @xcite , the encoder and decoder model converts flat mrs to structure mrs, and outputs the sentence through constrained decoding. template and neural network methods each have advantages and disadvantages. the template method guarantees a certain performance but limits the diversity and possibilities of the output. the neural network method generally performs better than the template method, but requires a lot of data and has limitations in the naturalness and semantical correctness of sentences @xcite . we propose a novel approach using the transformer decoder as a simple one-stage framework. in our model, gpt2-small @xcite is the backbone, and (s i , v i ) pairs of the meaning representation are put into multiple conditions to generate a sentence. in the previous works @xcite @xcite @xcite , when receiving the meaning representation as input, the value is delexicalized. specifically, all values corresponding to the same slot are delexicalized to a placeholder so that unseen inputs can be processed. however, @xcite ; @xcite report that delexicalization often leads to inappropriate behavior in scenarios. for example, the word "cheap" can be reflected in the utterance that matches the value of "less than $20". also, when food[italian] is given as slot @xcite , "italian food" is an appropriate phrase in a generated utterance, but in the case of food[fast food], "fast food food" is an incorrect phrase. additionally, the combination of eattype[coffee shop] and food @xcite is rather weird, and the combination of name[the rice boat] and area[riverside] is appropriate, so delexicalization doesn't fully utilize the characteristics of tokens. we, therefore, treat the slot as a special token and the value as a regular token of vocabulary without delexicalization in the training. nevertheless, in testing for unseen values, the model generates appropriate utterances and is described in section 4.3. our method is considered as generating a sentence as a simple one-stage framework directly from flat mrs. our model is tested on the e2e dataset. in addition to the evaluation metrics used in the e2e challenge, the systems are evaluated with bertscore @xcite . our approach shows the best performance in bleu, meteor, and bertscore and competitive performance in other metrics. by leveraging the pre-trained model gpt2, we quickly converge the model with only a few epochs and generate fluent utterances without considering the structure of the sentence. in addition, even if only 10% of the training data is used, it achieves performance comparable to previous systems. ## related work in many nlp tasks, the transformerbased @xcite models have recently shown good performance. gehrmann et al. (2018 ) is a previous work using the transformer encoder and decoder in the e2e task. bert @xcite composed only of the transformer encoder is used a lot in nlu tasks, and gpt @xcite composed only of the transformer decoder is used a lot in nlg tasks. these models are trained as large-scale open-domain corpora. by leveraging pre-trained models trained on large datasets and applying them to downstream tasks, many nlp tasks achieve better performance. the generation of utterances from mrs is quite similar to machine translation, one of the sequenceto-sequence tasks. also, in terms of generating sentences with certain restrictions, it is similar to style transfer @xcite @xcite , which is one of sequence+condition-to-sequence tasks. however, the generation of utterances from mrs is not exactly the same as the above tasks because of the condition-to-sequence perspective. since the e2e task does not have a given sequence as the input of the model, we approach the sentence generation using only the decoder without sequence encoding. we choose a language model of the transformer decoder that performs better than lstm and uses the pre-trained model gpt2 as a backbone. ## experiments we use the model provided by huggingface 1 to make it easy to use the pre-trained gpt2 trained by openai. ## conclusion this paper presents a simple one-stage approach to generating natural utterances from flat mrs. our system uses a pre-trained model language model to improve the performance of the system and shows that it is better than the system that won the challenge in human evaluation. even if our model is trained with only a small amount of sampling data due to the leveraging effect, it is comparable to the previous models. our approach is simple, efficient, easy to extend to multiple mrs (i.e. webnlg), and enables zero-shot generation without additional data through sim-delexicalization.