Does the Order of Training Samples Matter? Improving Neural Data-to-Text Generation with Curriculum Learning (2021)

Abstract: Recent advancements in data-to-text generation largely take on the form of neural end-to-end systems. Efforts have been dedicated to improving text generation systems by changing the order of training samples in a process known as curriculum learning. Past research on sequence-to-sequence learning showed that curriculum learning helps to improve both the performance and convergence speed. In this work, we delve into the same idea surrounding the training samples consisting of structured data and text pairs, where at each update, the curriculum framework selects training samples based on the model’s competence. Specifically, we experiment with various difficulty metrics and put forward a soft edit distance metric for ranking training samples. On our benchmarks, it shows faster convergence speed where training time is reduced by 38.7% and performance is boosted by 4.84 BLEU.

## introduction neural data-to-text generation has been the subject of much recent research. the task aims at transforming source-side structured data into targetside natural language descriptions @xcite . the process typically involves mini-matches which are randomly sampled with a fixed size from the training set to feed into the model at each training step. in this paper, we apply curriculum learning to this process, which was explored in neural machine translation @xcite , and show how it can help in neural data-to-text generation. the main idea in curriculum learning is to present the training data in a specific order, starting from easy examples and moving on to more difficult ones, as the learner becomes more competent. when starting out with easier instances, the risk of getting stuck in local optima early on in training is reduced, since the loss functions in neural models are typically highly non-convex @xcite . this learning paradigm enables flexible batch configurations by considering the material properties as well as the state of the learner. the idea brings in two potential benefits: (1) it speeds up the convergence and reduces the computational cost. (2) it boosts the model performance, without having to change the model or add data. with the release of large data-to-text datasets (e.g. wikibio @xcite , totto @xcite , e2e @xcite ), neural data-to-text generation is now at a point where training speed and the order of samples may begin to make a real difference. we here show the efficacy of curriculum learning with a general lstm-based sequence-to-sequence model and define difficulty metrics that can assess the training instances, using a sucessful competence function which estimates the model capability during training. such metrics have not yet been explored in neural data-to-text generation. in this paper, we explore the effectiveness of various difficulty metrics and propose a soft edit distance metric, which leads to substantial improvements over other metrics. crucially, we observe that difficulty metrics that consider data-text samples jointly lead to stronger improvements than metrics that consider text or data samples alone. in summary, this work makes the following contributions towards neural data-to-text generation: ## related work the idea of teaching algorithms in a similar manner as humans, incrementally from easy concepts to more difficult ones dates back to incremental learning, which was discussed in light of theories of cognitive development relating to the processes of acquisition in young children @xcite @xcite . @xcite first demonstrated empirically that curriculum learning approaches can decrease training times and improve generalization; later approach address these issues by changing the minibatch sampling strategy to also include model competence @xcite @xcite @xcite . while sample difficulty can be assessed for text samples and data samples or jointly, various measures have been proposed for text samples including n-gram frequency haffari (2009); platanios et al. (2019), token rarity, and sentence length @xcite . our approach considers data and text jointly, similar to edit distance metric -levenshtein (levenshtein, 1966) and @xcite , which was used as a content ordering metric in @xcite to measure the extent of alignment between data slots and text tokens. ## difficulty metrics for ease of discussion, we denote sequence to be s, which can be either data or text, or their concatenation. for comparison, the difficulty metrics use the unit tokens as tokenized by spacy foot_0 . we begin with discussion on length and word rarity, which were previously applied by @xcite ; platanios et al. ( 2019 ) on text sentences. ## results & analysis on human evaluation. for human evaluation, three annotators are instructed to evaluate 100 samples from the joint variant to see (1) if the text is fluent (score 0-5 with 5 being fully fluent), (2) if it misses information contained in the source data and ( 3 ) if it includes wrong information. these scores are averaged and presented in ## conclusion to conclude, we show that the sample order does indeed matter when taking into account model competence during training. further, we demonstrate that the proposed metrics are effective in speeding up model convergence. given that curriculum learning can be combined with pretty much any neural architecture, we recommend the use of curriculum learning for data-to-text generation. we believe this work offers insights into the annotation process of data with text labels where reduced number of labels are needed.