Grouped-Attention for Content-Selection and Content-Plan Generation (2021)

Abstract: Content-planning is an essential part of data-to-text generation to determine the order of data mentioned in generated texts. Recent neural data-to-text generation models employ Pointer Networks to explicitly learn content-plan given a set of attributes as input. They use LSTM to encode the input, which assumes a sequential relationship in the input. This may be sub-optimal to encode a set of attributes, where the attributes have a composite structure: the attributes are disordered while each attribute value is an ordered list of tokens. We handle this problem by proposing a neural content-planner that can capture both local and global contexts of such a structure. Specifically, we propose a novel attention mechanism called GSC-attention. A key component of the GSC-attention is grouped-attention, which is token-level attention constrained within each input attribute that enables our proposed model captures both local and global context. Moreover, our content-planner explicitly learns content-selection, which is integrated into the content-planner to select the most important data to be included in the generated text via an attention masking procedure. Experimental results show that our model outperforms the competitors by 4.92%, 4.70%, and 16.56% in terms of Damerau-Levenshtein Distance scores on three real-world datasets.

## introduction data-to-text generation @xcite is an important and challenging task in natural language processing. it aims to produce sentences given structured data. there are many downstream applications of data-to-text-generation, such as biography summarization @xcite , automatic weather forecasting @xcite , etc. traditional approaches follow a pipeline framework consisting of three stages: content-selection, content-planning, and surface-realization. contentselection selects the data to be expressed; contentplanning determines the structure of the sentences ## related work content-planning is an essential part of data-to-text generation to determine the order of data mentioned in generated sentences. in early data-to-text generation approaches, content-planning is done by creating hand-crafted rules @xcite , using template-based models @xcite , or exploiting machine learning models @xcite @xcite . content-planning is coupled with content-selection and surface-realization. contentselection mainly relies on hand-built heuristics @xcite and shallow statistical machine learning models @xcite @xcite . for surface-realization, earlier studies exploit template-based models @xcite and language models @xcite . as end-to-end training is becoming prevalent @xcite , recent data-totext generation approaches employ neural networks which can be trained end-to-end @xcite . these approaches use encoder-decoder frameworks @xcite @xcite . the encoder is used to transform the input into some vector representation. the decoder takes the vector representation as context to generate the target sentence. in both the encoder and the decoder, sequence neural network models, such as lstm @xcite and transformer @xcite , are used to process the data. studies in this line of work include biography summarization @xcite , weather forecasting @xcite , and game summarization @xcite . despite their success, these models may generate incoherent sentences since they do not have a proper content-plan. to improve the coherence of the generated text, recent studies re-introduce content-planning mechanism in neural approaches of data-to-text generation. @xcite propose a content-planning mechanism via link-based attention to model relationships among input data. it learns a matrix where each element indicates transition probability of two attributes. goldfarb-tarrant et al. ( 2020) the aforementioned models use contentplanning mechanisms with lstm or shallow trans-formation networks as the input encoder. however, lstm is sub-optimal to capture the relationships in the input. this is because the input is a set (e.g., a set of attributes), and each attributes may consist of multiple tokens, which form a hierarchical structure among the attributes instead of a sequence. thus, applying lstm on the input data may lead to improper representations of the input. in this paper, we propose a novel content-planner to capture the hierarchical structure of the input. ## preliminary let @xmath0, a 2 , ..., a n } be a set of n attributes of an entity. each attribute a @xmath1; v i is a pair of a key k i and a value v i (@xmath2, 2, ..., n). key k i denotes the type of an attribute. value the content-plan of a sentence specifies which attributes are selected to be included in the generated sentence and their order in the sentence. it consists of a sequence of tokens, where each token belongs to one of attributes. the contentplan can be represented by a sequence of index pairs @xmath3, j 1 , i 2 , j 2 , ..., i c , j c ]. here, i k indicates the index of an attribute, and j k indicates the index of a token within the attribute (@xmath4, 2, ..., c). given a set of attributes a, we aim to generate the content-plan p. note that our goal is not to generate a sentence given a set of attributes. our work aims to organize the attributes in a way that enables generating better (i.e., non-repetitive and coherent) sentences for downstream textual generation models. ## proposed model solution overview. we propose a novel end-toend model for content-plan generation. the model consists of four modules: an embedding and linear transformation module, an attribute-encoding module, a content-selection module, and a contentplanning module. the embedding and linear transformation module (cf. section 4.1) aims to obtain initial vector representations of attributes and tokens. the representations should maintain the order of tokens within an attribute and should not impose any order on the set of attributes. the attribute-encoding module (cf. section 4.2) aims to encode the attributes of an entity into fixedlength embeddings, which will be used by the content-selection and the content-planning modules as inputs. the attributes of an entity have a composite structure: the attributes are disordered while each attribute value is an ordered list of tokens. to encode such a structure, we propose a novel attention mechanism capable of learning a representation for each token of an attribute and the attribute as a whole. the content-selection module (cf. section 4.3) aims to accurately predict both content-selection labels and attribute-selection labels. the contentselection labels help highlight the selected attributes, whereas the attribute-selection labels provide more supervision signals to train parameters of the content-selection module. the content-planning module (cf. section 4.4) integrates the content-selection into the pointer networks to generate a content-plan. ## conclusions and future works we presented a model for generating a content-plan from a set of entity attributes. to capture the local and global contexts from the input set, we proposed a novel gsc-attention. this attention mechanism consists of three attention schemes, which combine the intra-attention among tokens in an attribute and the inter-attention among attributes. our contentplanner further integrates a content-selection mechanism via a masking strategy. experimental results on real-world datasets confirm the effectiveness of our model to generate a content-plan. despite outperforming all competitors, our generated content-plan can be further improved, especially for a large and homogeneous input set (e.g., the rotowire dataset). a further decoding strategy will be explored for future work. another interesting direction is to design a text generation model that exploits the proposed content-planner.