The Natural Language Pipeline, Neural Text Generation and Explainability (2020)

Abstract: End-to-end encoder-decoder approaches to data-to-text generation are often black boxes whose predictions are difficult to explain. Breaking up the end-to-end model into sub-modules is a natural way to address this problem. The traditional pre-neural Natural Language Generation (NLG) pipeline provides a framework for breaking up the end-to-end encoder-decoder. We survey recent papers that integrate traditional NLG submodules in neural approaches and analyse their explainability. Our survey is a first step towards building explainable neural NLG models.

## motivation the end-to-end encoder-decoder is a popular neural approach that is efficient to generate fluent texts. however it has often been shown to face some adequacy problems such as hallucination, repetition or omission of information. as the end-to-end encoder-decoder approaches are often "black box" approaches, such adequacy problems are difficult to understand and solve. in contrast, pre-neural nlg has often integrated a number of sub-modules implementing three main nlg sub-tasks @xcite : macroplanning ("what to say"), microplanning and surface realisation ("how to say"). to improve adequacy and provide for more explainable approaches, recent work has proposed integrating traditional pre-neural nlg sub-modules into neural nlg models. in this paper, we survey some foot_0 of this work, focusing mainly on generation from data-and meaning representations foot_1 . ## macroplanning macroplanning is the first subtask of the traditional pre-neural nlg pipeline. it answers the "what to say" question and can be decomposed into selecting and organising the content that should be expressed in the generated text. ## microplanning microplanning is the nlg sub-task which aims at defining "how to say" the information that was selected and structured during macroplanning. ## surface realisation surface realisation is the last nlg task and consists in creating a syntactically well-formed text out of the representations produced by the previous step. while surface realisation is at the heart of generation when generating from meaning representations, it is largely uncharted in data-and table-to-text nlg and results either from the de-coder language model (which decides on the words and thereby indirectly on the syntax of the generated text) or from the templates used for generation @xcite @xcite . ## conclusion explainable models enable a clear understanding of how the output generated by the model relates to its input. in this short paper, we surveyed a number of neural data-to-text generation models which implement some or all of the nlg pipeline sub-tasks with the aim of identifying methods which could help enhance explainability in neural nlg. our survey highlights two main ways of enhancing explainability: explicit intermediate structures produced by neural modules modeling the nlg pipeline subtasks or latent variables modeling the interface between these modules. thus @xcite 's supervised pipeline model outputs content plans, sentence templates and referring expressions which can all be examined, quantified and analysed thereby supporting a detailed qualitative analysis of each subtasks. similarly, @xcite output explicit text plans and text plan linearisations and @xcite text plans. in contrast, the models introduced in @xcite @xcite @xcite are based on latent variables which mediate the relation between input and output tokens and intuitively, model a document plan by mapping e.g., input rdf triples to text fragments. as illustrated in a third alternative way to support explainability is model analysis such as supported e.g., by the allennlp interpret toolkit @xcite which provides two alternative means for interpreting neural models. gradient-based methods explain a model's prediction by identifying the importance of input tokens based on the gradient of the loss with respect to the tokens @xcite while adversarial attacks highlight a model's capabilities by selectively modifying the input. in future work, we plan to investigate whether domain agnostic, linguistically inspired intermediate structures such as meaning representations could be used to both support explainability and improve performance. another interesting direction for further research would be to develop common evaluation benchmarks and metrics to enable a detailed analysis and interpretation of how neural nlg models perform for each of the nlg pipeline sub-tasks. finally, while most of the approaches we surveyed concentrate on modeling the interaction between content planning and micro-planning, it would be useful to investigate whether any of the methods highlighted in this paper could be exploited to explore and improve the explainability of the various micro-planning sub-tasks (lexicalisation, aggregation, regular expression generation, surface realisation).