# Natural Language Generation

## Summary


There's a lot of structured data that's perhaps easier to understand if described in a natural language. Highlights from a financial spreadsheet, next week's weather prediction, and short summary of a long technical report are some examples. **Natural Language Generation (NLG)** is the process of generating descriptions or narratives in natural language from structured data. 

NLG is a sub-field of Natural Language Processing (NLP). NLG often works closely with Natural Language Understanding (NLU), another sub-field of NLP. Where input is unstructured text, NLU helps in producing a structured representation that can be consumed by NLG. Generating language uses more of our brain than understanding it. Likewise, computers might find NLG a more difficult task than NLU. 

A mature NLG system can free humans from mundane writing, create narratives quickly, enable almost real-time reporting, and streamline operations.

## Discussion

### What are some applications of NLG?

There are plenty of practical NLG applications: analysis for business intelligence dashboards, IoT device status and maintenance reporting, individual client financial portfolio summaries, personalized customer communications, and more. NLG, along with NLU, is at the core of chatbots and voice assistants. 

A familiar example might be Gmail's Smart Compose. It reads email content and suggests short responses. The Associated Press uses NLG to generate thousands of corporate earnings reports in seconds. During the December 2019 UK elections, BBC News published 689 local stories of 100K words in 10 hours. Public datasets were used at the input and articles were generated in a style and tone suited for local audiences. 

For computer science domain, NLG has been used to write specifications from UML diagrams, or describe source code changes. 

Forge.ai processes unstructured data using mainly supervised NLU models. However, training data is limited. They overcame this by using NLG to synthesize training data. They used human-annotated examples plus knowledge sources. Similarly, synthesized electronic health records can enable de-identified data sharing among healthcare providers and train ML models. 


### What's expected of a good NLG system?

NLG must include in its response information that's most relevant to the user in the current context. This includes the level of detail that's to be included. For effective communication, information must be presented in a sensible order. Thus, organization and structure are important. One sentence must follow logically from another. Likewise, NLG must know when to group them into paragraphs or even sections. 

NLG must conform to the syntax of the language. In addition, it's a good idea to use common expressions. A sophisticated system could be trainable, domain-independent, and even capable of sarcasms and idiomatic expressions. 

Style matters. If the intent is to present high-level financial summary to top management, the writing style must be formal, concise and fact-based. If the intent is to convince readers of a certain point of view, the style could be argumentative with references to supporting evidence. 

On the whole, NLG is about making choices and pursuing specific communicative goals. 


### What's the typical pipeline in an NLG application?

A typical NLG pipeline has these basic steps:  

    + **Content Determination and Text Planning**: Determine the information to be communicated. Structure the information. It's also called *Macro Planning* or *Document Planning*. Information could come from a knowledge base. Selection of information must consider goals and preferences of both writer and reader.
    + **Sentence Planning**: Decide how information must be split into sentences and paragraphs. Plan for a flowing narrative. Also called *Micro Planning*, this involves techniques such as referring expressions, aggregation, lexicalization and grammaticalization.
    + **Surface Realization**: Generate individual sentences in a grammatically correct manner. This involves syntax selection and inflection.
    + **Physical Presentation**: The output could be written or spoken text. Either way, this step does articulation, punctuation and layout.

### Could you describe the main components or tasks in NLG?

We describe a few NLG components:  

    + **Aggregation**: Combining two sentences into one using a conjunction, such as "Sam has high blood pressure *and* low blood sugar." Aggregation could also be applied to paragraphs and higher-order structures.
    + **Lexicalization**: This is about word choice. Between 'depart' and 'leave', "the train departed" is more formal than "the train left".
    + **Referring Expression Generation**: Using preceding context, select words or phrases to identify domain entities. An example is the use of pronouns, such as "I just saw Mrs. Black. *She* has a high temperature."
    + **Using Discourse Markers**: The inclusion of the word 'also', makes this sentence more fluent: "If Sam goes to the hospital, he should *also* go to the store."
    + **Linguistic Realization**: Consider the sentence "There are 20 trains each day *from* Aberdeen *to* Glasgow." Following syntactic rules, NLG has added function words 'from' and 'to'. Due to morphology, the plural form of 'train' has been used. Due to orthography, the first word is capitalized and full stop added at the end.

### Which are main technical approaches to building NLG systems?

We've two approaches to NLG: 

    + **Template-based**: Texts have pre-defined structure with gaps. Gaps are filled with data. These systems evolved to allow greater control via scripts and business rules. These developed further to include linguistic capability to generate grammatically correct text.
    + **Dynamic Creation**: At a micro level, sentences are created dynamically from semantic representations and a desired linguistic structure. At a macro level, sentences are organized into a logical narrative as suited for the audience and the purpose of communication.For dynamic creation of text, a **Markov Chain** model was initially applied. Given the current word and its relationships with other words, this model predicts or generates the next word. However, such models often lacked structure and context. 

More recently, **Language Modelling** has been applied to NLG. Given recent words, the model predicts the next word. An *n-gram model* is a specific way to build a language model. Later, neural networks were successfully applied to language modelling. 

Some classify NLG into Basic NLG, Templated NLG and Advanced NLG. 


### How have neural networks been applied to build NLG models?

RNNs, LTSMs and transformer networks have been used for NLG models. RNNs exploit the sequential nature of text and "remember" previous words to predict the next word. LSTMs are better than RNNs at remembering longer word sequences. They also have a mechanism to selectively forget when context changes. 

**Transformers** have become popular due to their state-of-the-art performance. They look at the relationships among words in context. All words are represented individually in the vector space instead of reducing them to a single fixed-length vector. Transformers have been used to build language models and thereby enable NLG. Two popular ones are BERT and GPT-2. However, it's been remarked that GPT-2 lacks language understanding. 

Neural networks typically take an **end-to-end approach** to NLG. We could use an encoder-decoder architecture along with an attention layer. The encoder produces a hidden representation of the input text. The decoder generates the text. Compared to template-based systems, neural network models give less control but are more flexible and expressive. 


### How can I evaluate NLG models?

Evaluating NLG systems is a challenge in itself and this has been addressed since the mid-1990s. Evaluations are useful to assess the underlying theory of an NLG system, compare two NLG systems, or determine if a particular is showing improvements. In **black box evaluation**, we evaluate the entire system. In **glass box evaluation**, we evaluate the system's component parts such as document structuring or aggregation. 

Evaluations should look at **accuracy** (conveying the desired meaning) and **fluency** (text flows and is readable). These two factors are not necessarily correlated. Evaluations can be by humans or automated. Multiple methods and metrics are preferred. 

Evaluation metrics can be word-based (TER, BLEU, ROUGE), grammar-based (readability, spelling errors, characters per word, etc.) or measure semantic similarity. Another way to categorize the metrics is n-gram overlap (BLEU, GTM, CIDEr), content overlap (MASI, SPICE) and string distance (Levenshtein, TER). BLEU has been criticized by researchers. 

The E2E NLG Challenge (2017-18) used five metrics: BLEU, NIST, METEOR, ROUGE-L, and CIDEr; plus RankME. Rank-based Magnitude Estimation (RankME) enables reliable and consistent human evaluation. 


### Could you describe some tools to help with NLG?

Among the commercial tools are Arria NLG, AX Semantics, Yseop, Quill, and Wordsmith. Among the open source tools are SimpleNLG and NaturalOWL. Wordsmith is from Automated Insights, who can be regarded as a pioneer in NLG. Quill, from Narrative Science, makes use of deep learning. 

Some NLG platforms (such as Wordsmith) can be integrated with business intelligence platforms so that dashboards can be enhanced with descriptive explanations or highlights. 

## Milestones

1950

During the 1950s and 1960s, NLG is not seen as a dedicated field of study but as a component of **machine translation** systems. It's only in the 1970s that researchers look into NLG for the purpose of communication. In the 1980s, NLG becomes a dedicated field of research. Also in the 1950s, the approach is **template-based** and therefore NLG produces the same output to all listeners in all situations. 

Nov  
1970

Simmons and Slocum adapt **semantic networks** to the task of NLG. Nodes are word senses. Connecting paths are relations. Essentially, the network defines the grammar as ordered sets of syntactic transformations. The grammar includes voice, form, aspect, tense, mood, and so on. This is used by the algorithm when generating text. They implement this in LISP. 

1974

Davey makes the case that production (NLG) is not a reverse process of comprehension (NLU).  While language syntax may benefit comprehension, it's not sufficient for discourse-based production. From a given semantic representation, the form that the sentence would take is unpredictable. 

1992

As one of the earliest commercial applications of NLG, Goldberg et al. describe a system that generates bilingual weather reports from graphical weather predictions. 

Mar  
1997

Reiter and Dale note that the most common architecture for NLG is a **three-stage pipeline**. They describe each stage along with essential tasks that belong to each stage. They give guidelines to developers to build practical NLG systems. They also note that NLG may not be the right approach for some use cases. Instead, present information graphically, use mail-merge systems or employ human authors. 

Sep  
1989

Hovy notes that beyond the literal meaning of words, we often tune our communication in relation to the listener, the situation, or interpersonal goals. Hovy therefore studies the importance of **pragmatics**. NLG has often asked two questions: "what shall I say?" and "how shall I say it?" NLG should also concern itself with "why should I say it?" Hovy also implements a program called *PAULINE*, which he acknowledges being primitive and much more research needs to be done. 

1993

Miikkulainen proposes *DISCERN* as a sub-symbolic **neural network model** for NLG. It can create summaries or answer questions. The system includes parsers, at sentence level and story level. For generation, there's a story generator and a sentence generator. These are helped by a memory block and a lexicon. Distributed real-valued vectors represent words both at lexical and semantic levels. Representations are based on thematic case roles. Other early neural networks for NLG from the 1990s include *ANA* and *FIG*. 

Jun  
2000

The first *International Natural Language Generation Conference (INLG)* is held, as a continuation of international workshops on NLG held regularly during 1982-1998. INLG becomes an almost biennial event. Full papers of these conferences are available online.

2007

For college basketball in the US, *StatSheet* is launched to provide game previews, injury reports, and recaps. Much of this is generated automatically. Soon it becomes the top website for college basketball statistics. In 2011, StatSheet changes its name to Automated Insights and partners with NFL for automated content production. 

2013

NLG enters the fields of **data analysis and business intelligence**. A particular example is Allstate Industries adopting NLG. In 2015, Gartner names NLG as an official technology space. It predicts that by 2020, NLG will be a standard feature in 90% on these platforms. By 2016, NLG gets integrated into many BI and data visualization platforms such as Tableau, MicroStrategy and Power BI. 

Jun  
2016

By exploiting similarities across domains, Wen et al. propose a method to train models for **multi-domain NLG**. Synthesized out-of-domain training data is created by data counterfeiting. Then the model is fine-tuned using a much smaller dataset of in-domain utterances. Semantically Conditioned Long Short-Term Memory (SC-LSTM) network is used. Reading gates control semantic input features presented to the network. 

Aug  
2016

Researchers show that when users are presented with uncertain data, NLG improves decision making. They use graphs of weather forecasts with likelihoods. Decisions are better by 24% with NLG, and by 44% when NLG is combined with graphs. The improvement is as high a 87% for women. 

Mar  
2017

Researchers initiate the *E2E NLG Challenge*. Whereas there are many distinct steps in traditional text generation method, **end-to-end generation** attempts to combine all steps into a single model. Such a model would be trained on vast amounts of annotated text. For the Challenge, a crowdsourced dataset of 50k instances in the restaurant domain is used. Training data consists of meaning representations (MRs) and corresponding reference texts. In 2018, from 62 submissions, a seq2seq-based ensemble model with re-ranker is judged the overall winner.