Building Hierarchically Disentangled Language Models for Text Generation with Named Entities (2020)

Abstract: Named entities pose a unique challenge to traditional methods of language modeling. While several domains are characterised with a high proportion of named entities, the occurrence of specific entities varies widely. Cooking recipes, for example, contain a lot of named entities — viz. ingredients, cooking techniques (also called processes), and utensils. However, some ingredients occur frequently within the instructions while most occur rarely. In this paper, we build upon the previous work done on language models developed for text with named entities by introducing a Hierarchically Disentangled Model. Training is divided into multiple branches with each branch producing a model with overlapping subsets of vocabulary. We found the existing datasets insufficient to accurately judge the performance of the model. Hence, we have curated 158,473 cooking recipes from several publicly available online sources. To reliably derive the entities within this corpus, we employ a combination of Named Entity Recognition (NER) as well as an unsupervised method of interpretation using dependency parsing and POS tagging, followed by a further cleaning of the dataset. This unsupervised interpretation models instructions as action graphs and is specific to the corpus of cooking recipes, unlike NER which is a general method applicable to all corpora. To delve into the utility of our language model, we apply it to tasks such as graph-to-text generation and ingredients-to-recipe generation, comparing it to previous state-of-the-art baselines. We make our dataset (including annotations and processed action graphs) available for use, considering their potential use cases for language modeling and text generation research.

## introduction language modeling is an important task in natural language processing with several applications @xcite including auto complete @xcite among others. a key application of language modeling is text generation @xcite . recent advances in language models have led to impressive generation of programming code in languages such as java and c @xcite @xcite . similarly, efforts have been made towards application of language models in the context of cooking recipes @xcite . both these tasks, recipe and code generation, deal with corpora heavily populated with named entities. in case of cooking recipes these entities are thousands of ingredients used in recipes, hundreds of cooking processes which can be applied upon them and tens of utensils used across the different cuisines. these numerous entities immensely increase the vocabulary of the language model. typically, the vocabulary of corpora of cooking recipes is of the order of tens of thousands of tokens. the vocabulary of the corpus we prepared (more details will follow later in the manuscript) has about 54 thousand tokens and 6 thousand ingredients, which form the single largest entity type. it is amply clear that the language model has to deal with the language semantics comprising of thousands of english language tokens along with the way ingredients, utensils and processes interact with each other. @xcite provide credence to this argument. for instance, fruits occur 27,260 times in their corpus but the fruit 'apple' occurs only 720 times. and fruit is just one kind of ingredient. to some extent, all fruits can be modeled as one entity, as all of them largely behave in the same manner when written in a natural language. the processes like cutting and peeling are common to all fruits. thus @xcite propose using one entity level language model which predicts whether a fruit or a vegetable should occur at a given place, followed by a model specific to the entity type which predicts the probabilistic distribution of which exact fruit fits well. they show that this method greatly influences the quality of text generation for both recipe and code generation owing to the reduction in vocabulary and complexity for the models being trained. we note that programming languages have far less ambiguities and variations than natural language because they are meant to be compiled using a detailed protocol. due to this reason, the language models perform far better on code generation than natural language generation tasks even though the former has a lot of data types and variable names (which work like the entity types and their name). these models have already achieved a bleu score as high as 42.39 @xcite and models by @xcite achieved a validation perplexity as low as 2.39 (a perplexity of 1.0 would mean that the crossentropy loss is zero and is theoretically the best result). thus, we feel that any further improvement in the language model would be hard to quantitatively estimate for programming languages by existing metrics, which is why we focused on corpora of cooking recipes. there have been several advances in the field of recipe text generation. the first implemented system was called epicure @xcite and was a purely rule based generation algorithm. @xcite developed a system for generating salads using statistical search ingredient combinations. @xcite developed a modern neural recipe generation system by maintaining a state of available and used state. despite that it had a low bleu score. more recently, the release of recipe1m+ @xcite , has led to the development a related domain of recipe retrieval -retrieving recipes from a large scale dataset @xcite . in the past year, we have seen two new applications -recipe generation from image @xcite , and personalised recipe generation (recommendation) based on user reviews @xcite . the problem of ingredient to recipe generation is also tackled by many of these papers. aligned with the arguments made earlier, we decided to create our own dataset (which was further cleaned and annotated) for the development of the language model. taking inspiration from @xcite , we apply an unsupervised recipe interpretation pipeline to model the instructions as action graphs along with named entity recognition models which describe in detail in section 2. the language model is itself described in section 3. we further study some of the applications of our language models, testing it for tasks such as graph-to-recipe generation and ingredients-to-recipe generation. image-to-recipe generation as beyond the scope of this paper since @xcite shows that clues from images are a big help for the language model in predicting the recipe. ## language model as mentioned before, we build upon the work done by @xcite pertaining to the domain of building language models for natural language text containing high proportions of named entities. intuitively, the reason why language models suffer for such text can be understood through the following observations - a way to mitigate this problem would be to separate the learning of entity type (ex: fruit) specific knowledge, from entity (ex: apple) specific knowledge. in more intuitive words -all entities of the same type behave in certain similar ways. while it may be uncommon for a recipe to include "heat pizza dough on a grill", if we were to label all ingredients, cooking process and utensils, a pattern like "cooking process ingredient on a utensil" is far more likely to be found. thus it would be easier for a language model to learn such combinations of patterns because of increased occurrences and a decreased vocabulary since it doesn't have to consider the thousands of individual tokens belonging to these categories. @xcite introduced a "type model" to learn this behaviour. in the subsequent operation, the model (or a separate one) can try to learn what specific entity should appear at a place. given the context, it is easier to tell what cooking process or utensil or ingredient one should use. consider a recipe starting with "cooking process a utensil before you begin". a model can easily learn by seeing several examples that in such cases when a cooking process is applied at the start of the instructions section, the most frequent process is "preheat". the input sentence to the language model now becomes "preheat a utensil". strictly speaking, an oven would be a kitchen appliance, and not a utensil. but in the context of discussion, it has been deliberately marked as a utensil and is undoubtedly the best fit here. @xcite introduced an "entity composite model" to learn this behaviour. during inference, it relies on the output of the "type model". for simplicity, the trivial example above includes only three entities, we later explain our experiments with the higher more number of entities and their effects. also, only backward context was considered here, which may indeed be the case when, say, a forward lstm @xcite is used for learning the language model. however a bi-directional lstm @xcite allows much better performance, as also suggested by by @xcite . ## conclusion the hierarchically disentangled language models proposed in this manuscript clearly are an incremental improvement over @xcite , across the three settings we evaluated upon. first, we evaluated it intrinsically in terms of perplexity on two datasets on which it out-performs the primary baseline. we also show that the original dataset with lower number of distinct named entities is comparatively easier for both the models. whereas, the original baseline performed poorly over the newly procured, more complex dataset. this highlights the importance of fine-tuning the subnetworks as well as simplifying the input to entity transformation network -since knowledge about the previous types is already present within the rnn, explicity including it is not only redundant, but also makes the modeling task more complex. secondly, we used our language model as a decoder in an ingredientto-recipe generator. here it got lower perplexity than a baseline encoder-decoder model. finally, we used the language model as the decoder in an action graph-to-recipe model. in this setting the generated recipes had higher bleu compared with reference recipes than the same model using an lstm decoder. the latter two experiments together provide the evidence to the utility of such language models by transfer learning, in more complex tasks. an issue with the proposed language models is the annotation of named entities, which is domain specific and hard to automate. we also did not see much improvement by increasing levels of entity hierarchy, which only made the overall annotation tasks more complex. we encourage future contributions in further testing such language models in different text domains and improve upon our training strategy and architecture -whether specific to certain domains or for general purpose language modeling.