Show, Write, and Retrieve: Entity-aware Article Generation and Retrieval (2023)

Abstract: Article comprehension is an important challenge in natural language processing with many applications such as article generation or image-to-article retrieval. Prior work typically encodes all tokens in articles uniformly using pretrained language models. However, in many applications, such as understanding news stories, these articles are based on real-world events and may reference many named entities that are difficult to accurately recognize and predict by language models. To address this challenge, we propose an ENtity-aware article GeneratIoN and rEtrieval (ENGINE) framework, to explicitly incorporate named entities into language models. ENGINE has two main components: a named-entity extraction module to extract named entities from both metadata and embedded images associated with articles, and an entity-aware mechanism that enhances the model’s ability to recognize and predict entity names. We conducted experiments on three public datasets: GoodNews, VisualNews, and WikiText, where our results demonstrate that our model can boost both article generation and article retrieval performance, with a 4-5 perplexity improvement in article generation and a 3-4% boost in recall@1 in article retrieval. We release our implementation at [this http URL](https://github.com/Zhongping-Zhang/ENGINE).

## introduction comprehending articles enables a wide range of applications such as story generation @xcite , image-to-text retrieval @xcite , automated journalism @xcite , defending against misinformation @xcite , and writing wiki articles @xcite , among others. inspired by the impressive capability of large language models, recent work (e.g., @xcite ; @xcite ; @xcite ) generate or retrieve articles by training language models on massive datasets (e.g., the for article generation and retrieval. prior work @xcite @xcite , shown in (a), typically comprehends articles by uniformly encoding all text tokens. however, these approaches face challenges in accurately recognizing and predicting named entities. in our paper, shown in (b), we propose a method to extract named entities from embedded images and explicitly model the named entities in articles, boosting the performance of both article generation and article retrieval. pile @xcite or laion400m @xcite ). these models typically uniformly encode all text tokens in the articles including named entities @xcite @xcite . in other words, named entities like organizations, places, and dates are modeled together with other text, as illustrated in figure 1(a ). however, it poses a challenge for these models to accurately recognize and predict named entities, as they can be unique to specific articles. e.g., in figure 1 (a), entities like "ms. jolie" and "jolie pitt" may only appear in articles related to the celebrity "angelina jolie." directly extracting named entities from userprovided prompts is a straightforward entity-aware approach used by methods addressing news image captioning @xcite @xcite . however, these methods do not generalize well to article generation and retrieval since they rely on substantial contextual information (via the articles) as well as a direct indication of entities that may appear in predicted captions. for example, in figure 1 (a), the article mentions that angelina jolie pitt is an oscar winner, but this information is not present in the other metadata like the image captions. thus, a language model generating an article must infer this named entity information rather than directly extracting it from the metadata. even if a list of named entities were provided, an article generation model must determine where and when to use each of them. in contrast, in news image captioning, entities used in the caption almost always appear in the body of the article @xcite , and the image itself will directly inform what named entities should be used for its caption. thus, as we will show, adapting entity-aware mechanisms from related work (e.g., @xcite ) results in poor performance in our task. to address the aforementioned issues, we propose an entity-aware article generation and retrieval (engine) framework to explicitly incorporate and model named entities in articles. en-gine mainly consists two modules: a named-entity extraction module and an entity-aware module. in the named-entity extraction module, we show that providing a list of named entities for article generation improves performance. however, creating such lists does require a small overhead cost. thus, we also demonstrate we can improve performance without manual input. as shown in figure 1 (b), we observe that images associated with an article often contain information about the article's events. therefore, we explore leveraging large vision-language models to extract named entities from embedded images. specifically, we employ clip @xcite to automatically select a set of likely named entities from embedded images. in the entity-aware module, we introduce special tokens after each entity name to indicate its entity category. in this case, engine models the named entity and its entity category jointly. an additional benefit brought by our entity-aware mechanism is the named-entity recognition (ner) ability, i.e., our model not only recognizes and predicts the entity names but also predicts their entity categories simultaneously. in summary, the contributions of this paper are: ## related work article generation in recent work uses large-scale pretrained transformer models that can be separated into unconditional text generation @xcite and conditional text generation @xcite . generating articles via unconditional samples has been found to be less effective, since the models may interpret the first sentence of articles as a tweet and start posting responses @xcite . to enable controllable generation, gpt3 (brown et al., 2020) conditions article generation on titles and the initial sentences of articles. grover (zellers et al., 2020) decomposes news articles into distinct parts and conditions generation on metadata like the author or organization. in this paper, we further explore the impact of named entities and embedded article images. specifically, engine produces articles conditioned on both metadata and embedded article images, with a dedicated focus on the explicit extraction and modeling of named entities. article retrieval is commonly accomplished by image-text matching frameworks. early work on andrew mccutchen, 27, of the pittsburgh pirates is among the heirs apparent to become the next links in a long chain of baseball greats. image caption andrew @xmath0, 2012 
 it is a revolution that started march 5, 2012 
, when he signed a contract extension that will keep him in black and gold through 2018 
. it did not matter that the pirates were coming off their @xmath1. mccutchen saw a bright future where others saw a lost cause . now the pirates are heading to the playoffs again , led by an intense , charismatic and talented player who has become the biggest star in a city where sidney @xmath2, ben @xmath3. jeter , though , has @xmath4, while mccutchen'@xmath5. and maybe that is why it could take a handful of players to move the sport forward , rather than just one . jeter became jeter in @xmath6. . . . (a) named-entity extraction (b) entity-aware mechanism visual ner (clip, eva-clip, etc.) named entity recognition (ner) domain: nytimes.com date: 03-24-2014 title: seeking a sport's new face … extracted named entities prompts (metadata) extracted named entities language model target articles image-text matching has primarily focused on developing bespoke models @xcite @xcite with various retrieval loss functions, such as triplet loss @xcite or proxy anchor loss @xcite . however, these models are often domain-specific and limited in the expressiveness of the text. in recent work, large visionlanguage models @xcite @xcite have addressed these limitations by contrastive image-language pretraining on massive datasets (e.g., laion-400m @xcite ). these large vision-language models have demonstrated exceptional performance in zero-shot image-text matching. therefore, in our experiments, we employ the pretrained visionlanguage model eva-clip @xcite to extract visual and text representations from articles, and compute the cosine similarity between these representations to obtain the retrieval predictions. entity-aware mechanisms have been applied in closely related tasks, such as news image captioning, which aims to caption images based on articles and images. @xcite proposed an end-to-end framework that takes the concatenation of article and image features as input and outputs captions by an lstm decoder. however, this approach often fails to predict named entities that were not seen during training. thus, more recent work boosts performance by extracting entity representations from user-provided articles and inserts them into generated templates @xcite @xcite . in this paper, we effectively reverse the inputs and outputs of these approaches, i.e., we generate an article based on images and captions rather than generating captions based on images and articles. as discussed in the introduction, this shift breaks the assumptions used by entity-aware mechanisms in image captioning, causing them to not generalize well in our article generation and retrieval task. ## engine: entity-aware article generation and retrieval given user-provided prompts and embedded article images, our task aims to more accurately represent articles by recognizing and predicting named entities. thus, we incorporate candidate named entities as an additional control to our language model. we propose an entity-aware mechanism to jointly model entity names and their corresponding entity categories in section 3.1. to enrich the entity information available to language models, we introduce our named-entity extraction approach in section 3.2. finally, we introduce the learning strategy of engine in section 3.3. figure 2 provides an overview of our method. ## discussion in our paper, we mainly investigate modeling machine-generated articles, which can be used directly for generation while also can providing strong language features to support applications like article retrieval. however, actors can also use the same technology to generate articles for misinformation by modifying information of specific fields to realize two potential purposes: monetization (ad revenue through clicks) or propaganda (communicating targeted information) @xcite . thus, the development of a better article generator can not only help humans write high-quality articles but also potentially help train a more powerful discriminator. when comparing the results of the roberta detector for gpt2-1.5b and engine-xl, we find that only 25% of the articles that were predicted as human written came from the same generation prompts. thus, the two methods can provide differ-ent views of the same prompt, which can provide additional information for training an even more powerful machine generated text detector. we note that our contributions are largely architecture agnostic, so they could also be used in rnn-based generators, which may provide a larger distribution shift in the generated articles that may fool a discriminator trained only on transformer outputs. ## conclusion in this paper, we proposed engine, an entityaware article generation and retrieval method that explicitly incorporates and models named entities in language models. concretely, engine extracts named entities from both metadata and embedded images in articles, providing a more comprehensive source of entity information. in addition, we introduce an entity-aware mechanism to help engine recognize and predict named entities more effectively and accurately. engine outperforms current popular language models in quantitative and qualitative experiments on goodnews, visualnews, and wikitext. for example, engine outperforms gpt-j by roughly 2.5 perplexity points using only a quarter parameters of gpt-j and boost the performance of eva02-clip by 3-4 recall@1 accuracy in article retrieval experiments. the noticeable improvements demonstrate that engine can generate and retrieve articles more accurately and efficiently by effectively leveraging named entities.