T op G u NN : Fast NLP Training Data Augmentation using Large Corpora (2021)

Abstract: Acquiring training data for natural language processing systems can be expensive and time-consuming. Given a few training examples crafted by experts, large corpora can be mined for thousands of semantically similar examples that provide useful variability to improve model generalization. We present TopGuNN, a fast contextualized k-NN retrieval system that can efficiently index and search over contextual embeddings generated from large corpora. TopGuNN is demonstrated for a training data augmentation use case over the Gigaword corpus. Using approximate k-NN and an efficient architecture, TopGuNN performs queries over an embedding space of 4.63TB (approximately 1.5B embeddings) in less than a day.

## introduction to collect training data for natural language processing (nlp) models, researchers have to rely on manual labor-intensive methods like crowdsourcing or hiring domain experts. rather than relying on such techniques, we present topgunn, a system to make it quick and easy for researchers to create a larger training set, starting with just a few examples. large-scale language models can be effectively used to search for similar words or sentences; however, attempting to extract the most similar words from a large corpus can become intractable and time consuming. our system top-gunn utilizes a fast contextualized k-nn retrieval pipeline to quickly mine for a diverse set of training examples from large corpora. the system first creates a contextual word-level index from a corpus. then, given a query word in a training example, it finds new sentences with words used in similar contexts to the query word. figure 1 shows an example of the results of querying for the word "diagnosis" used in different contexts. topgunn pre-computes bert contextualized word embeddings over the entire corpus, and then efficiently searches through them when queried using approximate k-nn indexing algorithms. our system has been designed with efficiency and scalability in mind. we demonstrate its use by indexing the gigaword corpus, a large corpus for which we pre-computed 1.5b contextualized word embeddings (totaling 4.63tb), and using topgunn to run search queries over it. a detailed description of the system's architecture is given in section 3. ## use case: kairos event primitives our primary use case stems from our work on the darpa kairos program. 1 the darpa kairos program seeks to develop a schemabased ai system that can identify complex events in unstructured text and bring them to the attention of users like intelligence analysts. kairos systems are based on an ontology of abstracted event schemas which are complex event templates. complex event schemas are made up of a series of simpler events, and specify information about participant roles, temporal order, and causal relations between the simpler events. the simplest level event representations used in kairos are "event primitives". for each event primitive, a definition of the primitive is given along with the event's semantic roles. an example of a kairos event primitive is attack: label conflict.attack description a violent physical act causing harm or damage slot role slot argument constraints attacker per, org, gpe, sid target loc, gpe, fac, per, com, veh, wea, sid instr./means com, veh, wea place fac, loc, gpe temporal start and end (times specific to event) duration 1 second to multiple years each event primitive contained 2-5 example sentences. prior to topgunn, example sentences were selected by linguists who manually retrieved them from a corpus by keyword search. with top-gunn, we can find thousands of candidate sentences automatically and then annotators can make a quick pass to filter down to the final set. some work attempts to create event extraction systems without extensive training data. for instance, @xcite discusses how training could be performed using a single "bleached statement," or a definition of an event, without needing a large set of labeled training examples. rather than relying on such techniques, we design a system to make it quick and easy for annotators to create a larger training set . ## system design a diagram of topgunn is given in figure 2 . top-gunn is engineered to run in multiple stages: 1) pre-processing, 2) generating embeddings, 3) indexing, and 4) running queries. ## getting started with topgunn you can get started with topgunn on github: https://github.com/penn-topgunn/topgunn ## conclusion we have presented a system for fast training data augmentation from a large corpus. to the best of our knowledge, existing search approaches do not make use of contextual word embeddings to produce the high quality diverse results needed in training examples for tasks like our event extraction use case. we have open sourced our efficient, scalable system that makes the most efficient use of human-in-the-loop annotation. we also highlight several other nlp tasks where our system could facilitate training data augmentation in section 5. future work may include enabling topgunn to query for multi-word expressions (i.e. "put a name to"), hyphenated expressions (i.e. "pre-existing conditions"), or in the form of natural language questions as seen in @xcite . finally, identifying antonymy as studied in @xcite would be a valuable extension for more finegrained search results as synonyms and antonyms often occupy the same embedding space.