A Two-Stage Masked LM Method for Term Set Expansion (2020)

Abstract: We tackle the task of Term Set Expansion (TSE): given a small seed set of example terms from a semantic class, finding more members of that class. The task is of great practical utility, and also of theoretical utility as it requires generalization from few examples. Previous approaches to the TSE task can be characterized as either distributional or pattern-based. We harness the power of neural masked language models (MLM) and propose a novel TSE algorithm, which combines the pattern-based and distributional approaches. Due to the small size of the seed set, fine-tuning methods are not effective, calling for more creative use of the MLM. The gist of the idea is to use the MLM to first mine for informative patterns with respect to the seed set, and then to obtain more members of the seed class by generalizing these patterns. Our method outperforms state-of-the-art TSE algorithms. Implementation is available at: https://github.com/guykush/TermSetExpansion-MPB/

## introduction term set expansion (tse) is the task of expanding a small seed set of terms into a larger (ideally complete) set of terms that belong to the same semantic category. for example, the seed set {"orange", "apple"} should expand into a set of fruits, while {"orange", "blue"} into a set of colors, and {"apple","google"} into a set of tech companies. beyond being of great practical utility, the tse task is a challenging instance of a generalization from few examples problem. solving tse requires the algorithm to: (1) identify the desired concept class based on few examples; and (2) identify additional members of the class. we present an effective tse method which is based on querying large, pre-trained masked language models (mlms). pre-trained language models (lms) have been shown to contain semantic @xcite , syntactic @xcite @xcite and factual knowledge @xcite , and to be great starting points for transfer-learning to new tasks via fine-tuning on few examples. however, the tse seed sets are too small for fine-tuning, calling for a different approach. our method uses the mlms directly for the task they were trained for-language-modeling-by issuing word-completion queries and operating on the returned word distributions. 1 previous solutions to the tse problem (also called semantic class induction) can be roughly categorized into distributional and pattern-based approaches @xcite . our method can be seen as a combination of the two. the distributional approach to tse @xcite @xcite @xcite operates under the hypothesis that similar words appear in similar contexts @xcite . these methods represent each term in the vocabulary as an embedding vector that summarizes all the contexts the term appears in in a large corpus, and then look for terms with vectors that are similar to those of the seed term. the methods differ in their context definitions and in their way of computing similarities. a shortcoming of these methods is that they consider all occurrences of a term in the corpus when calculating its representation, including many contexts that are irrelevant to the concept at hand due to polysemy, noise in the corpus or noninformative contexts. 2 in contrast, the pattern-based approach consid-ers specific indicative patterns that signal the desired concept, looking for them in a large corpus, and extracting the terms that appear in them. patterns can be binary @xcite @xcite ("such as x or y"), indicating that both x and y belong to the same class, or unary @xcite ("fruits such as x", "first i painted the wall red, but then i repainted it x"), suggesting that x belongs to a certain category (fruit, color). the patterns can be determined manually @xcite or automatically @xcite . while well tailored patterns can be precise and interpretable, a notable shortcoming of pattern-based methods is their lack of coverage, due to the challenge of finding patterns that are specific enough to be accurate yet common enough in a large corpus to be useful. @xcite use patterns from nonnatural language (html) while gupta and manning (2014) restrict themselves to short patterns of 2-4 words to each side of the masked term. our method. by using mlms, we combine the power of the pattern-based and the distributional approaches: like the patterns-based approaches, we consider only specific, indicative corpus locations (retaining specificity and transparency). we then use the distributional nature of the neural lm to generalize across patterns and corpus locations. we use sentences with a single masked location as indicative patterns. for example, ''we took rexy, our pet , to the vet." is an indicative pattern for the house animals semantic class. given an initial set of seed terms, we first search the corpus for indicative patterns for members of the set (2.1). intuitively, an indicative pattern is a corpus location which is considered by an lm to be a good fit for all seed members. once we identified indicative patterns, we extend the set to terms that can appear in similar patterns. we propose two methods for doing this. the first method (2.2) queries an mlm for completions. while effective, this method restricts the expanded set to the lm vocabulary. the second method (2.3) uses the mlm to define a similarity metric over patterns, and searches the corpus for terms that appear in patterns that are similar to the indicative ones. to summarize, we embrace the pattern-based approach, while using distributional similarity for identifying good patterns as well as for generalizing across patterns. we operate in stages. first, we search the corpus for ℓ indicative masked patterns m 1 , ..., m ℓ , that are likely to signal the concept class in s g with high probability. then, we use the patterns to extend the set. ## experiments and results we refer to the method in section (2.2) as mpb1 and the method in section (2.3) as mpb2. setup. in our experiments we use bert @xcite as the mlm, and english wikipedia as the corpus. following previous tse work (e.g. @xcite )), we measure performance using map (using map 70 for the open set). for each method we report the average map over several runs (exact number mentioned under each table), each with a different random seed set of size 3. based on preliminary experiments, for mpb1 we use ℓ = 160 and @xmath0. 7 when comparing different systems (i.e, in ## conclusions we introduce an lm-based tse method, reaching state-of-the-art results. the method uses the power of lm predictions to locate indicative patterns for the concept class indicated by the seed terms, and then to generalize these patterns to other corpus locations. beyond strong tse results, our method demonstrates a novel use of pre-trained mlms, using their predictions directly rather than relying on their states for fine-tuning. huibin zhang, mingjie zhu, shuming shi, and ji-rong wen. 2009. employing topic models for patternbased semantic class discovery. in acl 2009, proceedings of the 47th annual meeting of the association for computational linguistics and the 4th international joint conference on natural language processing of the afnlp, 2-7 august 2009, singapore, pages 459-467.