Relevance, Diversity, and Exclusivity: Designing Keyword-augmentation Strategy for Zero-shot Classifiers (2024)

Abstract: Zero-shot text classification involves categorizing text into classes without labeled data, typically using a pre-trained language model to compute the correlation between text and class names. This makes it essential for class names to contain sufficient information. Existing methods incorporate semantically similar keywords related to class names, but the properties of effective keywords remain unclear. We demonstrate that effective keywords should possess three properties: 1) keyword relevance to the task objective, 2) inter-class exclusivity, and 3) intra-class diversity. We also propose an automatic method for acquiring keywords that satisfy these properties without additional knowledge bases or data. Experiments on nine real-world datasets show our method outperforms existing approaches in fully zero-shot and generalized zero-shot settings. Ablation studies further confirm the importance of all three properties for superior performance.

## introduction zero-shot text classification is the process of categorizing text into classes without any training data, which is essential in scenarios where creating a large amount of labeled data is impractical. to this end, most zero-shot classification techniques utilize signals that indicate the relationship between each instance and class, such as semantic textual similarity between instances and class names @xcite or the contextual word co-occurrence of the instance and the class name found in large language models like bert @xcite and t5 @xcite . the performance of zero-shot classifiers is heavily influenced by keywords related to each class (including the class name itself), as these classifiers use the keywords as queries to compute the similarity between each instance and class. for example, pet @xcite employs a masked language model like bert to estimate the class of a text instance by synthesizing a sentence from the text using a template such as "${text} this text is about @xcite .", calculating token probabilities at the masked position, and aggregating the probabilities of keywords related to each class (e.g., token "news" for class "news" and token "finance" for class "economics"). this class to related keywords mapping is sometimes referred to as a verbalizer @xcite . since determining optimal keywords for each class is hard, several works tried to determine proper related keywords for classes using external sources such as knowledge graphs @xcite or language models @xcite . regardless of whether it is manual or automatic, conventional ways to determine related keywords of each class often overlook the nature of classification. (1) keywords relevance to the objective: first, the keywords attached to each class should be relevant to the classification objective, while the conventional method always attaches the same keywords for the same class name. for example, a class-keywords mapping { "beauty" → ("mascara", "lipstick") } is suitable for product classification in e-commerce but may not be the best fit for movie classification. (2) intra-class diversity of keywords: second, the related keywords for a class should cover as broad a range of concepts as possible. existing methods do not always consider the diversity of keywords within a class. (3) inter-class exclusivity of keywords: third, the related keywords for each class should be as distinct as possible, ensuring that two or more classes do not share similar keywords. for instance, the classes "food" and "cell phone" might both have the keyword "apple," which can confuse zero-shot classifiers. existing methods can produce such confusing class-keyword mappings because the keyword assignment for each class is performed independently. in this paper, we explore the strategy of identifying optimal keywords for classes in zero-shot classifiers, considering the three properties mentioned above. through extensive experiments, we found that considering all properties is necessary for obtaining better zero-shot classification performance in popular classifiers. to generate the optimal keywords automatically, we propose a new generate-then-rerank framework re-dex (relevance, diversity, exclusivity) for keyword generation based on the concept of maximal marginal relevance (mmr) @xcite , which is often used in information retrieval. the extensive experiments demonstrate the effectiveness and versatility of the proposed method as it improved the performance of two types of state-of-the-art zero-shot classifiers drastically without any modifications to those methods @xcite @xcite @xcite across all zero-shot settings, including generalized zero-shot text classification (gztc) and fully-zero-shot text classification. our main contributions are as follows. ## conclusion this paper proposes a novel method for improving zero-shot text classification that finds keywords related to classes properly. our method generates diverse keyword candidates by a generative language model and reranks the candidates by an extended maximal marginal relevance method to acquire the keywords that are diverse within a class and exclusive among different classes. experimental results on fully zero-shot and generalized zero-shot text classification tasks demonstrate the effectiveness of the proposed method.