Prototype-Representations for Training Data Filtering in Weakly-Supervised Information Extraction (2022)

Abstract: The availability of high quality training data is still a bottleneck for the practical utilization of information extraction models, despite the breakthroughs in zero and few-shot learning techniques. This is further exacerbated for industry applications, where new tasks, domains, and specific use cases keep arising, which makes it impractical to depend on manually annotated data. Therefore, weak and distant supervision emerged as popular approaches to bootstrap training, utilizing labeling functions to guide the annotation process. Weakly-supervised annotation of training data is fast and efficient, however, it results in many irrelevant and out-of-context matches. This is a challenging problem that can degrade the performance in downstream models, or require a manual data cleaning step that can incur significant overhead. In this paper we present a prototype-based filtering approach, that can be utilized to denoise weakly supervised training data. The system is very simple, unsupervised, scalable, and requires little manual intervention, yet results in significant precision gains. We apply the technique in the task of attribute value extraction in e-commerce websites, and achieve up to 9% gain in precision for the downstream models, with a minimal drop in recall.

## introduction weak supervision and data programming have recently emerged as powerful techniques to support information extraction models. weak supervision is useful for dynamic environments, where new tasks or deployment domains keep emerging, and using manually annotated data is impractical. weakly supervised data programming aims to reformulate the training data annotation process into a programming paradigm. instead of manually annotating each training sample, the annotation process is handled through labeling functions, which are then used to automatically annotate the training corpora. these labeling functions are usually based on manually predefined patterns or regular expressions, that are matched against the target unannotated data. the weak supervision setup allows for an efficient and scalable training data collection process, but at the expense of accuracy. labeling functions can over-match, resulting in many irrelevant and out-of-context matches. this hinders the performance, and could require manual cleaning steps to lift quality, adding to the overall overhead. in this paper we present a data filtering system based on prototype-learning. prototype here refers to the correct contexts where a given target value is usually mentioned, based on a sample dataset. this can be drawn from a manually annotated corpus (making it more supervised), or it can be based on the centroid of the raw dataset (more noisy, but totally unsupervised). at runtime, we get the embedding of the prototype, along with context embedding of the annotated label for each training sample, and use outlier detection constructs to remove outliers. we calculate the distance between the prototype and the in-context label representation, and if the distance is above a certain threshold we filter it out. these out-of-context annotations are out-of-distribution, so approaches that utilize prototype-learning as part of the end-to-end network, or other noise-robust strategies, would not handle them properly. removing them before modeling is the best approach to reduce noise. the intuition behind this filtering logic is that value context embeddings that are distant from the prototype tend to be more noisy, likely reflecting outof-context matches. we use several prototype and data embedding techniques. we utilize the filtering technique for attribute value extraction (ave), an information extraction task that has recently been gaining much momentum in e-commerce applications. the goal of the ave task is to obtain structured product features from the unstructured natural language description of the product's page in e-commerce websites. this has several downstream applications in areas including product search, comparison, question answering, among others. due to the dynamic and large scale nature of this domain, contributions for the ave task often use weak and distant learning techniques to train the extraction models. our results show up to 9% absolute improvement in precision, with minimal drop in recall (about 1%), for the extraction model trained on the filtered data. ## related work prototype-based approaches for nlp have traditionally been used in the word representation literature @xcite . interest for prototype-based approaches increased significantly with the onset of prototypical networks (protonet), with successful utilization in few-shot learning classification in computer vision tasks @xcite , and contrastive learning models @xcite . protonet-based approaches compute one prototype per class as the class mean. these prototypes are then used in a nearest neighbour classifier to update the objective function. this is consistent with our overall setup. there have been many contributions since then utilizing prototypical networks for nlp tasks, mostly in few-shot learning in information extraction @xcite @xcite . but as far as we are aware, this paper is the first to use prototype representations for training data filtering in weak supervision models. there are other contributions utilizing data centroids for outlier detection, mostly through k-nearest neighbor formulation as well. the idea is to remove samples that are far from the cluster's centroid, in a clustering setup @xcite . and we in fact use a similar formulation in our filtering approach, with the context prototype as the target. the attribute value extraction task has traditionally been modeled using distant-supervision @xcite @xcite @xcite which is prone to noise. practical utilization of the ave task in production makes further use of weaksupervision and data programming techniques for training data collection @xcite . this further amplifies the noise issue, and makes training data filtering techniques more important. ## conclusion we presented an automatic filtering approach using prototype-based representations. we applied the approach on the ave task, and showed that using centroid-based prototypes outperforms gold-data