Nearest Neighbor Search over Vectorized Lexico-Syntactic Patterns for Relation Extraction from Financial Documents (2023)

Abstract: Relation extraction (RE) has achieved remarkable progress with the help of pre-trained language models. However, existing RE models are usually incapable of handling two situations: implicit expressions and long-tail relation classes, caused by language complexity and data sparsity. Further, these approaches and models are largely inaccessible to users who don’t have direct access to large language models (LLMs) and/or infrastructure for supervised training or fine-tuning. Rule-based systems also struggle with implicit expressions. Apart from this, Real world financial documents such as various 10-X reports (including 10-K, 10-Q, etc.) of publicly traded companies pose another challenge to rule-based systems in terms of longer and complex sentences. In this paper, we introduce a simple approach that consults training relations at test time through a nearest-neighbor search over dense vectors of lexico-syntactic patterns and provides a simple yet effective means to tackle the above issues. We evaluate our approach on REFinD and show that our method achieves state-of-the-art performance. We further show that it can provide a good start for human in the loop setup when a small number of annotations are available and it is also beneficial when domain experts can provide high quality patterns. Our code is available at 1.

## introduction relation extraction (re) from text is a fundamental problem in nlp and information retrieval, which facilitates various tasks like knowledge graph construction, question answering and semantic search. recent studies @xcite @xcite @xcite in supervised re take advantage of pre-trained language models figure 1: relation extraction example, here both organizations are connected with "acquired by" relation. implicit expression refers to the situation whereas relation is expressed as the underlying message that is not explicitly stated or shown. in figure 1 , relation "acquired_by(organization, organization)" occurs implicitly. such underlying messages can easily confuse the relation classifier. the other problem of long-tail relation classes is caused by data sparsity in training. for example, the refind dataset @xcite comprises 45.5 % of the no_relation instances. the most frequent class in the dataset -"per:title:title" has 4,468 training examples, while over 14 out of 22 classes have less than 500 examples. the majority class can easily dominate model predictions and lead to low performance on long-tail classes. recently, icl (in-context learning) based approach @xcite is utilized for re tasks. the approach achieves improvements over not only existing gpt-3 baselines, but also on fullysupervised baselines even with only a limited number of demonstrations provided in the prompt. specifically, it achieves sota performances on the semeval and scierc datasets, and competitive performances on the tacred and ( zhang et al however, retrieval of examples to demonstrate is a key factor in the overall performance on these pipelines. finding efficient demonstrates often relies on learning-based retrieval @xcite . these learning-based retrievers use annotated data and a llm. this type of retrieval strategy comes with the increased cost (api, infrastructure etc.), time as more experiments are required because most llms are black box and it also needs special expertise. apart from the implicit expression challenge mentioned above, refind poses another challenge to rule-based systems in terms of longer and complex sentences. for example, @xcite cites that the average sentence length in the re-find dataset is 53.7 while the average sentence length in the tacred dataset @xcite is 36.2. further, as per @xcite , refind includes more complex sentences than tacred, with an average entity-pair distance of 11, compared to 8 in tacred. because of this, writing rules at surface text level is a challenge. hence, rules at lexico-syntactic level is the need of the hour. however, strict matching of these rules can yield high precision but low recall result due to accuracy of syntactic parsing. hence, a robust fuzzy pattern matching system is required. inspired by recent studies @xcite @xcite using k-nearest neighbor to retrieve diverse expressions for language generation tasks, we introduce a simple but effective approach that consults training relations at test time through a nearest-neighbor search over dense vectors of lexico-syntactic patterns and provides a simple yet effective means to tackle the above issues. our method achieves an improvement of 1.18% over baseline (f1-score -0.7516). we achieved our results using commodity hardware within a day. that's why our approach is easier to deploy, lightweight and fast. we further show that our approach can provide a good start (f1-score of 0.5122) for human in the loop setup when a small number of annotations (approx. 10% of training data) are available and it is also beneficial (f1-score of 0.6939 with approx. 10% of training data) when domain experts can provide high quality patterns. ## results we conducted experiments with 4 different pattern types. to find the best value of k, we have created a 10% split from the training data and experimented with different values of k (1 to 20). during evaluation, we faced issues in generating dependency tree and/or sdp for around 8.7% instances and for those instances, indices containing sentence vectors were used as fallback strategy. the results in to explore the effectiveness of our approach in human in the loop situation, we conducted a few experiments as shown in figure 4 . we randomly selected n patterns per class from the training data and built indices with those patterns only. we tried different values of n. with @xmath1, we achieved an f1score of 0.5122 with around 10% of the original training data. it shows that the vectorized lexicosyntactic patterns and the cosine similarity based scoring can be a good start to label similar instances when the number of annotations are less. this method can be used in human in the loop setup to either filter out similar instances (explore) or to find similar instances (exploit) for further human review/annotation. to explore the effectiveness of our approach when domain experts are available and can provide high quality patterns specially for the task like this which is restricted to a particular domain, types of documents, types of entities and a handful of relations, we conducted a few experiments as shown in figure 4 . to approximate this experiment, we selected n training patterns from each class which occurs frequently in top k search when development patterns are classified correctly. we call these patterns the most-frequent patterns. we built indices with those patterns only. as shown in figure 4 , with @xmath2, we achieved an f1-score of 0.6939 (6.95% less than the best result) with around 10% of the original training data. it shows that it can bridge gaps quickly with a small amount of high quality patterns. randomly selected; @xmath3; @xmath4 ## conclusion our approach consults training relations at test time through a nearest-neighbor search over dense vectors of lexico-syntactic patterns. we evaluated our approach on refind and show that our method achieves state-of-the-art performance without any direct access to large language models (llms) or supervised training or fine-tuning or any handcrafted rules. we achieved our results using a commodity hardware within a day. that's why our approach is easier to deploy, lightweight and fast. we further explores that our approach can provide a good start for human in the loop setup when a small number of annotations are available and it can be also beneficial when domain experts can provide high quality patterns.