Angel: Enterprise Search System for the Non-Profit Industry (2023)

Abstract: Non-profit industry need a system for accurately matching fund-seekers (e.g., AMERICAN NATIONAL RED CROSS) with fund-givers (e.g., BILL AND MELINDA GATES FOUNDATION) aligned in cause (e.g., cancer) and target beneficiary group (e.g., children). In this paper, we create an enterprise search system “ANGEL” for the non-profit industry that takes a fund-giver’s mission description as input and returns a ranked list of fund-seekers as output, and vice-versa. ANGEL employs ColBERT, a neural information retrieval model, which we enhance by exploiting the two techniques of (a) Syntax-aware local attention (SLA) to combine syntactic information in the mission description with multi-head self-attention and (b) Dense Pseudo Relevance Feedback (DPRF) for augmentation of short mission descriptions. We create a mapping dictionary “non-profit-dict” to curate a “non-profit-search database” containing information on 594K fund-givers and 194K fund-seekers from IRS-990 filings for the non-profit industry search engines . We also curate a “non-profit-evaluation” dataset containing scored matching between 463 fund-givers and 100 fund-seekers. The research is in collaboration with a philanthropic startup that identifies itself as an “AI matching platform, fundraising assistant, and philanthropy search base.” Domain experts at the philanthropic startup annotate the non-profit evaluation dataset and continuously evaluate the performance of ANGEL. ANGEL achieves an improvement of 0.14 MAP@10 and 0.16 MRR@10 over the state-of-the-art baseline on the non-profit evaluation dataset. To the best of our knowledge, ours is the first effort at building an enterprise search engine based on neural information retrieval for the non-profit industry.

## introduction non-profit industry consists of non-profit foundations (npfs), non-profit service providers (npse), * equal contribution and independent donor individuals. npfs and independent donor individuals, who function as fund-givers (e.g., bill and melinda gates foundation), supply funds to npses, who function as fund-seekers (e.g american na-tional red cross). a significant number of financial transactions occur between fund-givers and fund-seekers. in 2020, americans gave @xmath0 50k. small-scale fund-seekers cannot hire "donor research analysts" due to the lack of budget, and the absence of an exhaustive search of aligned fundgivers makes small-scale fund-seekers repeatedly seek funds from known and prominent fund-givers. similarly, due to lack of time, fund-givers often find fund-seekers using personal connections and donate without understanding the impact they can create by exhaustively searching and donating to relevant fund-seekers. we create an enterprise search system "an-gel", in collaboration with a philanthropic startup, that can accurately match fund-givers with fundseekers using publicly available data on non-profit organizations in irs-990 (internal revenue service) filings * . angel reduces the overheads related to "donor research analysts" compensation for fundseeking organizations and search time for individual donors. the irs-990 filing of a non-profit organization (fund-giver or fund-seeker) consists of multiple forms, each containing multiple fields. some fields provide information on the organization's philanthropic cause, which is necessary for matching fund-givers with fund-seekers, and others provide information on the organization's finances used for filtering it based on spending. we create a mapping dictionary "non-profit-dict" for irs-990 data to accurately map similar fields to a common field and drop irrelevant fields to match fund-givers with fund-receivers. in this way, we curate a nonprofit-search database for the non-profit industry search engines containing information on 594k fund-givers and 194k fund-seekers. enterprise search engines use information retrieval (ir) models that take a query as input and return a ranked list of relevant documents from the search database. textual ir has witnessed the use of language models to obtain contextual vector representations of queries and documents for vector-based matching instead of keyword-based matching. neural information retrieval models have shown considerable improvement in accuracy compared to keyword-based information retrieval models like bm25 @xcite . to our knowledge, existing enterprise search engines (e.g. propublica non-profit explorer) for the non-profit industry do not use neural information retrieval models. enterprise search engines based on neural information retrieval (ir) employ models such as col-bert @xcite , that uses bert encoder @xcite to obtain contextualized token vectors representing mission descriptions of fund-seekers and matches them with the contextualized token vectors of grant descriptions of fund-givers. we augment the capability of colbert for the task of enterprise search in non-profit industry by exploiting the two techniques of (a) syntax-aware local attention (sla) and (b) dense pseudo relevance feedback (dprf). we train our models on msmarco passage ranking dataset @xcite . to evaluate the search quality of angel, we curate a non-profit-evaluation dataset from the non-profitsearch database. the dataset is a 463 by 100 matrix, where each row represents a fund-giver and each column represents a fund-seeker. each index in the 2-dimensional matrix is a matching score ranging from 0 to 9. domain experts from the philanthropic startup have annotated this dataset to evaluate an-gel. we compare the performance of angel with colbertv2 @xcite on the non-profit-evaluation dataset. through ablation study, we observe that angel based on colbert performs better than colbert-v2, a more potent ir model, on the non-profit-evaluation dataset. our contributions are: ## dataset we curate a "non-profit-search" database and a human-annotated "non-profit-evaluation" dataset in collaboration with a philanthropic startup to test the system's capability of matching fund-givers with fund receivers. in this section, we first give details about the irs-990 filings used to populate our dataset. after that, we discuss the variable mapping created to curate our non-profit-search database efficiently. once the non-profit-search database is populated, a subset of it is given to the philanthropic startup for annotation that results in the non-profit-evaluation dataset. at the end of this section, we discuss the strategy for annotation. ## experiments we compare the performance of angel (sla-dprf-colbert) model with the official "col-bertv2 checkpoint" * , which has been trained for retrieval for a significantly higher number of iterations (approximately 200k). we discuss the model training configurations in detail in b. for evaluating the performance of our models, we use the nonprofit-evaluation dataset. the dataset contains 463 queries and 100 documents. each document has a relevance score on a scale of 0 to 9. the threshold to classify a document as relevant is decided empirically. the average number of relevant documents per query for a threshold of 2 is 47.44. the average number of relevant documents per query for a threshold of 3 is 16.72. the average number of relevant documents per query for a threshold of 4 is 4.76. we selected a threshold of 3 as it neither gives too many relevant documents, nor too less relevant documents per query. ## results our results show 44.5% gain in map@10 and 49.7% gain in mrr@10 score over the baseline on the non-profit-evaluation dataset from using densepsuedo-relevance-feedback and syntax-aware-selfattention in conjunction. the results for these experiments are presented in ## summary, conclusion, and future work in this paper, we show the motivation behind creating an enterprise search engine "angel" for the non-profit domain. training angel on the multilingual version of the msmarco dataset with the ir objective, to support non-english queries and documents remains as future work. propublica non-profit explorer fails to perform accurate full-text search over its search database as the top-2 search results are not relevant to the input query "girl education".