Semantic search with domain-specific word-embedding and production monitoring in Fintech (2020)

Abstract: We present an end-to-end information retrieval system with domain-specific custom language models for accurate search terms expansion. The text mining pipeline tackles several challenges faced in an industry-setting, including multi-lingual jargon-rich unstructured text and privacy compliance. Combined with a novel statistical approach for word embedding evaluations, the models can be monitored in a production setting. Our approach is used in the real world in risk management in the financial sector and has wide applicability to other domains.

## introduction many industry-grade search engines exist (e.g. elasticsearch) as general-purpose systems. while they have a general understanding of languages and includes search algorithms for document retrieval, their accuracy can be sub-optimal. this accuracy ultimately depends on domain specificity and the terms provided by the users. this issue is particularly apparent in domains such as risk management. a critical function of the risk officers is knowing the relevant set of search terms to retrieve documents related to a specific topic. however, constructing a search criteria with all the appropriate search terms/phrases and their logical relations in the search engine remains complex, and can be prune to false-negatives and false-positives. while entity set expansion systems exist (among the earliest includes google sets), they are mostly using general-purpose language models @xcite and thus not domain-specific. in addition, building such an industry-grade domain-specific semantic search engine is challenging as this involves several additional considerations beyond simply an accurate language model. this includes: 1) extensive use of jargons, abbreviations, and unstructured data, 2) multiple languages relating to the same topic, 3) privacy compliance, and 4) performance monitoring of models on production. in this paper, we describe an end-to-end information retrieval (ir) system with custom embeddings to address the aforementioned challenges. the system utilizes multiple advancements in natural language processing (nlp) to process anonymized banking data. capitalizing on our custom language models (in multiple languages and with multiple n-grams), we enable the automatic suggestion of highly relevant similar keywords, to ease the burden of requiring the user to build complex search queries. additional pre-trained contextual embeddings are utilized for ranking by relevance. lastly, we develop novel statistical methodologies for monitoring the stability of our language models on production. ## approach we develop an end-to-end ir system, as depicted in figure 1 . the system, implemented in python, consists of 1) a text mining pipeline that preprocesses the records for indexing and for building language models, and 2) a front-end where the user queries the database with one or more search terms. the search terms are processed the same way by the text mining pipeline as for the records, and are further expanded to include similar words based on the trained domain-specific language models. the similar a b changed old model, a new model, b repeat p times bootstrap distribution spearman's ρ density old model, a new model, b sample m words calculate cosine similarities embedding monitoring some change no change changed some change no change per cos sim bin null distr generate for test distr word expansion is thus a corpus-based approach and is based on the premise that similar words are distributionally related in similar contexts (known as the distributional hypothesis (harris, 1954)). ## experiments we collect around 160k records in the financial sector. to compare across the different word embeddings, we use both pre-trained and custom-trained models. pre-trained models have been previously trained on a large corpus such as the wikipedia, common crawl, or google news. popular models include ones based on word2vec @xcite or glove @xcite . while using these general-purpose language models saves training time and the need for custom preprocessing of data, they potentially lack domain-specific word semantics. therefore, we also train custom word2vec skip-gram word embeddings based on our datasets for english, french, and dutch languages. we evaluate our models directly with the anonymized data as anonymization is required for compliance and regulatory reasons. for extrinsic word embedding evaluations, we build a gated recurrent unit (gru)based model with an embedding layer, followed by a 64-unit gru, a 16-unit fully connected layer with relu activation, and an output layer with sigmoid activation. the model is trained with adam optimizer and binary cross entropy loss function. we also evaluate the quality of the information retrieval, viz. the precision achieved by relevance ranking based on the different sentence embedding models. several models are examined, including tf-idf, average of the word2vec (google news), word2vec (domain-specific), or bert @xcite , cls token of bert, sentence bert (sbert) @xcite , laser @xcite , and different versions of universal sentence encoders (use) @xcite and multilingual universal sentence encoders (muse) @xcite . with the exception of tf-idf and domain-specific word2vec, the other models are all taken as pre-trained models. ## conclusions we present an end-to-end ir system used in production as a semantic search engine with intelligent keyword expansion and continual model monitoring. it addresses several challenges in handling multilingual domain-specific unstructured text and privacy compliance, and simplifying the derivation of keywords within a semantic class. our experiments show that the custom word embeddings are distinct from general-purpose models, can present more relevant search terms, and can be monitored on production based on a novel statistical approach. we also show that different sentence embeddings differ vastly in performance and some, based on experimentation, can be used to improve relevance ranking. overall our proposed solution is applicable for use in other domains.