Learning Numeracy: A Simple Yet Effective Number Embedding Approach Using Knowledge Graph (2021)

Abstract: Numeracy plays a key role in natural language understanding. However, existing NLP approaches, not only traditional word2vec approach or contextualized transformer-based language models, fail to learn numeracy. As the result, the performance of these models is limited when they are applied to number-intensive applications in clinical and financial domains. In this work, we propose a simple number embedding approach based on knowledge graph. We construct a knowledge graph consisting of number entities and magnitude relations. Knowledge graph embedding method is then applied to obtain number vectors. Our approach is easy to implement, and experiment results on various numeracy-related NLP tasks demonstrate the effectiveness and efficiency of our method.

## introduction numeracy is the ability to reason and to apply numerical concepts, and numbers play a key role in natural language understanding. for example, investors will probably react differently to the news "aapl earnings increase by 2%" vs. "aapl earnings increase by 20%". similarly, in clinical setting, "heart rate is 140 beats per minute" vs. "heart rate is 60 beats per minute" will likely result in different decisions from physicians. in particular, healthcare providers often use the textual triage notes in emergency room to predict which patient to be discharged or admitted. take a triage note for example, "pt had unwitnessed glf. was initially confused as per co-workers. gcs now 14. nauseated." this note is labelled as discharge as a gcs (i.e., glasgow coma scale) of 14 indicates that the patient can response well (gcs ranges from 3 to 15, 3 being completely unresponsive and 15 being responsive). the number 14 plays a key role in the discharge decision. using our proposed number embedding approach, the number is encoded into the same dimensional space as words while keeping the numeracy, so that we can use deep nlp model, say lstm, to better represent the triage note. the model will explicitly learn that a gcs following a large number embedding may indicate discharge while a gcs following a small number embedding may indicate admitted. while numeracy is critical in such domains where numbers are prevalent, most existing nlp models are not designed explicitly to handle numbers. numbers are either directly discarded in pre-processing, or treated as a unk token @xcite . prior literature also shows that neither traditional word embeddings such as word2vec nor the contextualized transformer-based language model such as bert can handle numbers and process numeracy tasks effectively @xcite . one straightforward way to encode numbers in nlp tasks is to map a number's value directly to its embedding (e.g., "twenty-four" embeds to @xcite ). still, this strategy performs poorly while the nlp task involves a large amount of numbers with a wide range @xcite . therefore, encoding numbers into high-dimensional vector space may potentially overcome the difficulties and preserve numeric semantics. along this line, @xcite proposes a number embedding method dice, where number vectors are obtained via mathematical operations. however, this method has a high computational cost due to the math operation which limits its use in encoding a large number of numbers. in this work, we propose nekg (number embeddings from knowledge graph), a simple yet effective number embedding method that produces numeracy-preserving embeddings via a knowledge graph structure. nekg is independent of corpus and creates deterministic number embeddings. to explicitly preserve numeracy, we first construct a knowledge graph consisting of only number entities and magnitude relationships. we then apply transe, a knowledge graph embedding method algorithm 1: number embeddings from knowledge graph input number embedding range [min, max], embedding dimension dim, and the target number n. output the embedding vector embd of the target number n. //constructing numerical knowledge graph. 01: loop 02: triple ← (h, "islesst han", t) for each neighboring pair (h, t) within [min, max] 03: append @xmath0: end loop 05: model ← transe (@xmath1. 06: model.fit (t riples) // inferring embedding for a target number. 07: if @xmath2: 08: @xmath3: else : 10: @xmath4.get_embeddings (n) @xcite , to embed the number entities into a vector space. in this way, numeracypreserved embeddings can be obtained directly without using sophisticated mathematical operations. to obtain the embedding of an out-of-vocabulary (oov) number, we propose an interpolation method that uses the weighted average of its two neighbors' embeddings based on cosine similarity. we experiment our method on several numeracy-related tasks, including evaluating embeddings on their ability to capture magnitude @xcite , and solving numeracy tasks (list maximum, decoding, and addition) @xcite . we also apply our method in a downstream financial nlp task that predicts the magnitude of numbers in market comments @xcite . experiments show that our approach is efficient, achieving comparable, and even better performance than existing numeracy-preserving methods. ## related work numbers are ubiquitous and numeracy plays an important role in nlp applications and domains such as financial and clinical documents @xcite @xcite . however, most of existing work simply ignores the numbers in the pre-processing step @xcite and thus leads to suboptimal performance. see @xcite for an overview. @xcite studies different strategies to model numerals, and @xcite proposes a joint learning model for handling numbers in text. still, a recent work @xcite shows that common word embedding models cannot deal with numbers precisely. according to @xcite , most models fail to interpolate or extrapolate to oov numerals. the main reason that causes such poor performance with number-intensive tasks is that the existing word embedding methods are not specifically designed to capture numerical relationships. to handle number embeddings specifically, some new nlp models are proposed. one closely related work to ours is dice @xcite which devises an independent-of-corpus and deterministic approach to assign embeddings for numbers. however, dice derives numerical embedding based on engineered mathematical operations, which could be computationally costly for encoding a large number of numbers. our work differs from dice in that we infer numeracypreserving embeddings automatically from a specially designed knowledge graph. compared to dice, our approach is simple and efficient yet achieves comparable or even better performance. ## methods the high-level idea of our approach nekg is to preserve numeracy and numeric semantics (e.g., magnitude, addition) via knowledge graph. knowledge graph is a network of entities, their semantic types, properties, and relationships, built based on entity-relation triples @xcite . in our method, we make use of a simply structured knowledge graph consisting of only numbers and their magnitude relationships. we embed the knowledge graph in a vector space using a graph embedding method for obtaining embeddings of the number entities. numbers that are not in the original knowledge graph, i.e., out-of-vocabulary (oov) numbers, can be inferred by an interpolation method. see algorithm 1 for the full descrip-tion of our algorithm. the code is available at https://github.com/hduanac/nekg . ## experiments we evaluate nekg on two tasks: task 1 evaluates nekg on the ability to capture magnitude @xcite ; task 2 examines the numeracy of nekg on list maximum, decoding, and addition tasks @xcite . ## application of nekg on numeracy-600k numbers play an important role in financial documents. in this section, we evaluate nekg in a real-world financial prediction dataset: numeracy-600k @xcite . the numeracy-600k task is a standard numeracy task studied in prior nlp research. the goal of the prediction task is to classify number magnitude in market comments. specifically, given a sentence from market comments (e.g, aapl price is up 2% to $142), the task aims to predict the magnitude of the target num-ber, say 2%, using the surrounding texts and other numbers in the sentence. training details and results. the total number of magnitude classes in our experiment is 5. for example, the task defines the magnitude of numbers 0 ≤ @xmath5, magnitude of numbers 1 ≤ @xmath6, and so on. following (sundararaman et al., 2020), we implement a bi-lstm neural network to make predictions for target numbers that are masked by a certain random vector. the word embeddings are initialized with glove, and the numbers other than the target number in the sentence are initialized with nekg or dice. we include several baseline models where numbers are completely ignored, simulating a common practice of dealing with numbers in text. the prediction results are presented in ## conclusion in this paper, we propose a simple yet effective number embedding method nekg derived from knowledge graph. we evaluate our method on several numeracy-related tasks. the results show that nekg can achieve better performance than the existing number embedding methods and run significantly faster. with the release of nekg, we hope researchers and practitioners can utilize nekg to handle numbers effectively in their work.