T ex S mart: A System for Enhanced Natural Language Understanding (2021)

Abstract: This paper introduces TexSmart, a text understanding system that supports fine-grained named entity recognition (NER) and enhanced semantic analysis functionalities. Compared to most previous publicly available text understanding systems and tools, TexSmart holds some unique features. First, the NER function of TexSmart supports over 1,000 entity types, while most other public tools typically support several to (at most) dozens of entity types. Second, TexSmart introduces new semantic analysis functions like semantic expansion and deep semantic representation, that are absent in most previous systems. Third, a spectrum of algorithms (from very fast algorithms to those that are relatively slow but more accurate) are implemented for one function in TexSmart, to fulfill the requirements of different academic and industrial applications. The adoption of unsupervised or weakly-supervised algorithms is especially emphasized, with the goal of easily updating our models to include fresh data with less human annotation efforts.

## introduction the long-term goal of natural language processing (nlp) is to help computers understand natural language as well as we do, which is one of the most fundamental and representative challenges for artificial intelligence. natural language understanding includes a broad variety of tasks covering lexical analysis, syntactic analysis and semantic analysis. in this paper we introduce texsmart, a new text understanding system that provides enhanced named entity recognition (ner) and semantic analysis functionalities besides standard nlp modules. compared to most previous publiclyavailable text understanding systems @xcite @xcite @xcite , texsmart holds the following key characteristics: second, texsmart provides two advanced semantic analysis functionalities: semantic expansion, and deep semantic representation for a few entity types. these two functions are not available in most previous public text understanding systems. semantic expansion suggests a list of related entities for an entity in the input sentence (as shown in figure 1 ). it provides more information about the semantic meaning of an entity. semantic expansion could also benefit upper-layer applications like web search (e.g., for query suggestion) and recommendation systems. for time and quantity entities, in addition to recognizing them from a sentence, texsmart also tries to parse them into deep representations (as shown in figure 1 ). this kind of deep representations is essential for some nlp applications. for example, when a chatbot is processing query no. ## system modules compared to most other public text understanding systems, texsmart supports three unique modules, i.e., fine-grained ner, semantic expansion and deep semantic representation. besides, traditional tasks supported by both texsmart and many other systems include word segmentation, part-of-speech (pos) tagging, coarse-grained ner, constituency parsing, semantic role labeling, text classification and text matching. below we first introduce the unique modules, and then describe the traditional tasks, followed by system usage. ## system usage two ways are available to use texsmart: calling the http api directly, or downloading one version of the offline sdk. note that for the same input text, the results from the http api and the sdk may be slightly different, because the http api employs a larger knowledge base and supports more text understanding tasks and algorithms. the detailed comparison between the sdk and the http api is available in https://ai.tencent.com/ ailab/nlp/texsmart/en/instructions.html . offline toolkit (sdk) so far the sdk supports linux, windows, and windows subsystem for linux (wsl). mac os support will be added in v0.3.0. programming languages supported include c, c++, python (both version 2 and version 3) and java (@xmath0.6.0). example codes for using the sdk with different programming languages are in the ./examples sub-folder. for example, the python codes in ./examples/python/en_nlu_example1.py show how to use the texsmart sdk to process an english sentence. the c++ codes in ./examples/c_cpp/src/nlu_cpp_example1.cc show how to use the sdk to analyze both an english sentence and a chinese sentence. http api the http api of texsmart contains two parts: the text understanding api and the text matching api. the text understanding api can be accessed via http-post and the url is available on the web page. 4 the text matching api is used to calculate the similarity between a pair of sentences. similar to the text understanding api, the text matching api also supports access via http-post and the url is available on the web page. foot_5 4 system evaluation ## conclusion in this paper we have presented texsmart, a text understanding system that supports fine-grained ner, enhanced semantic analysis, as well as some common text understanding functionalities. we have introduced the main functions of texsmart and key algorithms for implementing the functions. we have also reported some evaluation results on major modules of texsmart.