Predicting Degrees of Technicality in Automatic Terminology Extraction (2020)

Abstract: While automatic term extraction is a well-researched area, computational approaches to distinguish between degrees of technicality are still understudied. We semi-automatically create a German gold standard of technicality across four domains, and illustrate the impact of a web-crawled general-language corpus on technicality prediction. When defining a classification approach that combines general-language and domain-specific word embeddings, we go beyond previous work and align vector spaces to gain comparative embeddings. We suggest two novel models to exploit general- vs. domain-specific comparisons: a simple neural network model with pre-computed comparative-embedding information as input, and a multi-channel model computing the comparison internally. Both models outperform previous approaches, with the multi-channel model performing best.

## introduction automatic term extraction, i.e. the task of extracting linguistic expressions characteristic to a specialized domain, is a long-researched field within natural language processing. assessing the technicality of the extracted terms, however, is still a niche within this area: technicality refers to the degree to which a term is specialized and exclusively used by experts in a domain. up to date, studies on term technicality are mostly restricted to medical terminology and relate to the communication between doctors and patients. especially in times of growing amounts of domain-specific websites with both lay and expert users (e.g. diy 'do-it-yourself' communities, such as 1-2-do.com), the communication between experts and lays becomes increasingly important across all specialized domains. furthermore, term technicality prediction is important for a range of tasks such as automatic thesaurus creation, assessing text specialization, and domain knowledge acquisition. above all, predicting technicality can be considered a more fine-grained and expressive form of terminology extraction. in this work, we first semi-automatically collect german specialized domain corpora to create a gold standard of term technicality across four domains: automotive, cooking, hunting and diy. based on a qualitative analysis of terminological phenomena and variants of ambiguity across domain-specific and general-language corpora, we then suggest two methods to explicitly integrate not only vector space model representations derived from the corpora, but also comparisons across the vector spaces. in a first approach, we enrich the combined general-language and domain-specific word embeddings with a difference vector as input for a classification system. in a second approach we design a multi-channel feed-forward neural network with a siamese network component to represent the vector comparison internally. ## related work existing studies on technicality predominantly focus on levels of familiarity or difficulty of terminology in medical, biomedical or health domains. term familiarity refers to a user's subjective understanding of term technicality. these studies typically rely on classical readability features such as frequency, term length, syllable count, the dale-chall readability formula and affixes @xcite @xcite . they further make use of domain-specific terminology attributes such as neo-classical word components, given that medical terminology is strongly influenced by greek and latin @xcite . besides the feature specification, the majority of studies exploits contrastive approaches. contrastive approaches compare a term's distribution in a domain and a reference corpus, for example a general-language corpus. furthermore, for technicality prediction, often expert (medical) texts are compared against reference lay texts. only a small number of studies relies on context-based approaches, e.g. @xcite use a contextual network; @xcite exploit language models; pérez (2016) compares collocation networks. for standard term extraction, contrastive techniques represent one of the main strands of methodologies, by comparing a term candidate's frequencies in a domain-specific and a general-language corpus @xcite @xcite @xcite @xcite . recent approaches use word embeddings trained separately on contrastive corpora; e.g. @xcite concatenate general and domain-specific word embeddings and use them as input for classifiers, such as a multilayer perceptron. similarly, @xcite and @xcite apply such a concatenation to represent a term in one language, as data enrichment pre-step for bilingual terminology extraction. in sum, approaches using contrastive corpora are popular in both automatic term extraction and term technicality prediction studies. the few approaches that use word embeddings as basis for a contrastive approach separately train word embeddings on general-language and domain corpora. in our work, we extend these methodologies by aligning vector spaces in order to more adequately represent meaning variation across corpora. ## definition of technicality according to @xcite , there is no consensus among researchers about what exactly defines technicality. they provide an overview of what characterizes technical vocabulary, and observe two main categories. on the one hand, technical terms often exhibit a narrow range of senses specific to the domain. they are only understood by a limited set of people, because they require domain knowledge. on the other hand, there are terms which are also frequently used in general language. these terms are ambiguous: they carry specialized meanings in a particular domain which are different to the general-language meanings. as ha and hyland (2017), we see technicality as a continuum. in the course of this paper, we adopt a simplified handling and distinguish between three broad classes of technicality: technical terms, basic terms and non-terms. ## data and gold standard creation data. we collect german texts for four domains: automotive, cooking, diy and hunting. besides including technical handbooks, we crawl topicspecific data from wikipedia foot_0 and similar resources such as cooking recipes from cooking homepages (e.g. kochwiki.de), and car repair and diy instructions from wikihow.de. as general-language reference corpus, we use sdewac @xcite , a cleaned version of the web-crawled corpus dewac @xcite . all corpora are lemmatized and pos-tagged with the treetagger @xcite , and reduced to content words (nouns, verbs and adjectives). we follow the preprocessing steps described in @xcite that led to the best results in that study. the corpus sizes are shown in gold standard. we select all words as term candidates with a minimum frequency of 10 in both the domain corpus and sdewac. the gold standard thus contains both simple and complex terms, the latter in the form of closed compounds. we did not extract multi-word terms other than closed compounds because we would have needed specific procedures to identify them (e.g. by chunking or by using association measures to identify valid collocations). even more importantly, multi-word expressions are prone to variation (e.g. one could say 'wood drill' or 'drill for wood') and it is likely to not find all variants in the glossaries and other resources we use to create the gold standard. instead of relying on labour-intensive human annotations, we determine the technicality labels semi-automatically. first, we collect domainspecific glossaries for each domain, i.e. textual glosses and specialized terms with their meanings foot_1 . these glossaries contain terms which require domain knowledge (especially if they are ambiguous) and thus need to be explained to a lay person, i.e. they contain technical terms. secondly, we collect thematic basic vocabulary lists (from thematic base vocabulary books, thematic vocabulary training lists for foreign apprentices, etc.). these lists contain the basic terminology of a domain, with a low level of technicality. finally, we collect indices and tables of contents of domain-specific handbooks, which include all kinds of terminological vocabulary foot_2 . we label the data as follows: the resulting sizes of the gold standards per domain are presented in ## models baselines as baseline, we use a decision tree classifier (dt) with three standard features commonly used for term familiarity prediction: frequency (corpus-size normalized), word length and character n-grams. further, we implement the approach by @xcite using a multilayer perceptron (mlp) and the concatenation of general-language word embeddings (gen) and domain-specific word embeddings (spec) of a term candidate as input (mlp, gen⊕spec), in comparison to using only one of the embeddings. we learn two separate word2vec sgns vector spaces @xcite for gen and spec. centering and batch normalization across neural models we apply batch normalization @xcite , which normalizes the output of a preceding activation layer by subtracting the batch mean and then dividing by the batch standard deviation. this reduces the effect of inhomogeneous input data, in our case the different domain corpora. we further length-normalize and apply element-wise column mean-centering to the embeddings, which has proven to be beneficial as preprocessing step for rotational alignment of vector spaces @xcite and as a general post-processing step for word embeddings @xcite . note that the reason for the beneficial effect of centering is still unclear. @xcite provide an intuitive explanation that centering moves randomly similar embeddings further apart, while @xcite consider centering as an operation making vectors "more isotropic", i.e., more uniformly distributed across the directions in the space. comparative embeddings and multi-channel model simple vector concatenation does not in-corporate any kind of comparison of the embeddings. we thus suggest two novel models to exploit general-vs. domain-specific comparisons: comparative embeddings (mlp, con⊕diff) use an mlp classifier and add a difference vector to the input vector concatenation gen⊕spec. since the word embeddings were trained separately on different corpora, this model requires an alignment of the vector spaces. we use a state-of-the-art alignment method @xcite , where the best rotation gw of a vector space g onto a vector space s is determined, with the rotation matrix w . w is computed as @xcite . after the alignment, unit length is applied again (since the vectors are not unit length after alignment anymore) and the absolute difference vector (diff) is computed. the concatenation vector gen ⊕ spec ⊕ diff is then taken as input to the model. as our second model, we use a multi-channel feed-forward neural network (multi-channel). the network takes as input the unaligned gen and spec vectors, and processes each gen and spec in a different channel. the third channel is a variant of a siamese network @xcite , a dual-channel network with shared weights. both gen and spec are processed through the shared weight layer, in order to map them onto the same space. then the element-wise absolute difference is computed, and the output of all three channels is concatenated. the network is defined as: ## results we use macro-precision, recall and f1-score for evaluation, to put more weight on the correctness of the smaller classes base term and technical term. the experiment results are shown in the multi-layer perceptron (mlp) results outperform the decision-tree (dt) baseline with standard term familiarity prediction features. using only a general-language vector gen for classification performs better than using only a domainspecific vector spec, and the concatenation of both the models integrating a notion of vector comparison perform best, with the multi-channel network achieving slightly better results than the mlp comparative embeddings. centering improves all but one results; i.e., it has an overall beneficial effect for our task. ## conclusion we semi-automatically created the first large-scale gold standard for technicality prediction across domains and proposed two novel neural network models to fine-tune automatic terminology extraction by distinguishing between degrees of technicality. the models integrate general-vs. domain-specific word embedding information in different ways. an adapted siamese multi-channel network model performed best, and centering has an overall beneficial effect on pre-processing the vector spaces.