Unsupervised Methods for Domain Specific Ambiguity Detection. The Case of G erman Physics Language (2023)

Abstract: Many terms used in physics have a different meaning or usage pattern in general language, constituting a learning barrier in physics teaching. The systematic identification of such terms is considered to be useful for science education as well as for terminology extraction. This article compares three methods based on vector semantics and a simple frequency-based baseline for automatically identifying terms used in general language with domain-specific use in physics. For evaluation, we use ambiguity scores from a survey among physicists and data about the number of term senses from Wiktionary. We show that the so-called Vector Initialization method obtains the best results.

## introduction in science, it is common to refer to specific concepts using terms which are also used in everyday language but with a more specific or different meaning. at the same time, terms from science are assimilated into general language, often with a transformed meaning and use. since these terms have a domain-specific use within science, they are a potential source of ambiguity, generating problems for successful communication and learning. more specifically, in science education, it has been found that students' conceptions are often related to terms' general meanings non-congruent with the scientific ones @xcite . in physics teaching, for example, words like work, energy, momentum, impulse, power, and mass have a narrower definition and a meaning that often differs entirely from the one used in everyday language @xcite . @xcite argue that terms with multiple meanings are more difficult to learn, demanding further negotiation, expansion, and correct contextualization of their meanings. they also showed in a study with words from the chemistry teaching (e.g., solution, polar, and compound) that disassociating the scientific meaning from the one already acquired in everyday life is often hard. moreover, @xcite show that students' ability to distinguish the different senses of a term correlates with test scores in the corresponding discipline. by recognizing that terms with different meanings and uses in science and general language represent a learning barrier, their automatic identification within a discipline becomes a relevant task, supporting awareness of their use in teaching @xcite @xcite , or even supporting specific teaching strategies for these cases @xcite . in natural language processing (nlp), identifying semantic differences between domains (synchronic lexical semantic change) is similar to identifying lexical changes in time (diachronic lexical semantic change). in both cases, we can use properties of word embeddings to detect shifts in the relative positions in the embeddings space. the synchronic lexical semantic change has recently received attention in engineering requirements @xcite @xcite for the detection of potential sources of ambiguity. this task is also investigated in terminology extraction @xcite , where statistical measures might not identify terms commonly used in specific and general contexts as part of a field's terminology. since word embeddings give a concise representation of a word's use (and, according to the distributional hypothesis, the meaning), many authors use them to study the domain-specific meaning of words. however, when computing word embeddings from two different corpora, we will end up with incomparable embedding spaces. the main differences in the proposed approaches deal with the solutions used to overcome this problem. the present paper aims to compare three methods and a simple baseline for identifying general language words with a deviant meaning in physics. for the evaluation, we use two data sets, one obtained by collecting expert judgments on a small number of nouns and a larger data set derived from wiktionary. to the best of our knowledge, this is the first evaluation of the automatic identification of general terms with a specific meaning in the science education domain. ## related work as mentioned above, we cannot immediately compare word embeddings derived from different corpora since the dimensions are randomly initialized, and the same dimensions in the two models will not correspond. the method vector initialization @xcite was the first to use neural network embeddings and solves this problem by initializing the embeddings' training from the previous corpus embeddings, and then comparing the position of the embedding before and after training. the hypothesis is that the embeddings' displacement after training reflect the word lexical change. this method was initially aimed at diachronic lexical change identification but can also be used for synchronic lexical change. specific for the synchronic lexical change identification, @xcite and @xcite use a variation of the vector initialization method to investigate lexical ambiguity between specific domains: they use marked target words before further training the embeddings. however, they need to select target words for the analysis based on their frequencies in both domain-specific corpora. other authors proposed to make the vector spaces comparable by defining a linear transformation between embedding spaces based on the solution of the orthogonal procrustes problem @xcite @xcite . in the orthogonal procrustes analysis, a mapping matrix is determined using singular value decomposition that rotates one of the vector spaces. the optimal alignment is the one that minimizes the distances between the embeddings of the same word in both vector spaces. @xcite added a pre-processing step to the procedure: the alignment of the mean center of the vector spaces before determining the mapping matrix. the method proposed by @xcite , named here as similar words, indirectly measures the similarity of embeddings from different spaces: they generate two lists of the most similar words using two vector spaces for a target word. finally, they compute a rank correlation between the two lists to get an ambiguity score. the methods above are based on static embeddings, like word2vec (skip-gram). this way, we obtain a general word representation in each context. @xcite use dynamic embeddings, using the average embedding of up to 1000 bert embeddings in the corpus. they use a supervised regression model to identify domain-specific terms. therefore we cannot compare their results to those from the unsupervised approaches discussed above. martinc et al. (2020) also use averaged contextual embeddings. they use the same fine-tuned bert embeddings for the general and domain-specific corpus but take the average for examples from each corpus separately. thus the two averages obtained for each corpus are in the same embedding space and can be compared immediately. beyond qualitative evaluation, authors evaluate their method's results using manually created rankings @xcite @xcite . @xcite only evaluated his method and ambiguity between specific domains, whereas @xcite systematically compared methods but, for synchronic lexical change, evaluated the methods only with the ranking of a few words used in general language and in the context of cooking. @xcite evaluated two methods (their own regression model and the unsupervised approach from @xcite ) on three different domains, generating lists of domain-specific terms. these lists were then evaluated manually, using precision as an evaluation measure; however, recall could not be assessed. ## experimental setup in the following, we will present some (technical) information about our implementation of four methods for the identification of terms in general language with a specific sense in physics: vector initialization, orthogonal procrustes, similar words and relative frequency. the general idea behind the methods we compare was already described in section 2. we generated three vector spaces using the skip-gram method from gensim ( řehřek and sojka, 2011 ). the first (model 1) is obtained from a general language corpus. the second (model 2) follows the approach proposed by @xcite : we train the model on the physics corpus, but initialize all vectors withe the embeddings from model 1. the third vector space (model 3) is obtained from the physics corpus alone. the embeddings have 200 dimensions, and the window size used in training was 5. in all cases we computed embeddings only for words occurring at least 10 times. for the vector initialization method, we calculate the cosine value between the embeddings from model 1 and model 2. words with the lowest cosine values will assumably have the most significant semantic displacement. for the orthogonal procrustes method, we align the vector spaces from models 1 and 3 after the pre-processing step proposed by @xcite . after alignment, we also calculate the cosine values between the embeddings, expecting words with the same usage pattern in the two contexts to be more aligned after the procedure. finally, for the similar words method, the ambiguity score will be determined by comparing the most similar words of the terms from models 1 and 3. to evaluate the methods, as a simple baseline, we also sort the words according to the relative frequency, assuming that all words frequently used in physics have a specific meaning in this domain. ## data identification of terms with domain-specific meaning requires two corpora, a general language corpus and a domain-specific corpus. the general language corpus should be, in principle, non-specific and large, representing, to some extent, everyday language. the domain-specific corpus (physics corpus) reflects the communicative context of our interest, namely physics teaching. for the present study, we use a german news corpus denews2020 @xcite . for the specific corpus, we use a corpus of german texts on physics, mostly highlevel textbooks @xcite . however, for a more solid base for evaluation, we collected a set of 48 nouns, including the words mentioned above, along with more problematic and also unproblematic terms. in a survey, we asked participants to what extent, on a scale from 1 (same meaning) to 5 (totally different meaning), the meaning of a term differs in everyday use and the physical context. the ambiguity score for each term is the mean value of the answers in the survey. 14 subjects completed the survey. they were german native speakers with at least a master's degree in physics or physics teaching, including teachers, physicists, and science education researchers. we used survey data for evaluating the methods using the pearson correlation between the ambiguity score and the metric obtained for each word from the methods. for a second experiment, we collected words from wiktionary and counted how many senses for a word are marked as being specific for physics. e.g., a word like kraft (force) appears with four senses in wiktionary, one of them referring to physics. since we compare senses that differ between physics and general language, we evaluated the ranking generated with each method, from potentially more to less ambiguous. we take from wiktionary the binary information, no physics sense (0), and one or more sense in physics (1). then, we calculate the area under the curve (auc) to evaluate the ranking with this binary information. the total of words used (shown in ## results figure 2 displays the words of the survey with their averaged survey score and computed score. the participants did not perceive platte (board) as ambiguous. looking at some sentences with the term, we observe that this term is used in german very often referring to music albums. we believe the participants would hardly consider this sense while answering the survey. figure 3 shows the effect of the vector initialization method, displaying two terms on their original position and on their position after continued training on the tecophy corpus. we see that the terms move in the direction of other typical physics terms. ## conclusion lexical ambiguity is a general challenge in communicative situations and an important issue in science education. identifying domain specific ambiguity is needed to support the appropriate use of language in teaching and specific methodologies for terminology acquisition. in our research, the vector initialization method proved to be the most effective for identifying lexically ambiguous words. this method achieved the highest pearson correlation with the survey and auc calculated with the wiktionary data. however, in a different study by @xcite , the vector initialization method performed poorly when ranking 22 target words. such conflicting results may be due to the differences in the tasks involved, namely ranking target words versus automatically identifying lexical change. moreover, identifying lexical changes can aid in terminology extraction @xcite , since it can uncover terms with a specialized meaning within a particular domain, despite being frequently used in general language. such terms may not be found purely based on their frequency. our research shows that the vector initialization method holds more promise than orthogonal procrustes as an additional technique for terminology extraction in science education. a direction for future work is to bring the method to individual occurrences of a word: when finding an instance of an ambiguous word, we would like to be able to see whether the general or domain-specific meaning of the word is intended. this could finally help to see whether students use a word in the correct sense or whether they are misled by the everyday meaning of a specific term. for this purpose, we plan to explore methods based on contextual embeddings and evaluate their applicability to science education.