Semantic Ambiguity Detection in Sentence Classification using Task-Specific Embeddings (2023)

Abstract: Ambiguity is a major obstacle to providing services based on sentence classification. However, because of the structural limitations of the service, there may not be sufficient contextual information to resolve the ambiguity. In this situation, we focus on ambiguity detection so that service design considering ambiguity is possible. We utilize similarity in a semantic space to detect ambiguity in service scenarios and training data. In addition, we apply task-specific embedding to improve performance. Our results demonstrate that ambiguities and resulting labeling errors in training data or scenarios can be detected. Additionally, we confirm that it can be used to debug services

## introduction the ability to accurately access the meanings of sentences is a key component of voice recognitionbased agent services. this task is made difficult by the inherent ambiguity of some sentences, which can refer to different meanings in different contexts. adot( ) foot_1 is a voice recognition-based service agent, akin to amazon's alexa, for korean users. this paper describes how we dealt with ambiguity, from a perspective of adot's developer. for example, as illustrated in figure 1 , the korean word "앞" has two different meanings depending on the context. as a result, when the media content is being played, the command "앞으로 가봐" may have two completely opposite meanings.("move forwards" vs. "move to previous"). in a test conducted within our company, approximately 61.7 percent of respondents interpreted it as "move forwards" and 38.3 percent as "move to previous". although there is no universally valid interpretation in this scenario, each respondent was certain that their own interpretation was the correct one. (likert-scale @xmath0.2/7). this semantic ambiguity leads to a labeling error of training data and scenarios in specification 3 . from the perspective of service providers who must provide specific services in response to user input, we contend that this is a challenging problem. in addition, ambiguity is very important in terms of cost. manual inspection of the training data or specification is required for ambiguity handling, which is expensive in terms of cost and resources. recently, many researchers have attempted to handle ambiguity with various approaches such as semantic space (rodd, 2020), entity linking @xcite , and attribute attention @xcite . in addition, the primary purpose of these studies is to resolve ambiguity using context. in spite of this, we do not have sufficient contextual information to apply these disambiguation methods because of the characteristics of a very short input sentence (which has 6.97 korean letters and 2.05 words on average) and a one-turn based service. although contextual information required for disambiguation is not available within the scope of the sentence classifier, we can still exploit nonlinguistic traits, such as user habits and device information, which are accessible from the outside classifier to determine service. therefore, we chose to approach this problem through detection rather than disambiguation. detecting the presence of ambiguity in scenarios would allow for the consideration of external contextual information when providing the service. this paper proposes a process of detecting ambiguity in scenarios or training data through similarity. to improve ambiguity detection performance, we apply task-specific embedding. we conduct an experiment detecting ambiguity and labeling error from ambiguity in adot's scenarios and training data. additionally, we investigate pearson correlation coefficients between similarity and the degree to which users expect two sentences to belong to the same class (this is referred to as user-aware class relevance). this study's findings affirm that similarity provides a means to identify scenarios and training data with potential ambiguity. in this process, we confirm that user-aware class relevance correlates with the similarity. furthermore, it was confirmed that the training data that resulted in the misclassification could be specified. ## related works ambiguity is a long-standing problem in natural language processing (nlp) tasks such as word sense disambiguation @xcite , entity disambiguation @xcite , and database search result disambiguation @xcite in the taskoriented dialogue systems. @xcite utilized domain-specific data to address the structural ambiguity of sentences. this ambiguity is largely divided into four categories @xcite . these studies are focused on resolving the ambiguity. however, we did not have contextual information to apply to these disambiguation methods. in nlp, the sentence similarity task, which evaluates the similarity between pairs of sentences using several techniques, has been investigated extensively. traditionally, edit distance has been used to quantify superficial similarities @xcite . despite being straightforward, it is noteworthy because of its similarity to manual identification of training data that induce misclassification. recent studies have measured the similarity between pairs of sentences by projecting them into a meaningful semantic space using various neural networks, such as word2vec @xcite , glove @xcite , elmo @xcite , and bert @xcite , or using siamese architectures, such as sentence-bert @xcite . in this paper, inspired by @xcite , we utilize a task-specific embedding, extracted from a trained classification model, to identify semantic ambiguities. ## methods -task-specific embeddings we note that each node of the trained classification model stores weights adjusted for the task, and the values output by each node in the inference process contain fragmentary information useful for inference @xcite . we create task-specific embedding vectors using the output values of each node in the classification model. via this process, we expect to be able to create an embedding with better expressive power for ambiguity detection, albeit biased, compared to off-the-shelf embeddings, such as pre-trained lm or word2vec. assuming a classification model f θ with n layers on top of the embedding layer, we create a vector v i representing the i-th layer as follows: (v 1 = embedding vector, v @xmath1.) whole layer vector (wlv) wlv consists of output of all nodes from the model's embedding layer to the layer below the final output layer. wlv contains the most amount of data and reflects all information flows in the model. after representation layer vector (arlv) wlv contains considerable information, but the embedding layer accounts for most of it, which is problematic (for instance, 25,640 nodes out of a total of 40,424 nodes are included in the embedding layer of adot's classification model). arlv is defined using nodes from the layer following the embedding layer to the layer immediately preceding the output layer to avoid undue influence of the embedding layer. conclusion layer vector (clv) although wlv and arlv contain a significant amount of data, they ignore discrete functions and consequently struggle with the corresponding differences. for example, when a layer uses an activation function such as relu, the difference in the value input to the function might be negligible. however, the resultant value after processing the function might be completely different. clv is only defined using the layer immediately preceding the output layer of the entire model to account for these characteristics. we anticipate that the overall model's judgment process is structured in this layer. @xmath2.1 datasets ## discussions powerful task-specific representation this study demonstrates the feasibility of detecting ambiguity through similarity and highlights the potential benefits of utilizing task-specific embedding to enhance performance. the experimental results show a positive correlation between model size and detection performance according to the scaling law. future research efforts for ambiguity detection will focus on generating task-specific embeddings using larger models for superior results. ## conclusion we introduce two processes to detect training data and scenarios that induce ambiguity. moreover, task-specific embedding is adopted to improve detection performance. comprehensive analysis reveals that compared to off-the-shelf embeddings, sent2vec and kogpt2, task-specific embedding is much more suitable for ambiguity detection. moreover, the results demonstrate the viability of identifying training data that cause misclassification. in the future work, we aim to compare the performance of task-specific embedding using a wider range of models and node selection methods. side-effect test set this set is a test set to evaluate the side effects caused by label modification. through modification, even if it is possible to correctly classify misclassified utterances, the classification results of utterances that have been accurately analyzed should not be changed. we collected 20,000 utterances that the current classifier accurately analyzes and expect the data to be analyzed accurately even after modification.