Evaluation of Universal Semantic Representation ( USR ) (2023)

Abstract: Universal Semantic Representation (USR) is designed as a language-independent information packaging system that captures information at three levels: (a) Lexico-conceptual, (b) Syntactico-Semantic, and (c) Discourse. Unlike other representations that mainly encode predicates and their argument structures, our proposed representation captures the speaker’s vivakṣā- how the speaker views the activity. The idea of “speaker’s vivakṣā is inspired by Indian Grammatical Tradition. There can be some amount of idiosyncrasy of the speaker in the annotation since it is the speaker’s view- point that has been captured in the annotation. Hence the evaluation metrics of such resources need to be also thought through from scratch. This paper presents an extensive evaluation procedure of this semantic representation from two perspectives (a) Inter- Annotator Agreement and (b) one downstream task, namely multilingual Natural Language Generation. We also qualitatively evaluate the experience of natural language generation by manual parsing of USR, so as to understand the readability of USR. We have achieved above 80% Inter-Annotator Agreement for USR annotations and above 80% semantic closeness in multi-lingual generation tasks suggesting the reliability of USR annotations and utility for multi-lingual generations. The qualitative evaluation also suggests high readability and hence the utility of USR as a semantic representation.

## introduction semantic representations (semrep henceforth) generally encode predicate-argument structure of a verb @xcite and @xcite , framenet @xcite along with some other grammatical information ranging from lexico-syntactic level information such as tense-number-person (amr @xcite , mrs @xcite to discourse level information such as topicfocus, co-referencing and discourse relations (pdt @xcite , ucca @xcite . however, no semantic representation, that we are aware of attempts to capture what we term as the speaker's vivaks . ā -how the speaker views the activity. we design a universal semantic representation (usr) that encodes "speaker's vivaks . ā". the idea is inspired from the indian grammatical tradition (igt henceforth). igt views languages as a holistic phenomenon. words are not derived as isolated units in pān . ini's grammar, but as units that are semantically connected with other words in the sentence @xcite . sentences are connected across the discourse. this is explicitly recognized by the paninian rule (a 2.1.1) : samarthah . padavidhih . this paper presents the evaluation metrics of usr from two perspectives: (a) the reliability of usrs (b) the utility of usr in the domain of multi-lingual generation. sentences are generated in multiple languages to check the universality of information encoded in usrs. we use simple reliable measures to evaluate and understand these properties. the quantitative evaluation metrics are presented from two perspectives: (a) the reliability of usrs, (b) the utility of usr. the utility is evaluated for multilingual generation measured through semantic textual similarity (sts). we use simple reliable measures to evaluate and understand these properties. the qualitative evaluation focuses on the usability of usrs in terms of readability of usr to generate natural language is examined. we also verify the adequacy of usr by manually generating natural language sentences from usrs. a detailed analysis validates the proposed system as well as indicates areas of improvement. the feedback from these analyses is important for improving the information content and representation strategy of usrs. section 2 presents the design of usr. section 3 studies evaluation metrics in the context of other related works. section 4 describes the quantitative evaluation metrics with results; while the qualitative measure is recorded in section 5. ## design of usr unlike other representations that mainly encode predicates and their argument structures, the proposed representation captures the speaker's vivaks . the activity of 'seeing' licenses 3 an animate seer and a seen entity. that is the semantic frame for 2 śabdes . varthadānābhiprāyo vivaks . ā "vivaks . ā is the intention of the speaker with regard to the meaning to be conveyed by the words" @xcite . @xcite has also defined the term "vaktur-vivaks . ā", in the same sense . as per "vivaks . ātah . kārakān . i" @xcite ) kāraka roles in a sentence also apply according to the desire of the speaker. the verb that every human being who knows the meaning of 'seeing' knows. but in communication, along with choosing the appropriate semantic frame, there occur two other important factors: (a) how the speaker conceptually cognizes the situation? (b) which linguistic expressions are available to translate that cognition into languages. for example, in the above examples, does the speaker want to express mira's agency, or does (s)he want to foreground the appearance of the moon over the seer's agency? this is termed as the speaker's vivaks . ā. depending on that, the speaker would choose the best appropriate linguistic expressions to convey his/her thoughts. our application task, namely natural language generation (nlg) also motivates the requirement of capturing the speaker's vivaks . ā in semrep. in order to generate a coherent and cohesive text, we require generative cues. speaker's vivaks . ā motivates those cues and we have decided to capture them in usrs through morphosemantics and dependency relations intra-sententially and also through discourse-level information. usr encodes information at three basic levels: (a) lexico-conceptual (b) syntactico-semantic and (c) discourse. this semantic information in usr is organized as features (in rows) and values, where the discourse relation and discourse co-referencing are accomplished through inter-usr linking which is established through sentence_id. word_index anchoring as shown in lexico-conceptual level: conceptual information which is generally expressed in terms of atomic words, multiword expressions or derived words are captured at this level. currently, this level has information at 4 layers in usr. these layers (or rows) are (i) concept row; (ii) semantic category; (iii) morpho-semantic and (iv) speaker's view. each entry to the concept row is an unambiguous representation of a concept. the ambiguity of a word is resolved in a very unique way in usr. many semreps use wordnet sense id as concepts. we propose to represent a concept in a multilingual set-up. for example, the lexeme in hindi pad . ha expresses two concepts: 'study' (as in the boy studies in 7th standard) and 'read' ('the boy reads a book'). this kind of ambiguity is handled at the conceptual level in the concept dictionary. this dictionary concept sanskrit hindi english bangla row pad . ha_1 pat . ha_1 pad . ha_1 read_1 par . a_1 pad . ha_2 adhi+ī_1 pad . ha_2 study_1 par . a_2 usr has the concept label entry in the concept row which during generation selects concepts from the respective language cell depending on which language to be generated. in the current concept dictionary, there are 142037 labels for which hindi and english concept labels are specified. for, 130948 concepts, sanskrit labels are also attested in the dictionary. at the lexico-conceptual level, the semantic category row specifies the semantic category of a concept. currently, four generic named entity categories are being annotated, namely-per(son), org(anisation), place and other. apart from that, we mark animacy and mass categories. syntactico-semantic level: two types of relations capture information at this level: kāraka and kāraketara ('other than kāraka') (kulkarni 2010) at the dependency row. pān . ini's system of knowledge representation is based on kāraka theory. there are six kārakas pointing out the relations between an event (or state) and its participants. they are kartā, karma (object), karan . a (instrument), sampradāna (beneficiary), apādāna (source) and adhikaran . a (time and location of action). kāraketara relations include relations between (a) noun and its modifiers; (b) verb and its verbal modifiers. there are a total of 42 dependency relations postulated till now in our work. discourse level: language as a mode of communication always occurs as a discourse in which a sentence generally has a connection or trace with the previous and following sentence. discourse relations map such inter-sentential information which forms a coherent text. co-reference is another discourse strategy to indicate two entities within a sentence or across sentences having the same referent. in usr, all intra-sentential discourse information is encoded in the dependency row and intersentential discourse information is captured in the discourse row. currently, we are representing a few inter-sentential discourse relations as described in das (2016) following igt. they are pratibandha (if. . . then), samānkāla (when. . . then), kāran . akāryabhāva (although), hetu-hetumadabhāva (because), asāphalaya (but), anantarkālinatva (then). more relations are being identified and a contrastive study with rst and pdtb tagsets are also being carried out. at present, if no explicit relation across usrs is marked, the default relation 'and' is presumed. ## related works on evaluation evaluation of semantic representations is a multidimensional task due to many qualitative parameters that need to be evaluated. usual parameters of interest are the utility of the semrep, invariance, universality (cross-lingual potential), usability, computational efficiency etc @xcite . human evaluation is one of the important methods for measuring the accuracy of generation tasks. a human evaluator can determine the accuracy, give a qualitative ranking based on the naturalness/fluency as well as completeness of information encoded in a given semantic representation. several human evaluation based methods are in practice such as the wmt tasks @xcite , direct assessment @xcite , hume @xcite for ucca, hter @xcite , or smatch @xcite applicable to amrs. human evaluations, besides being more accurate for semrep evaluations, can also mark strengths and weaknesses of the generation, further indicating possible improvements. however, human evaluation would usually require skilled annotators as well as well-designed annotation guidelines to en-sure objectivity. hence, human evaluation is effective but can be resource and time-inefficient @xcite . human evaluation reliability and consistency are measured through inter-annotator agreement (iaa). automated evaluations are the alternative to human evaluations, as they can be consistent, as well as resource efficient. however, the notion of semantic similarity is still not fully captured by the machine. several word based, vector based and embeddings based measures are available for the same @xcite . in this paper, we attempt to strike a balance between both human and automatic evaluation of usr and propose two kinds of evaluation: (a) qualitative and (b) quantitative. type exp name quality dataset measure parameter quantitative iaa reliability geo_simple human evaluation -agreement %, cohen's kappa quantitative nlg utility correctness, geo_6 pairwise cosine with embeddings completeness qualitative generation usability/ geo_6 + human evaluationexperience readability verified_sentences effort, difficulty level we have extensively used the idea of semantic textual similarity (sts) in our evaluations, measured through human evaluation as well as by standard measures like pairwise cosine similarity. here, we build a usr for a reference sentence r, then use that usr to either manually or automatically generate a sentence (g). if r and g are semantically close, we can say that the usr correctly and adequately captures the reference sentence meaning. ## qualitative evaluation it is important to understand and record the experience of people involved in creating and using usrs. we are particularly interested in the readability of usr, because the idea is to create a gold standard usr bank which is only possible when human annotators can effortlessly read usr and correct it as needed. in this paper, readability is tested in terms of correctness and ease of generating sentences from a usr. if a human generator succeeds in generating a correct sentence with minor or no assistance, that shows that the usr is readable as well as adequate for correct sentence generation. we conducted a study and the following survey to check the readability of usrs by human beings. human generators (14) with mixed prior knowledge and experience with usrs are given the manual generation task. the experience distribution of generators is as given in sts scores, measuring accuracy, for reference sentence and generated sentence were computed. a cross-sectional view of the correctness vs the complexity level is given in figure 1 . it is evident that generators could produce a high number of semantically correct (same meaning, and minor variations in meaning) sentences. the errors seen were mostly missing terms like 'almost, 'may-be', gnp and tam (past vs present) variations. for example: for the reference sentence (translated): sun is about 15 million km away from the earth. some generators did not include the word 'about'. figure 2 clearly indicates that the human generators could find the desired help in the usr guidelines. most human generators found the usr guidelines exhaustive and could use the document to clarify their doubts. the help was mostly sought for the dependency relations, as the list of dependencies is exhaustive, and remembering all can be an arduous task for a novice. of the reported consultation of the usr guidelines, novice generators @xmath0. the generators were also ## conclusion in this paper, we have introduced a new semrep called universal semantic representation (usr). this is a very new initiative that attempts to capture the speaker's vivaks . ā and is inspired from indian grammatical tradition. the lexico-conceptual, syntactico-semantic and discourse level information is encoded in a structured format in which usrs are interlinked to express the meaning of discourse as a whole. this paper presents the design of the usr and also records its detailed, multidimensional evaluation for reliability and its utility for natural language generation. empirical evidence suggests high reliability as well as reliable semantic similarity scores for natural language generations done in multiple indic languages namely hindi, bangla and telugu. the qualitative evaluation strongly suggests that usr is easy to read and use with some training. thus usrs are suitable for natural language generation tasks, and can be used as a universal semantic representation.