An Abstract Specification of V ox ML as an Annotation Language (2023)

Abstract: VoxML is a modeling language used to map natural language expressions into real time visualizations using real-world semantic knowledge of objects and events. Its utility has been demonstrated in embodied simulation environmens and in agent-object interactions in situated human-agent communicative. It is enriched to work with notions of affordances, both Gibsonian and Telic, and habitat for various interactions between the rational agent (human) and an object. This paper aims to specify VoxML as an annotation language in general abstract terms. It then shows how it works on annotating linguistic data that express visually perceptible human-object interactions. The annotation structures thus generated will be interpreted against the enriched minimal model created by VoxML as a modeling language while supporting the modeling purposes of VoxML linguistically.

## introduction as introduced by @xcite , voxml is a modeling language encoding the spatial and visual components of an object's conceptual structure. 1 it allows for 3d visual interpretations and simulations of objects, motions, and actions as minimal models from verbal descriptions. the data structure associated with this is called a voxeme, and the library of voxemes is referred to as a voxicon. voxml elements are conceptually grounded by a conventional inventory of semantic types @xcite . they are also enriched with a representation of how and when an object affords interaction with another object or an agent. this is a natural extension of gibson's notion of object affordance @xcite to functional and goaldirected aspects of generative lexicon's qualia structure @xcite , and is situationally grounded within a semantically interpreted 4d simulation environment (temporally interpreted 3d space), called voxworld @xcite . voxml has also been proposed for annotating visual information as part of the iso 24617 series of international standards on semantic annotation schemes, such as @xcite and @xcite . voxml, as an annotation language, should be specified in abstract terms, general enough to be interoperable with other annotating languages, especially as part of such iso standards, while licensing various implementations in concrete terms. in order to address these requirements, this paper aims to formulate an abstract syntax of voxml based on a metamodel. it develops as follows: section 2, motivating voxml as an annotation language, section 3, specification of an annotation scheme, based on voxml, section 4, interpretation of annotation-based logical forms with respect to the voxml minimal model, and section 5, concluding remarks. ## motivating voxml as an annotation language interpreting actions and motions requires situated background information about their agents or related objects, occurrence conditions, and enriched lexical information. the interpretation of base annotation structures, anchored to lexical markables for annotating visual perceptions, depends on various sorts of parametric information besides their associated dictionary definitions. a significant part of any model for situated com-munication is an encoding of the semantic type, functions, purposes, and uses introduced by the "objects under discussion". for example, a semantic model of perceived object teleology, as introduced by generative lexicon (gl) with the qualia structure, for example, @xcite , as well as object affordances @xcite is useful to help ground expression meaning to speaker intent. as an illustration, consider first how such information is encoded and then exploited in reasoning. knowledge of objects can be partially contextualized through their qualia structure @xcite , where each qualia role can be seen as answering a specific question about the object it is bound to: formal, the is-a relation; ## interpretation with respect to the voxml minimal model voxemes in voxml create a minimal model. each of the annotation-based semantic forms, as in ( 10 ), is interpreted with respect to this minimal model by referring to its respective voxemes. ## concluding remarks the paper specified the voxml-based annotation scheme in formal terms. the example of the action of mary picking up a glass from the table showed how that particular example was annotated and how its logical forms were interpreted with a voxml model while referring to the voxicon. each voxeme in the lexicon, especially that of objects, contains information enriched with the notions of habitat, affordance, and embodiment. as the voxicon develops into a full scale, the task of interpreting annotated language data involving complex interactions between humans and objects can easily be managed. for purposes of exposition, the discussion here focused on the annotation of one short narrative in english involving one verb, pick up, and one object, glass. the proposed voxml-based annotation scheme needs to be applied to large data with a great variety to test the effectiveness of interpreting its annotation structures and corresponding semantic forms against the voxml model. at the same time, such an application calls for the need to enlarge the size and variety of the voxicon for modeling purposes as well. the evaluation of the voxml-based annotation scheme and the extension of the voxicon remain as future tasks.