Beyond emojis: an insight into the IKON language (2022)

Abstract: This paper presents a new iconic language, the IKON language, and its philosophical, linguistic, and graphical principles. We examine some case studies to highlight the semantic complexity of the visual representation of meanings. We also introduce the Iconometer test to validate our icons and their application to the medical domain, through the creation of iconic sentences.

## introduction since its introduction in the early 1970s, textual computer-mediated communication (cmc) has been enriched by visual elements that express emotion and attitude: emoticons (sideways faces typed in ascii characters), emojis (designed, like emoticons, to facilitate emotion expression in textbased conversation, but visually richer, more iconic, and more complex), stickers (larger, more elaborate, character-driven illustrations, or animations to which text is sometimes attached) @xcite . graphic symbols have been extensively utilized in communication all over the globe, particularly on social media and instant messaging services. more recently, studies have examined the use of emojis in other dimensions. for example, consider the usage of emojis or symbols to gauge consumer satisfaction with a product or service in the business field @xcite . emojis have been investigated in the medical industry to assess patients' symptoms @xcite . apart from emojis -that are not considered a language by most linguists -there are also visual languages, that were created to enable a full visual communication (e.g. bliss, zlango, iconji, etc.). nonetheless, several limitations have been found in these visual communication tools. for example, some of them are based on national languages reproducing their inconsistencies and difficulties; some were conceived to be handwritten and so are very stylized and abstract; some have a too simple grammar, that does not allow sufficient precision in conveying complex meanings. the ikon language was conceived to address these limitations. ikon allows semantic compositionality by joining icons (as in bliss, locos and piktoperanto), the use of grammar categories (bliss), and the consistent use of iconemes (as in vcm), high iconicity (as in aac languages and emoji). ikon aims to reduce abstractness and language dependency. our contribution has multiple aims: i) to examine ikon theoretical approach and its application to a few case studies based on semantic dimensions such as modality, verbs of motion, of perception, and of communication; ii) to present the iconometer test, a crucial tool to understand how individuals interpret ikon language; iii) to propose an application of ikon in the medical domain through a bachelor's thesis project developed by a member of our team. the remainder of the paper is structured as follows: section 2 briefly discusses examples of iconic languages and their semantic approach. section 3 specifies ikon's theoretical approach. section 4 brings a few case studies of our icons. section 5 presents the iconometer, the next step to evaluate the designed icons. section 6 describes ikon sentences in the dentist-patient frame. finally, conclusions are reported in section 7. ## semantic analysis in icon languages iconic languages have been used successfully in human-computer interfaces, visual programming, and human-human communication. they have, in most cases, a limited vocabulary of icons and a specific application. there are also "natural visual languages" that use logograms such as chinese, mayan, and egyptian @xcite . we will provide a short semantic analysis of emojis, emojitaliano, and augmentative and alternative communication (aac). the most popular icons today are emojis, which roused discussions about to what extent they are a language and whether it is universal. emojis are widely used to express the user's communicative intent functioning as tone marking or as a word in a verbal cluster. however, emojis lack grammatical function words and morphosyntax. in spite of that, some consider it an emergent graphical language @xcite . emojitaliano is an autonomous communicative emoji code born for the italian language. it was created for the translation of pinocchio (2017), launched on twitter by f. chiusaroli, j. monti, f. sangati within the scritture brevi community ( https://www.scritturebrevi.it/ ). emojitalianobot on telegram then supported the translation project. it contains the grammar and dictionary of the iconic system. emojitaliano consists of a repertoire of lexical correspondences and a grammatical structure predefined that reflects the content found in pinocchio. it respects linguistic principles such as linearity, economy, and arbitrariness. emojitaliano does not have a high degree of iconicity (similarity between form and meaning of a sign) because many solutions are the result of an idiomatic or culturally marked decision not related to human experiences @xcite . for example, emojitaliano represents the abstract concept of guilt with man + woman + apple, representing the referent using a biblical and culture-specific metaphor. the field of (aac) has created various technologies to facilitate communication for people who cannot communicate through language in the standard way. different approaches exist to develop aac iconic languages. from the semantic point of view, these systems developed three ways to represent language: i) single meaning pictures ii) alphabet-based methods are often subdivided to include spelling, word prediction, and orthographic word selection iii) semantic compaction uses multimeaning icons in sequences to represent language. minspeak, for example, uses semantic compaction @xcite . non-linear aac has been proposed based on semantic roles and verb valency @xcite . the attention of researchers now focuses on the automatic detection of icons' meaning, using machine learning and word embedding techniques @xcite . nevertheless, in the creation of a visual language, it is also essential to empirically assess the degree of polysemy of a given icon, how its meaning is conveyed according to different levels of knowledge of users, evaluating the ambiguity within the system @xcite . ## case studies a semantic criterion, namely the inherent conceptual content of the event, is used to group meanings and relative icons ## iconometer test iconometer is a software developed by the univ. of geneva @xcite to implement the theoretical approach proposed by @xcite to assess the degree of polysemy of a visual representation (icon, diagram, figurative image, photograph) and measure its adequacy to its prescribed meaning. iconometer was previously used within the ikon language to evaluate icons from the family domain and gender symbols used to signify gender. @xcite . family icons representing family relationships with different gender signifiers were tested: only gender symbols, only haircuts, or both gender signifiers. the results demonstrated no significant difference in accuracy or certainty when comparing gender symbols, haircuts, or the combined gender signifiers. the research contributed in two ways: i) ikon language makes use of gender signifiers; ii) few family icons were subject to reconsideration due to low certainty in the test (e.g., grandfather misunderstood for stepfather). our current objective is to compare the level of certainty that participants had on interpretations of icons in different domains: family, modality (e.g., icons for must and can), operators, contrastive icons, motion verbs. in some cases, we proposed two or more versions of the same icon; in others, only a single version was displayed to assess how it was perceived. the new iconometer test presents the participant with 30 visual images and no text. below the images are 8 possible meanings and the option to write a personal answer. the participant must distribute a total of 100 points among the different meanings according to certainty. the participant must give the most points to the meaning that seems most certain and has the option to give 100 points to a single meaning. currently, we have not yet a sufficient population to draw conclusions and we leave the discussion of results and consideration about the mechanism of meaning assignment for future work. ## ikon in context: the medical domain to get a useful set of iconic vocabulary for a typical emergency dental treatment situation, a set of questions and sentences that are important in the anamnesis and treatment of patients were developed, e.g., questions about previous illnesses of various organs, diabetes, or medication allergies. the aim was to design unique pictograms in the figure 8: the simpler iconic sentence does cold cause your toothache? does hot cause your toothache? the development of intuitive content is shown using the example of the question sentence: do you have a toothache with hot or cold food or drinks? (figure 7 ). the concept of contrastive representation in a pictogram, like above for hot and cold, is certainly helpful in isolated observation. however, grasping this principle and paying attention to the form requires an additional cognitive effort, thus creating possibilities for misinterpretation. in this context, it seems redundant. also, the icons for eating and drinking seem redundant and confusing. the and/or sign also caused difficulties for some respondents. in light of that, it is easier to construct an iconic sentence that does not follow the syntax of questions formulated in the modern indo-european languages of europe. accordingly, figure 8 shows the same question being split into two sentences with a simplified syntax. ## conclusions in this work, we presented the ikon language with a core set of about 500 core concepts. new meanings are semantically analyzed and then translated into a visual representation. in this process, ikon follows defined criteria that assure coherence and flexibility within the system while continuously growing its vocabulary. we examined concepts grouped according to the conceptual event encoded, such as modality (e.g., must), perception, motion, communication verbs. these are complex events or subject to cultural variation. their examination gave an insight into the semantic analysis required to design a visual correspondent: from the understanding of the semantic frame of a word (description of a type of event, relation, entity, participants) to semiotic and non-verbal language analysis. we then introduced modern linguistic resources that can be helpful for their depiction. however, only testing the different icons will tell us which one performs best among speakers of different backgrounds. positively, cultural variation plays a significant role in our work, and ikon aims at giving equal representation. we presented the iconometer test, previously used to test the family icons and gender signifiers. the test is essential to assess the adequacy of the prescribed icons. the new test is ongoing as we do not have a sufficient diversified population yet. therefore, we plan to analyze and discuss results in future work and review icons that do not perform well on the test. finally, we brought an example of the ikon language application in the dentist-patient discourse showing that medical content can be transferred successfully into an iconic language. building iconic sentences is possible and beneficial, in that helps people with language impairment or in a situation of linguistic barrier to communicate in such a complex domain as healthcare. however, the study demonstrated that semantic considerations adopted for a single icon may not work in a more complex syntax because of the cognitive effort required.