Converting Multilayer Glosses into Semantic and Pragmatic forms with GENLIS (2021)

Abstract: This paper presents work carried out to transform glosses of a fable in Italian Sign Language (LIS) into a text which is then read by a TTS synthesizer from an SSML modified version of the same text. Whereas many systems exist that generate sign language from a text, we decided to do the reverse operation and generate text from LIS. For that purpose we used a version of the fable The Tortoise and the Hare, signed and made available on Youtube by ALBA cooperativa sociale, which was annotated manually by second author for her master’s thesis. In order to achieve our goal, we converted the multilayer glosses into linear Prolog terms to be fed to the generator. In the paper we focus on the main problems encountered in the transformation of the glosses into a semantically and pragmatically consistent representation. The main problems have been caused by the complexities of a text like a fable which requires coreference mechanisms and speech acts to be implemented in the representation which are often unexpressed and constitute implicit information.

## introduction this paper presents work carried out for the automatic generation of written text in italian language starting from glosses of fables in italian sign language (lis). the paper focuses on the semantic and pragmatic representation that has been created by the system genlis that feeds the generator. whereas many systems exist that generate sign language from a text @xcite @xcite , we decided to do the reverse operation and generate text from lis. a number of systems exist for american sign language that have attempted the same operation but only on a simple sentential basis and starting from visual recognition @xcite @xcite ). the possibility to produce glosses automatically from video capture and image recognition (but see @xcite ) is not available for lis, so we chose not to tackle the visual recognition phase and to start directly on the output, i.e. glosses foot_0 . glosses for lis are partly domain dependent in the sense that annotating sentences is a different task from annotating a dialogue, and this in turn is different from annotating a story or a fable. among the many types of text that we could work on we chose the most difficult one: a fable, which is a mixture of narrative text and dialogues. for that purpose we used a version of the fable the tortoise and the hare, signed and kindly made available by alba cooperativa sociale. the signed story was annotated manually into glosses by second author -who is a lis translator -for her master's thesis (see also @xcite ). the fable has two main characters -and other secondary characters -with totally different personalities which may interact in dialogues, or may be simply narrated thus producing an overall complex textual structure. ## semantic and pragmatic as will be explained below, main problems have been caused by the complexities of a text like a fable -which is partly a dialogue and partly narration -and requires coreference mechanisms and speech acts to be implemented in order to convert glosses into a semantically and pragmatically consistent representations. the final text is organized into discourse units (hence dus) or turns where each one may contain one or more sentences, and is associated with a unique turn identifier and a unique speaker. eventually we came up with 30 dus, 54 sentences and 91 propositions. the full project is presented on a website https://genlis.vercel.app/ . the website contains full representations for the all the dus of the fable, showing the conversion process step by step. every du starts by the video clip of the actor performing the lis narration of the current du; this is followed by the multilayered annotation 2 which is then turned into the 9 slot prolog consistent vector-like term. the transcribed vector is then enriched by semantic information and then by pragmatic information. the final step is the italian sentences produced by the generator 3 , which are then spoken aloud by the speech synthesizer on any mac or pc. the final part of genlis addresses the speech synthesizer with a set of prosodic markers to induce correct pauses, voice volume, intonational movements 4 . for lack of space we cannot comment on this part of the system: we can only say that we are using ssml on available speech synthesizers that accept it, to produce an expressive and semantically correct recital of the story. state-of-the-art generation systems work mostly on the basis of a machine learning approach @xcite , @xcite , which crucially requires an adequate amount of training data to feed the model. in our case training data are not available 5 also because glosses for lis are partly domain dependent as said above. in our case, we decided to generate text from a lis version of the fable the tortoise and the hare which has two main characters with totally different personalities. as will be clear from the sections below, we decided to follow a traditional approach which apart from the starting phase -content determination made available by the glosses -continues with text structuring, sentence aggregation, 2 manual annotation of simpler texts -either narrative or conversational -is not a highly time-consuming activity and can be carried out by an expert in a relatively short time. 3 we are not aware of the existence of many generators for italian @xcite apart from the ones built by some of our collaborators (see @xcite ) who were also partly the authors of a smaller version of the current one. the generator is now a general tool to generate most italian sentence structures, and has been used in a number of other applications, like question-answering from a discourse model (see @xcite ). genlis is written in the logic programming language prolog @xcite @xcite , which makes available dcg (definite clause grammar) rules together with difference lists to support text generation. the sequence of processes carried out by the system are represented in figure 2 below. semantic forms are composed by main predicate, propositional attributes (such as e.g. mood, negation, verbal tense), arguments and adjuncts. furthermore, each argument has its own internal structure. semantic forms constitute the string that is eventually fed as input to the generator and then processed, in order to generate italian sentences. we will describe below both the process of conversion of glosses into semantic forms and the structure of semantic forms. we will skip the first step in the whole process, which is producing the glosses and is done manually. as described in detail in another paper @xcite , manual glosses may contain arbitrarily many layers but they have the goal to interpret the signs in a shared manner. they are basically multi-layer text annotations written in tables, which can be done using one of the many software available for the task -at first we used elan 6 , but then we produced our own schemes to suit the requirements of the generator. in fact, in order for the multi-layer glosses to be analysed by the generator, it has been necessary to transform them into a 9 slot prolog term. thus, each annotation tier has been inserted in slots in a term, as follows: 6 it can be downloaded here https://archive.mpi.nl/tla/elan visited on april 2021 gls(duind,aff,adv,syn,agr,nms,ms,ars,qrs) ## special rules required by implicit elements there is a number of rules that need to be organized mainly inside the conversion portion of the system. these rules regard a number of specific features that are missing in the lis glosses and in the sign language as a whole. they concern definiteness assignment, that is the need to add an article in italian sentences in front of a nominal expression, which could also be zero article. then there is the need to vary the direct speech introductory verb which is otherwise always reported as dire/say. eventually there is the need to map tense, mood and person/gender/number onto all verb complexes. ## evaluation and discussion evaluation can be done manually or automatically @xcite . in order to do it automatically one would need a corpus of fables to be used for training which we currently don't have available. one should also take into account the need to measure how well glosses for lis have been created and have been used by the system to produce a naturally sounding italian text which resembles a fable. also this evaluation is difficult to make for the same reason. we turned to human evaluation for lack of a better opportunity now left for the future. in order to evaluate the output of the generator we wrote manually a version of the story which was more adherent to what is expected from children fables and is attested in online versions of this fable. at the same time, the made up version had to respect as faithfully as possible the signed version produced by the signer in the video. the result is a story which is pleasant to listen to by children and adults, as we tested in a primary school classroom for an experiment. now comes the evaluation of the generated story that we are able to produce by a comparison with the manually created story -that we make available in full in the supplementary materials. the comparison was done at the beginning in order to produce the peripheral rules presented in the section above. what has been left unchanged is discussed here below. we decided to grade each discourse unit or subunit by a four levels graded scale: 1 = no difference, 2 = slight differences, 3 = noticeable differences, 4 = very different. ## conclusion in this paper we presented the conversion process produced by genlis, a system that generates italian text from glosses of the italian sign language (lis). the signed text we chose is a fable, i.e. a semantically and pragmatically difficult text to generate. we described all the steps that are required to convert a vector-like representation of the multilayered annotation scheme used for transcribing signs into glosses. to complete our experiment, we did an evaluation by comparing the output of the generator to a manually written version of the story to suit stylistic requirements for fables and came to the conclusion that the result is acceptable but for a few particularly difficult utterances. eventually we only had 8 semantically relevant mismatches over 63 as a whole. however we had to overcome a number of problematic issues at a morphological, syntactic and semantic level which were successfully solved thanks to peripheral rules executed at the end of the generation process. future work includes improving the algorithm to generate a story which is more natural and pleasant to listen to. it shall also address separately either dialogues or narrative texts in order to produce a consistent and more generalized conversion process from glosses to spoken utterances.