M ulti3 G eneration: Multitask, Multilingual, Multimodal Language Generation (2022)

Abstract: This paper presents the Multitask, Multilingual, Multimodal Language Generation COST Action – Multi3Generation (CA18231), an interdisciplinary network of research groups working on different aspects of language generation. This “meta-paper” will serve as reference for citations of the Action in future publications. It presents the objectives, challenges and a the links for the achieved outcomes.

## introduction multi3generation 1 fosters the development of a network of researchers and technologists across interdisciplinary fields working on topics related to language generation (lg). we frame lg broadly as the set of tasks where the ultimate goal involves generating language. in contrast to the more classical definition of natural language generation (nlg), this also includes tasks not concerned with lg in an immediate sense, but that can inform or improve lg models. the action focuses on four core challenges: (a) data and information representation challenges, such as those involving inputs of different sources: images, videos, knowledge bases (kbs) and graphs; (b) machine learning (ml) challenges of modern approaches, such as mapping of inputs to different correct outputs, e.g. structured prediction and representation learning; (c) interaction in applications of lg, such as © 2022 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 https://www.cost.eu/actions/ca18231/ . the action is funded by the european commission and is running from june 2019 till september 2023. dialogue systems, conversational search interfaces and human-robot interaction due to the uncertainty derived from the changing environment and the non-deterministic fashion of interaction; (d) kb exploitation: structured knowledge is key to natural language processing (nlp) tasks, including nlg, supporting ml methods that require expansion, filtering, disambiguation or user adaptation of generated content. the action addresses these challenges by answering the following questions: the overall expected impact of the action is to bring about a significant change in progress towards effective solutions for computational challenges involving lg with respect to multitask, multilingual and multimodal aspects. in particular, multi3generation is focusing on the integration of these three aspects and how they can benefit lg solutions. the action's specific objectives for capacity building are: in order to accomplish the objectives of the action, its members are encouraged to produce novel outcomes and establish critical mass, as well as to engage in joint applications for european and national funding for research projects within the fields covered by the action. ## outcomes since its inception, the action fostered collaborations that has produced more than 24 publications foot_0 , ranging from surveys to approaches to specific lg problems. among the collaborations are the short term missions (stms), visits among researchers that take part in the action foot_1 . furthermore, a series of datasets foot_2 have been developed and made available for diverse number of lgrelated problems. another important outcome of the action is the organization of training schools in 2022, one on the topic of "representation mediated multimodality" foot_3 and another one on the topic of "automatically creating text from data" foot_4 . ## acknowledgements this publication is based upon work from cost action multi3generation -multitask, multilingual, multimodal language generation (ca18231), supported by cost (european cooperation in science and technology).