ERC Advanced Grant Project CALCULUS : Extending the Boundary of Machine Translation (2024)

Abstract: The CALCULUS project, drawing on human capabilities of imagination and commonsense for natural language understanding (NLU), aims to advance machine-based NLU by integrating traditional AI concepts with contemporary machine learning techniques. It focuses on developing anticipatory event representations from both textual and visual data, connecting language structure to visual spatial organization and incorporating broad knowledge domains. Through testing these models in NLU tasks and evaluating their ability to predict untrained spatial and temporal details using real-world metrics, CALCULUS employs machine learning methods, including Bayesian techniques and neural networks, especially in data-sparse scenarios. The project’s culmination involves creating demonstrators that transform written stories into dynamic videos, showcasing the interdisciplinary expertise of the project leader in natural language processing, language and visual data analysis, information retrieval, and machine learning, all vital for the project’s achievements. In the CALCULUS project, our exploration of machine translation extends beyond the conventional text-to-text framework. We are broadening the horizons of machine translation by delving into the essence of transforming the formats of data distribution while keeping the meaning. This innovative approach involves converting information from one modality into another, transcending traditional linguistic boundaries. Our project includes novel work on translating text into images and videos, brain signals into images and videos.

## objective the calculus project, drawing on human capabilities of imagination and commonsense for natural language understanding (nlu), aims to advance machine-based nlu by integrating traditional ai concepts with contemporary machine learning techniques. it focuses on developing anticipatory event representations from both textual and visual data, connecting language structure to visual spatial organization and incorporating broad knowledge domains. anticipatory event representations refer to representations that are able to predict what content is highly probable to be communicated next in a discourse. cal-culus tests these models in nlu tasks and uses ©2024 the authors. this article is licensed under a creative commons 4.0 licence, no derivative works, attribution, cc-by-nd. ## relation with machine translation in the calculus project, we are broadening the horizons of machine translation by delving into the essence of transforming the formats of data distribution while keeping the meaning. this innovative approach involves converting information from one modality into another, transcending traditional linguistic boundaries. our project includes novel work on translating text into images, videos and layouts and brain signals to stimuli as illustrated below. the proposed models for multimodal translation can be a source of inspiration for future language translation (e.g., noise reduction in diffusion models, loss functions that preserve the structure of the source input).