D eep SPIN : Deep Structured Prediction for Natural Language Processing (2022)

Abstract: DeepSPIN is a research project funded by the European Research Council (ERC) whose goal is to develop new neural structured prediction methods, models, and algorithms for improving the quality, interpretability, and data-efficiency of natural language processing (NLP) systems, with special emphasis on machine translation and quality estimation. We describe in this paper the latest findings from this project.

## description the deepspin project 1 is an @xcite hosted at instituto de telecomunicações. part of the work has been done in collaboration with unbabel, an sme in the crowdsourcing translation industry. the main goal of deepspin is to bring together deep learning and structured prediction techniques to solve structured problems in nlp. the three main objectives are: developing better decoding strategies; making neural networks more interpretable through the induction of sparse structure; and incorporating of weak supervision to reduce the need for labeled data. we focus here on the applications to mt, including some of the recent results obtained in the project. better decoding strategies. our initial work on sparse sequence-to-sequence models @xcite proposed a new class of decoders (called "entmax decoders", shown in fig. 1 ) which operate over a sparse probability distribution over © 2022 the authors. this article is licensed under a creative commons 3.0 licence, no derivative works, attribution, cc-by-nd. 1 project website: https://deep-spin.github.io . at on , this so and here the tree of life . view look glimpse kind looking way vision gaze is another 92.9% 5.9% 1.3% <0.1% 49.8% 27.1% 19.9% 2.0% 0.9% 0.2% <0.1% <0.1% 95.7% 5.9% 1.3% words, which prunes hypotheses automatically. in @xcite , we have shown that entmax decoders are better calibrated and less prone to the length bias problem and developed a new label smoothing technique. we also presented entmax sampling for text generation, with improved generation quality @xcite . another line of work concerns modeling of context in machine translation. we introduced conditional cross-mutual information (cxmi), a technique to measure the effective use of contextual information by context-aware systems, and context-aware word dropout, which increases its use, leading to improvements @xcite . we also compared the models' use of context to that of humans for translating ambiguous words, using the latter as extra supervision @xcite .