Transfer and multi-task learning have traditionally focused on either a single source-target pair or very few, similar tasks. Ideally, the linguistic levels of morphology, syntax and semantics would benefit each other by being trained in a single model. We introduce a joint many-task model together with a strategy for successively growing its depth to solve increasingly complex tasks. Higher layers include shortcut connections to lower-level task predictions to reflect linguistic hierarchies. We use a simple regularization term to allow for optimizing all model weights to improve one task's loss without exhibiting catastrophic interference of the other tasks. Our single end-to-end model obtains state-of-the-art or competitive results on five different tasks from tagging, parsing, relatedness, and entailment tasks.
REF jointly learns POS tagging, chunking, dependency parsing, semantic relatedness, and textual entailment by considering linguistic hierarchy and achieves state-of-the-results on five datasets.
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks
ArXiv
ArXiv
Computer Science