  In this work, we present our approach and findings for SemEval-2021 Task 5 -
Toxic Spans Detection. The task's main aim was to identify spans to which a
given text's toxicity could be attributed. The task is challenging mainly due
to two constraints: the small training dataset and imbalanced class
distribution. Our paper investigates two techniques, semi-supervised learning
and learning with Self-Adjusting Dice Loss, for tackling these challenges. Our
submitted system (ranked ninth on the leader board) consisted of an ensemble of
various pre-trained Transformer Language Models trained using either of the
above-proposed techniques.
