  Interpretability or explainability is an emerging research field in NLP. From
a user-centric point of view, the goal is to build models that provide proper
justification for their decisions, similar to those of humans, by requiring the
models to satisfy additional constraints. To this end, we introduce a new
application on legal text where, contrary to mainstream literature targeting
word-level rationales, we conceive rationales as selected paragraphs in
multi-paragraph structured court cases. We also release a new dataset
comprising European Court of Human Rights cases, including annotations for
paragraph-level rationales. We use this dataset to study the effect of already
proposed rationale constraints, i.e., sparsity, continuity, and
comprehensiveness, formulated as regularizers. Our findings indicate that some
of these constraints are not beneficial in paragraph-level rationale
extraction, while others need re-formulation to better handle the multi-label
nature of the task we consider. We also introduce a new constraint,
singularity, which further improves the quality of rationales, even compared
with noisy rationale supervision. Experimental results indicate that the newly
introduced task is very challenging and there is a large scope for further
research.
