  Explainable AI(XAI)is a domain focused on providing interpretability and
explainability of a decision-making process. In the domain of law, in addition
to system and data transparency, it also requires the (legal-) decision-model
transparency and the ability to understand the models inner working when
arriving at the decision. This paper provides the first approaches to using a
popular image processing technique, Grad-CAM, to showcase the explainability
concept for legal texts. With the help of adapted Grad-CAM metrics, we show the
interplay between the choice of embeddings, its consideration of contextual
information, and their effect on downstream processing.
