YNU - HPCC at S em E val-2023 Task 6: LEGAL - BERT Based Hierarchical B i LSTM with CRF for Rhetorical Roles Prediction (2023)

Abstract: To understand a legal document for real-world applications, SemEval-2023 Task 6 proposes a shared Subtask A, rhetorical roles (RRs) prediction, which requires a system to automatically assign a RR label for each semantical segment in a legal text. In this paper, we propose a LEGAL-BERT based hierarchical BiLSTM model with conditional random field (CRF) for RR prediction, which primarily consists of two parts: word-level and sentence-level encoders. The word-level encoder first adopts a legal-domain pre-trained language model, LEGAL-BERT, initially word-embedding words in each sentence in a document and a word-level BiLSTM further encoding such sentence representation. The sentence-level encoder then uses an attentive pooling method for sentence embedding and a sentence-level BiLSTM for document modeling. Finally, a CRF is utilized to predict RRs for each sentence. The officially released results show that our method outperformed the baseline systems. Our team won 7th rank out of 27 participants in Subtask A.

## introduction in populous countries, the number of pending legal cases has been rising exponentially. for instance, in india, according to the national judicial data grid 1 , as of 04 july 2022, the supreme court of india had approximately 6 million cases pending, and with the addition of other local courts, there will be far more than 6 million cases pending. it is urgent to require an automatic legal system, which helps practitioners extract accurate and valid information from legal documents, for efficient legal processing. to this end, semeval-2023 proposes a shared task court judgment prediction with explanation (cjpe), respectively. focusing on the rr prediction task, it requires a system to structure unstructured legal documents into semantical segments that are aligned with rr labels such as preamble, fact, ratio, etc. such segmentations as fundamental building blocks are crucial for many legal artificial intelligence (ai) applications, e.g., judgment summarizing, judgment outcome prediction, and precedent search. due to the shared task having preprocessed legal documents by splitting them into several semantical sentences as coherent units, rr prediction in subtask a could be regarded as a sentence-wise sequence labeling task that predicts a rr label for each sentence in a legal document. with the increasing growth of deep learning (dl) and advanced pre-trained language models (plms) @xcite , various natural language processing (nlp) tasks have been effectively addressed, such as sentiment analysis @xcite , named entity recognition @xcite , and text classification @xcite . for instance, scibert-hsln @xcite introduced a sequential sentence classification method for rr predictions. to further consider the problem of low shift probability in sentence labels, several works adopted label shift prediction as an auxiliary task for accurate rr prediction @xcite . although general plms, e.g., bert @xcite , robert @xcite ( @xcite , have shown a robust ability to effectively model most natural language texts and be deployed in real-world applications, they are difficult to apply directly for rr prediction tasks. the main reason is that legal texts differ from general texts used for existing model training. most of the legal texts are much longer than the input (512 tokens) of most pre-trained models. moreover, legal texts contain a lot of specialized terminology, and pre-trained models that are not trained on the legal corpus may not work particularly well. to address the above problem, we proposed legal-bert-based @xcite hierarchical bilstm (bidirectional long short-term memory) with conditional random field (crf) @xcite , denoted as hlbert-crf, for rr prediction. legal-bert is a legalspecific plm that further trains the bert model on legal domain data. hlbert-crf primarily consists of two parts: word-level and sentence-level encoder. word-level encoder based on legal-bert is first proposed to encode sentence representations via bilstm. sentence representations are then integrated via attention pooling and further encoded via bilstm at the sentence level, resulting in a document representation that contains a set of sentence embeddings. finally, a crf-based classifier predicts probabilities over various rr labels. moreover, we found that using a hard-voting strategy gained more performance. a detailed experiment is conducted on subtask a of semeval-2023 shared task 6, revealing our team won the 7th rank out of 27 participant teams. the remainder of the paper is structured as follows. section 2 provides a detailed description of the proposed method. extensive experiments are conducted and analyzed in section 3, and conclusions are drawn in section 4. ## system description this section mainly describes the proposed hlbert-crf model for rr predictions. as shown in fig. 1 , hlbert-crf primarily consists of two parts, including word-level and sentencelevel encoders. the word-level encoder is intented to generate sentence representations; the sentencelevel encoder aims to generate document representation containing sentence embeddings that are utilized for predicting sentence-wise rr labels by the crf module. before introducing the proposed model in detail, we first formulate the rr prediction tasks. ## experimental results in this section, extensive experiments were conducted and analyzed. ## conclusions in this paper, we proposed an hlbert-crf model for rr prediction. hlbert-crf utilizes hierarchical bilstm with crf structure the backbone and introduces legal-specific plm, legal-bert, for knowledge transferring. as a result, hlbert-crf with a hard-voting strategy won the 7th in semeval-2023 shared task 6 (subtask a). in the future, we will explore more complex legal text-based nlp tasks by devising more efficient structures and explainable modules.