Upload 16 files

Legal-LED_IN_ABS/README.md

@@ -0,0 +1,202 @@
+---
+library_name: peft
+base_model: nsi319/legal-led-base-16384
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.10.0

Legal-LED_IN_ABS/adapter_config.json

@@ -0,0 +1,81 @@
+{
+  "alpha_pattern": {},
+  "auto_mapping": null,
+  "base_model_name_or_path": "nsi319/legal-led-base-16384",
+  "bias": "none",
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_dropout": 0.1,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "r": 4,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "led.encoder.layers.4.self_attn.longformer_self_attn.key",
+    "led.decoder.layers.1.encoder_attn.k_proj",
+    "led.decoder.layers.1.self_attn.v_proj",
+    "led.decoder.layers.2.self_attn.k_proj",
+    "led.encoder.layers.1.self_attn.longformer_self_attn.key",
+    "led.encoder.layers.2.self_attn.longformer_self_attn.value",
+    "led.encoder.layers.0.self_attn.longformer_self_attn.query",
+    "led.decoder.layers.5.encoder_attn.k_proj",
+    "led.encoder.layers.5.self_attn.longformer_self_attn.query",
+    "led.decoder.layers.5.self_attn.v_proj",
+    "led.encoder.layers.3.self_attn.longformer_self_attn.key",
+    "led.decoder.layers.3.encoder_attn.q_proj",
+    "led.decoder.layers.2.encoder_attn.v_proj",
+    "led.encoder.layers.0.self_attn.longformer_self_attn.value",
+    "led.decoder.layers.2.self_attn.q_proj",
+    "led.encoder.layers.1.self_attn.longformer_self_attn.query",
+    "led.decoder.layers.3.self_attn.q_proj",
+    "led.decoder.layers.5.encoder_attn.v_proj",
+    "led.decoder.layers.4.self_attn.k_proj",
+    "led.decoder.layers.1.self_attn.q_proj",
+    "led.encoder.layers.1.self_attn.longformer_self_attn.value",
+    "led.encoder.layers.3.self_attn.longformer_self_attn.query",
+    "led.encoder.layers.0.self_attn.longformer_self_attn.key",
+    "led.encoder.layers.4.self_attn.longformer_self_attn.query",
+    "led.decoder.layers.0.self_attn.k_proj",
+    "led.decoder.layers.2.self_attn.v_proj",
+    "led.decoder.layers.0.self_attn.v_proj",
+    "led.encoder.layers.3.self_attn.longformer_self_attn.value",
+    "led.encoder.layers.5.self_attn.longformer_self_attn.value",
+    "led.decoder.layers.4.encoder_attn.k_proj",
+    "led.decoder.layers.5.encoder_attn.q_proj",
+    "led.decoder.layers.0.encoder_attn.v_proj",
+    "led.encoder.layers.2.self_attn.longformer_self_attn.query",
+    "led.decoder.layers.0.self_attn.q_proj",
+    "led.decoder.layers.3.self_attn.k_proj",
+    "led.decoder.layers.0.encoder_attn.q_proj",
+    "led.decoder.layers.3.self_attn.v_proj",
+    "led.decoder.layers.4.encoder_attn.q_proj",
+    "led.decoder.layers.2.encoder_attn.k_proj",
+    "led.encoder.layers.4.self_attn.longformer_self_attn.value",
+    "led.decoder.layers.1.self_attn.k_proj",
+    "led.decoder.layers.1.encoder_attn.q_proj",
+    "led.decoder.layers.4.self_attn.q_proj",
+    "led.decoder.layers.4.encoder_attn.v_proj",
+    "led.decoder.layers.1.encoder_attn.v_proj",
+    "led.decoder.layers.4.self_attn.v_proj",
+    "led.decoder.layers.5.self_attn.q_proj",
+    "led.encoder.layers.2.self_attn.longformer_self_attn.key",
+    "led.decoder.layers.2.encoder_attn.q_proj",
+    "led.decoder.layers.5.self_attn.k_proj",
+    "led.decoder.layers.3.encoder_attn.v_proj",
+    "led.encoder.layers.5.self_attn.longformer_self_attn.key",
+    "led.decoder.layers.3.encoder_attn.k_proj",
+    "led.decoder.layers.0.encoder_attn.k_proj"
+  ],
+  "task_type": "SUMMARIZATION",
+  "use_dora": false,
+  "use_rslora": false
+}

Legal-LED_IN_ABS/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:768d20dd4b448772f8c62f5cfc1c20494d506678aa1d68f94746956aec85fe5b
+size 1342456

Legal-LED_IN_ABS/optimizer.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:0d341317be5693df2dd1d9636341166d40c8af8467051bd34cdc7d0fda452ee0
+size 2745082

Legal-LED_IN_ABS/rng_state.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2bee791231a99ff85057b54f035f640c56d457ea6ced79092eee7bd4510f79ba
+size 14244

Legal-LED_IN_ABS/scheduler.pt

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d20436381cde63e4ef4154c4facf935d7fceed96cba77d169e67689352a860ba
+size 1064

Legal-LED_IN_ABS/special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "<unk>",
+    "lstrip": false,
+    "normalized": true,
+    "rstrip": false,
+    "single_word": false
+  }
+}

Legal-LED_IN_ABS/tokenizer_config.json

@@ -0,0 +1,57 @@
+{
+  "add_prefix_space": false,
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "3": {
+      "content": "<unk>",
+      "lstrip": false,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "50264": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": true,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "eos_token": "</s>",
+  "errors": "replace",
+  "mask_token": "<mask>",
+  "model_max_length": 16384,
+  "pad_token": "<pad>",
+  "sep_token": "</s>",
+  "tokenizer_class": "LEDTokenizer",
+  "trim_offsets": true,
+  "unk_token": "<unk>"
+}

Legal-LED_IN_ABS/trainer_state.json

@@ -0,0 +1,279 @@
+{
+  "best_metric": 28.664,
+  "best_model_checkpoint": "results/Legal-LED/IndiaABS/exp1\\checkpoint-1757",
+  "epoch": 0.9997155049786629,
+  "eval_steps": 500,
+  "global_step": 1757,
+  "is_hyper_param_search": false,
+  "is_local_process_zero": true,
+  "is_world_process_zero": true,
+  "log_history": [
+    {
+      "epoch": 0.03,
+      "grad_norm": 5.772658348083496,
+      "learning_rate": 1.225e-05,
+      "loss": 4.2551,
+      "step": 50
+    },
+    {
+      "epoch": 0.06,
+      "grad_norm": 2.3984649181365967,
+      "learning_rate": 2.4750000000000002e-05,
+      "loss": 3.4607,
+      "step": 100
+    },
+    {
+      "epoch": 0.09,
+      "grad_norm": 1.634371042251587,
+      "learning_rate": 3.7250000000000004e-05,
+      "loss": 2.7537,
+      "step": 150
+    },
+    {
+      "epoch": 0.11,
+      "grad_norm": 1.7180700302124023,
+      "learning_rate": 4.975e-05,
+      "loss": 2.5124,
+      "step": 200
+    },
+    {
+      "epoch": 0.14,
+      "grad_norm": 1.7759343385696411,
+      "learning_rate": 4.842646114322415e-05,
+      "loss": 2.3961,
+      "step": 250
+    },
+    {
+      "epoch": 0.17,
+      "grad_norm": 1.5487947463989258,
+      "learning_rate": 4.6820809248554915e-05,
+      "loss": 2.373,
+      "step": 300
+    },
+    {
+      "epoch": 0.2,
+      "grad_norm": 1.6271101236343384,
+      "learning_rate": 4.521515735388568e-05,
+      "loss": 2.3289,
+      "step": 350
+    },
+    {
+      "epoch": 0.23,
+      "grad_norm": 1.7767431735992432,
+      "learning_rate": 4.360950545921644e-05,
+      "loss": 2.3136,
+      "step": 400
+    },
+    {
+      "epoch": 0.26,
+      "grad_norm": 1.5476394891738892,
+      "learning_rate": 4.2003853564547206e-05,
+      "loss": 2.228,
+      "step": 450
+    },
+    {
+      "epoch": 0.28,
+      "grad_norm": 2.2447686195373535,
+      "learning_rate": 4.039820166987797e-05,
+      "loss": 2.2193,
+      "step": 500
+    },
+    {
+      "epoch": 0.31,
+      "grad_norm": 2.0830910205841064,
+      "learning_rate": 3.879254977520873e-05,
+      "loss": 2.244,
+      "step": 550
+    },
+    {
+      "epoch": 0.34,
+      "grad_norm": 1.8925021886825562,
+      "learning_rate": 3.7186897880539504e-05,
+      "loss": 2.2185,
+      "step": 600
+    },
+    {
+      "epoch": 0.37,
+      "grad_norm": 1.9989118576049805,
+      "learning_rate": 3.558124598587027e-05,
+      "loss": 2.2333,
+      "step": 650
+    },
+    {
+      "epoch": 0.4,
+      "grad_norm": 2.6363413333892822,
+      "learning_rate": 3.397559409120103e-05,
+      "loss": 2.1538,
+      "step": 700
+    },
+    {
+      "epoch": 0.43,
+      "grad_norm": 2.3027186393737793,
+      "learning_rate": 3.2369942196531794e-05,
+      "loss": 2.2577,
+      "step": 750
+    },
+    {
+      "epoch": 0.46,
+      "grad_norm": 1.6798630952835083,
+      "learning_rate": 3.076429030186256e-05,
+      "loss": 2.203,
+      "step": 800
+    },
+    {
+      "epoch": 0.48,
+      "grad_norm": 2.2237415313720703,
+      "learning_rate": 2.915863840719332e-05,
+      "loss": 2.1747,
+      "step": 850
+    },
+    {
+      "epoch": 0.51,
+      "grad_norm": 2.019179105758667,
+      "learning_rate": 2.755298651252409e-05,
+      "loss": 2.1642,
+      "step": 900
+    },
+    {
+      "epoch": 0.54,
+      "grad_norm": 1.8116823434829712,
+      "learning_rate": 2.5947334617854852e-05,
+      "loss": 2.0992,
+      "step": 950
+    },
+    {
+      "epoch": 0.57,
+      "grad_norm": 2.7756757736206055,
+      "learning_rate": 2.4341682723185612e-05,
+      "loss": 2.073,
+      "step": 1000
+    },
+    {
+      "epoch": 0.6,
+      "grad_norm": 2.4382846355438232,
+      "learning_rate": 2.2736030828516376e-05,
+      "loss": 2.1295,
+      "step": 1050
+    },
+    {
+      "epoch": 0.63,
+      "grad_norm": 1.680977702140808,
+      "learning_rate": 2.1130378933847143e-05,
+      "loss": 2.1131,
+      "step": 1100
+    },
+    {
+      "epoch": 0.65,
+      "grad_norm": 2.0901172161102295,
+      "learning_rate": 1.9524727039177907e-05,
+      "loss": 2.145,
+      "step": 1150
+    },
+    {
+      "epoch": 0.68,
+      "grad_norm": 1.953476071357727,
+      "learning_rate": 1.791907514450867e-05,
+      "loss": 2.1414,
+      "step": 1200
+    },
+    {
+      "epoch": 0.71,
+      "grad_norm": 2.1726927757263184,
+      "learning_rate": 1.6313423249839434e-05,
+      "loss": 2.1293,
+      "step": 1250
+    },
+    {
+      "epoch": 0.74,
+      "grad_norm": 1.8229079246520996,
+      "learning_rate": 1.4707771355170199e-05,
+      "loss": 2.1369,
+      "step": 1300
+    },
+    {
+      "epoch": 0.77,
+      "grad_norm": 2.71445631980896,
+      "learning_rate": 1.3102119460500964e-05,
+      "loss": 2.0934,
+      "step": 1350
+    },
+    {
+      "epoch": 0.8,
+      "grad_norm": 2.6795523166656494,
+      "learning_rate": 1.1496467565831728e-05,
+      "loss": 2.0734,
+      "step": 1400
+    },
+    {
+      "epoch": 0.83,
+      "grad_norm": 2.1377625465393066,
+      "learning_rate": 9.890815671162493e-06,
+      "loss": 2.1422,
+      "step": 1450
+    },
+    {
+      "epoch": 0.85,
+      "grad_norm": 1.7045388221740723,
+      "learning_rate": 8.285163776493257e-06,
+      "loss": 2.0615,
+      "step": 1500
+    },
+    {
+      "epoch": 0.88,
+      "grad_norm": 2.001119375228882,
+      "learning_rate": 6.679511881824021e-06,
+      "loss": 2.0652,
+      "step": 1550
+    },
+    {
+      "epoch": 0.91,
+      "grad_norm": 2.5513315200805664,
+      "learning_rate": 5.0738599871547856e-06,
+      "loss": 2.1465,
+      "step": 1600
+    },
+    {
+      "epoch": 0.94,
+      "grad_norm": 1.8909486532211304,
+      "learning_rate": 3.468208092485549e-06,
+      "loss": 2.0798,
+      "step": 1650
+    },
+    {
+      "epoch": 0.97,
+      "grad_norm": 1.9534186124801636,
+      "learning_rate": 1.8625561978163134e-06,
+      "loss": 2.0924,
+      "step": 1700
+    },
+    {
+      "epoch": 1.0,
+      "grad_norm": 2.080364465713501,
+      "learning_rate": 2.569043031470777e-07,
+      "loss": 2.0598,
+      "step": 1750
+    },
+    {
+      "epoch": 1.0,
+      "eval_gen_len": 815.03,
+      "eval_loss": 2.0449941158294678,
+      "eval_rouge1": 55.6012,
+      "eval_rouge2": 28.664,
+      "eval_rougeL": 26.4007,
+      "eval_rougeLsum": 53.0454,
+      "eval_runtime": 14551.3385,
+      "eval_samples_per_second": 0.007,
+      "eval_steps_per_second": 0.007,
+      "step": 1757
+    }
+  ],
+  "logging_steps": 50,
+  "max_steps": 1757,
+  "num_input_tokens_seen": 0,
+  "num_train_epochs": 1,
+  "save_steps": 500,
+  "total_flos": 3.806868794454835e+16,
+  "train_batch_size": 1,
+  "trial_name": null,
+  "trial_params": null
+}

Legal-LED_IN_ABS/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b887e532dd0881e53b48f5222fe9c8cb2db6be354c3d43fca9c4d476016987c1
+size 5048

app.py

@@ -0,0 +1,51 @@
+import streamlit as st
+from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
+from peft import PeftModel
+
+# Loading LED IN Model
+base_model = "nsi319/legal-led-base-16384"
+led = AutoModelForSeq2SeqLM.from_pretrained(base_model)
+adapter_model_in = f"Legal-LED_IN_ABS"
+led_in = PeftModel.from_pretrained(led, adapter_model_in)
+led_in_tokenizer = AutoTokenizer.from_pretrained(base_model)
+
+# Generating Summary
+def summarize(model, tokenizer, text):
+    input_tokenized = tokenizer.encode(text, return_tensors='pt', max_length=8192, truncation=True)
+    summary_ids = model.generate(input_tokenized, num_beams=4, length_penalty=0.1, min_length=32, max_length=512)
+    summary = [tokenizer.decode(g, skip_special_tokens=True, clean_up_tokenization_spaces=False) for g in summary_ids][0]
+    return summary
+
+# Reading Txt File
+def read_txt_file(file):
+    text = file.read().decode('utf-8')
+    return text
+
+st.set_page_config(page_title="Legal AI Summarizer", page_icon="img.png")
+title = "Legal AI Summarizer"
+col1, col2 = st.columns([1,7])
+with col1:
+    st.image("img.png")
+with col2: st.title(title)
+st.write("Stuck with long legal documents? Our AI summarizer can help!  Just copy-paste the text or upload a .txt file, and it will give you a quick and easy summary in plain English, so you can understand the key points without all the legalese.")
+
+if "user_text" not in st.session_state:
+    st.session_state.user_text = ""
+
+upload_file = st.file_uploader("Upload a .txt file", type="txt")
+
+if upload_file is not None:
+    user_text = read_txt_file(upload_file)
+else:
+    user_text = st.text_area("Paste your legal document here:", value=st.session_state.user_text, height=300)
+
+if st.button("Generate Summary"):
+    with st.spinner("Generating summary..."):
+        try:
+            summary_text = summarize(led_in, led_in_tokenizer, user_text)
+            st.session_state.user_text = user_text
+            st.write("")
+            st.success(summary_text)
+            print(summary_text)
+        except Exception as e:
+            st.error(f"An error occurred: {e}")

requirements.txt

@@ -0,0 +1,4 @@
+streamlit
+transformers
+peft
+torch