Update README.md

README.md

@@ -1,199 +1,284 @@
 ---
 library_name: transformers
-tags: []
+widget:
+- messages:
+  - role: user
+    content: How does the brain work?
+inference:
+  parameters:
+    max_new_tokens: 200
+extra_gated_heading: Access Gemma on Hugging Face
+extra_gated_prompt: >-
+  To access Gemma on Hugging Face, you’re required to review and agree to
+  Google’s usage license. To do this, please ensure you’re logged-in to Hugging
+  Face and click below. Requests are processed immediately.
+extra_gated_button_content: Acknowledge license
+datasets:
+- yatharth97/10k_reports_gemma
 ---
 
-# Model Card for Model ID
+# yatharth-gemma-7b-it-10k Model Card
 
-<!-- Provide a quick summary of what the model is/does. -->
+**Reference Model Page**: [Gemma](https://ai.google.dev/gemma/docs)
 
+This model card pertains to the version of the Gemma model that has been fine-tuned on a dataset of 10K reports, specifically to enhance performance on tasks related to answering questions about these reports
 
 
-## Model Details
+**Authors**: Yatharth Mahesh Sant
 
-### Model Description
+## Model Information
 
-<!-- Provide a longer summary of what this model is. -->
+Summary description and brief definition of inputs and outputs.
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+### Description
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+The model presented here is an advanced adaptation of the Gemma 2B-IT, a member of the Gemma family of lightweight yet state-of-the-art models developed by Google. Leveraging the breakthrough research and technology that brought forth the Gemini models, our fine-tuned iteration specializes in parsing and understanding financial texts, particularly those found in 10-K reports.
 
-### Model Sources [optional]
+Dubbed the "yatharth-gemma-2B-it-10k" this model retains the text-to-text, decoder-only architecture of its progenitors, functioning optimally in English. What sets it apart is its refined focus on question-answering tasks specific to the intricate domain of 10-K reports — an invaluable resource for financial analysts, investors, and regulatory professionals seeking AI-driven insights.
 
-<!-- Provide the basic links for the model. -->
+Preserving the open-weights philosophy of the original Gemma models, this variant has been instruction-tuned with a curated dataset of 10-K reports. It not only demonstrates an enhanced proficiency in generating accurate, context-aware responses to user queries but also maintains the flexibility and efficiency that allow deployment in various settings, from personal computers to cloud-based environments.
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+The "yatharth-gemma-2B-it-10k" upholds the Gemma tradition of facilitating text generation tasks such as summarization and complex reasoning. Its unique optimization for financial reports exemplifies our commitment to pushing the boundaries of specialized AI, providing an unparalleled tool for dissecting and interpreting one of the business world's most information-dense documents.
 
-## Uses
+By marrying the accessibility of the Gemma models with the niche expertise required to navigate 10-K reports, we extend the frontiers of what's possible with AI, democratizing cutting-edge technology to empower financial analysis and decision-making.
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+### Usage
 
-### Direct Use
+Below we share some code snippets on how to get quickly started with running the model. First make sure to `pip install -U transformers`, then copy the snippet from the section that is relevant for your usecase.
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+#### Fine-tuning the model
 
-[More Information Needed]
+You can find fine-tuning scripts and notebook under the [`examples/` directory](https://huggingface.co/google/gemma-2b/tree/main/examples) of [`google/gemma-2b`](https://huggingface.co/google/gemma-2b) repository. To adapt it to this model, simply change the model-id to `yatharth97/yatharth-gemma-2b-it-10k`.
+In that repository, we provide:
 
-### Downstream Use [optional]
+* A script to perform Supervised Fine-Tuning (SFT) on UltraChat dataset using QLoRA
+* A script to perform SFT using FSDP on TPU devices
+* A notebook that you can run on a free-tier Google Colab instance to perform SFT on English quotes dataset
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 
-[More Information Needed]
+#### Running the model on a CPU
 
-### Out-of-Scope Use
+As explained below, we recommend `torch.bfloat16` as the default dtype. You can use [a different precision](#precisions) if necessary.
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
 
-[More Information Needed]
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained(
+    "yatharth97/yatharth-gemma-2b-it-10k",
+    torch_dtype=torch.bfloat16
+)
 
-## Bias, Risks, and Limitations
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt")
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-[More Information Needed]
 
-### Recommendations
+#### Running the model on a single / multi GPU
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+```python
+# pip install accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
 
-## How to Get Started with the Model
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained(
+    "yatharth97/yatharth-gemma-2b-it-10k",
+    device_map="auto",
+    torch_dtype=torch.bfloat16
+)
 
-Use the code below to get started with the model.
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-[More Information Needed]
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-## Training Details
+<a name="precisions"></a>
+#### Running the model on a GPU using different precisions
 
-### Training Data
+The native weights of this model were exported in `bfloat16` precision. You can use `float16`, which may be faster on certain hardware, indicating the `torch_dtype` when loading the model. For convenience, the `float16` revision of the repo contains a copy of the weights already converted to that precision.
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+You can also use `float32` if you skip the dtype, but no precision increase will occur (model weights will just be upcasted to `float32`). See examples below.
 
-[More Information Needed]
+* _Using `torch.float16`_
 
-### Training Procedure
+```python
+# pip install accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import torch
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained(
+    "yatharth97/yatharth-gemma-2b-it-10k",
+    device_map="auto",
+    torch_dtype=torch.float16,
+    revision="float16",
+)
 
-#### Preprocessing [optional]
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-[More Information Needed]
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
+* _Using `torch.bfloat16`_
 
-#### Training Hyperparameters
+```python
+# pip install accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k", device_map="auto", torch_dtype=torch.bfloat16)
 
-#### Speeds, Sizes, Times [optional]
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-[More Information Needed]
+* _Upcasting to `torch.float32`_
 
-## Evaluation
+```python
+# pip install accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained(
+    "yatharth97/yatharth-gemma-2b-it-10k",
+    device_map="auto"
+)
 
-### Testing Data, Factors & Metrics
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-#### Testing Data
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-<!-- This should link to a Dataset Card if possible. -->
+#### Quantized Versions through `bitsandbytes`
 
-[More Information Needed]
+* _Using 8-bit precision (int8)_
 
-#### Factors
+```python
+# pip install bitsandbytes accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+quantization_config = BitsAndBytesConfig(load_in_8bit=True)
 
-[More Information Needed]
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k", quantization_config=quantization_config)
 
-#### Metrics
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-[More Information Needed]
+* _Using 4-bit precision_
 
-### Results
+```python
+# pip install bitsandbytes accelerate
+from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
 
-[More Information Needed]
+quantization_config = BitsAndBytesConfig(load_in_4bit=True)
 
-#### Summary
+tokenizer = AutoTokenizer.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k")
+model = AutoModelForCausalLM.from_pretrained("yatharth97/yatharth-gemma-2b-it-10k", quantization_config=quantization_config)
 
+input_text = 'Can you tell me what the Total Debt was in 2023?'
+input_ids = tokenizer(input_text, return_tensors="pt").to("cuda")
 
+outputs = model.generate(**input_ids)
+print(tokenizer.decode(outputs[0]))
+```
 
-## Model Examination [optional]
 
-<!-- Relevant interpretability work for the model goes here -->
+#### Other optimizations
 
-[More Information Needed]
+* _Flash Attention 2_
 
-## Environmental Impact
+First make sure to install `flash-attn` in your environment `pip install flash-attn`
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+```diff
+model = AutoModelForCausalLM.from_pretrained(
+    model_id, 
+    torch_dtype=torch.float16, 
++   attn_implementation="flash_attention_2"
+).to(0)
+```
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+### Chat Template
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+The instruction-tuned models use a chat template that must be adhered to for conversational use.
+The easiest way to apply it is using the tokenizer's built-in chat template, as shown in the following snippet.
 
-## Technical Specifications [optional]
+Let's load the model and apply the chat template to a conversation. In this example, we'll start with a single user interaction:
 
-### Model Architecture and Objective
+```py
+from transformers import AutoTokenizer, AutoModelForCausalLM
+import transformers
+import torch
 
-[More Information Needed]
+model_id = "yatharth97/yatharth-gemma-2b-it-10k"
+dtype = torch.bfloat16
 
-### Compute Infrastructure
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    device_map="cuda",
+    torch_dtype=dtype,
+)
 
-[More Information Needed]
+chat = [
+    { "role": "user", "content": "Can you tell me what the Total Debt was in 2023?" },
+]
+prompt = tokenizer.apply_chat_template(chat, tokenize=False, add_generation_prompt=True)
+```
 
-#### Hardware
+At this point, the prompt contains the following text:
 
-[More Information Needed]
+```
+<bos><start_of_turn>user
+Can you tell me what the Total Debt was in 2023?<end_of_turn>
+<start_of_turn>model
+```
 
-#### Software
+As you can see, each turn is preceded by a `<start_of_turn>` delimiter and then the role of the entity
+(either `user`, for content supplied by the user, or `model` for LLM responses). Turns finish with
+the `<end_of_turn>` token.
 
-[More Information Needed]
+You can follow this format to build the prompt manually, if you need to do it without the tokenizer's
+chat template.
 
-## Citation [optional]
+After the prompt is ready, generation can be performed like this:
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+```py
+inputs = tokenizer.encode(prompt, add_special_tokens=False, return_tensors="pt")
+outputs = model.generate(input_ids=inputs.to(model.device), max_new_tokens=150)
+print(tokenizer.decode(outputs[0]))
+```
 
-**BibTeX:**
+### Inputs and outputs
 
-[More Information Needed]
+*   **Input:** Text string, such as a question, a prompt, or a 10K document to be
+    summarized.
+*   **Output:** Generated English-language text in response to the input, such
+    as an answer to a question, or a summary of uploaded 10K document. For summarization currently a separate model is being used.
 
-**APA:**
+## Model Data
 
-[More Information Needed]
+Data used for model training and how the data was processed.
 
-## Glossary [optional]
+### Training Dataset
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
-
-[More Information Needed]
-
-## More Information [optional]
-
-[More Information Needed]
-
-## Model Card Authors [optional]
-
-[More Information Needed]
-
-## Model Card Contact
-
-[More Information Needed]
+This model is fine tuned on the dataset: "yatharth97/10k_reports_gemma" which has a conversational based format allowing the user to ask questions about the uploaded 10K report