Update README.md (#2)

- Update README.md (82d5c4e74d71a38469bd522669aa6adea80e4ace)


Co-authored-by: Satish Kumar Mishra <zicsx@users.noreply.huggingface.co>

README.md

@@ -105,24 +105,20 @@ model-index:
       name: Open LLM Leaderboard
 ---
 ---
-# Model Card for Project Indus
-
-<!-- Provide a quick summary of what the model is/does. [Optional] -->
-The model is a pretrained model in Hindi and dialects which is instruct tuned .
-
 
+# Model Card for Project Indus
 
+<!-- Provide a quick summary of what the model is/does. -->
+Project Indus LLM is a groundbreaking open-source language model tailored for Hindi and its dialects, designed to enhance natural language processing and generation across diverse Indian linguistic applications.
 
-#  Table of Contents
+# Table of Contents
 
-- [Model Card for Indus](#model-card-for--model_id-)
 - [Table of Contents](#table-of-contents)
-- [Table of Contents](#table-of-contents-1)
 - [Model Details](#model-details)
   - [Model Description](#model-description)
 - [Uses](#uses)
   - [Direct Use](#direct-use)
-  - [Downstream Use [Optional]](#downstream-use-optional)
+  - [Downstream Use](#downstream-use)
   - [Out-of-Scope Use](#out-of-scope-use)
 - [Bias, Risks, and Limitations](#bias-risks-and-limitations)
   - [Recommendations](#recommendations)
@@ -130,7 +126,6 @@ The model is a pretrained model in Hindi and dialects which is instruct tuned .
   - [Training Data](#training-data)
   - [Training Procedure](#training-procedure)
     - [Preprocessing](#preprocessing)
-    - [Speeds, Sizes, Times](#speeds-sizes-times)
 - [Evaluation](#evaluation)
   - [Testing Data, Factors & Metrics](#testing-data-factors--metrics)
     - [Testing Data](#testing-data)
@@ -138,41 +133,39 @@ The model is a pretrained model in Hindi and dialects which is instruct tuned .
     - [Metrics](#metrics)
   - [Results](#results)
 - [Model Examination](#model-examination)
-- [Environmental Impact](#environmental-impact)
-- [Technical Specifications [optional]](#technical-specifications-optional)
+- [Technical Specifications](#technical-specifications)
   - [Model Architecture and Objective](#model-architecture-and-objective)
   - [Compute Infrastructure](#compute-infrastructure)
     - [Hardware](#hardware)
     - [Software](#software)
 - [Citation](#citation)
-- [Glossary [optional]](#glossary-optional)
-- [More Information [optional]](#more-information-optional)
-- [Model Card Authors [optional]](#model-card-authors-optional)
+- [Glossary](#glossary)
+- [More Information](#more-information)
+- [Model Card Authors](#model-card-authors)
 - [Model Card Contact](#model-card-contact)
 - [How to Get Started with the Model](#how-to-get-started-with-the-model)
 
-
 # Model Details
 
 ## Model Description
 
+Project Indus LLM aims to provide a robust language model for Indian languages, starting with Hindi and its dialects. This open-source foundational model, hosted on Hugging Face, is tailored for easy integration and further development by researchers and developers focusing on Indian linguistic diversity.
+
 <!-- Provide a longer summary of what this model is/does. -->
-TThe model is a pretrained model in Hindi and dialects which is instruct tuned.
+The model is a pretrained model in Hindi and dialects which is instruct tuned.
 
 - **Developed by:** Nikhil Malhotra, Nilesh Brahme, Satish Mishra, Vinay Sharma (Makers Lab, TechMahindra)
 - **Model type:** Foundational Language model
 - **Language(s) (NLP):** hin, bho, mai, doi
 - **License:** other
 - **Parent Model:** It is a grounds up model built on GPT-2 architecture starting from tokenizer to decoder
-- **Resources for more information:** https://www.techmahindra.com/en-in/innovation/the-indus-project/
+- **Resources for more information:** <https://www.techmahindra.com/en-in/innovation/the-indus-project/>
   
-
-
-
 # Uses
 
 <!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
 Uses include question and answeting and conversation in Hindi and Dialects. The model would be reward tuned to be used across various industries
+
 1. Call center
 2. Healthcare
 3. Automotive
@@ -182,200 +175,272 @@ Uses include question and answeting and conversation in Hindi and Dialects. The
 
 <!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
 <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
-Direct use is as a foundationla model on Hindi and dialects
-
-
+Project Indus can be directly used for generating text, simulating conversation, and other text generation tasks without additional training.
 
-## Downstream Use [Optional]
+## Downstream Use
 
 <!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
 <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
- Uses include question and answeting and conversation in Hindi and Dialects. The model would be reward tuned to be used across various industries
+
+Uses include question and answeting and conversation in Hindi and Dialects. The model would be reward tuned to be used across various industries
+
 1. Call center
 2. Healthcare
 3. Automotive
 4. Telecom
 
-
-
-
 ## Out-of-Scope Use
 
 <!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
 <!-- If the user enters content, print that. If not, but they enter a task in the list, use that. If neither, say "more info needed." -->
-Cannot be used for fill in the blanks, Multiple Q&A etc. at the moment
-
-
+Project Indus is not designed for high-stakes decision-making tasks such as medical diagnosis or legal advice, nor can it be used for fill-in-the-blank exercises, multiple Q&A, and similar applications at the moment.
 
 # Bias, Risks, and Limitations
 
 <!-- This section is meant to convey both technical and sociotechnical limitations. -->
 
 Significant research has explored bias and fairness issues with language models
-(see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). 
+(see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)).
 Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups.
 We have taken care across various biases by trying to remove them from training data. However since the model is a generative model, it would tend to produce hallucinations.
-Any disturbing or harmful sterotype produced by the model is purely un-intentional and coincidental. 
-
+Any disturbing or harmful sterotype produced by the model is purely un-intentional and coincidental.
 
 ## Recommendations
 
 <!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
-Recommendation is to not use biases and negative connotation for the model
 
+It is recommended to avoid biases and negative connotations in the model, and regular updates along with community feedback are crucial for addressing any emergent bias or misuse scenarios.
 
-##Indic EVAL Leaderboard
+# Training Details
 
-![image/png](https://cdn-uploads.huggingface.co/production/uploads/63a336a6b5fc9ab9f63e0805/flXcUdwLvi4YH3lfx7Sir.png)
+The model was trained on a curated dataset comprising various sources of Hindi text, including literature, news articles, and web content.
 
-# Training Details
+## Infrastructure
+
+- **Training Infrastructure:** Utilized high-performance computing resources provided by CDAC, featuring NVIDIA A100 GPUs.
+- **Running Infrastructure:** Tested for both GPU (NVIDIA GeForce RTX 3070 or higher) and CPU (Intel Xeon Platinum 8580) environments.
 
 ## Training Data
 
 <!-- This should link to a Data Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+The Project Indus LLM was trained on a diverse and extensive dataset comprising various sources of Hindi text and its dialects. The data collection and curation process was meticulously designed to cater to the linguistic diversity and complexity of Indian languages, particularly focusing on Hindi and its 37 dialects.
+
+### Data Sources and Collection
+
+Data was collected in three main buckets:
+
+1. **Open-Source Hindi Data**: This included publicly available sources from the internet across different categories such as news, and non-news. Automated scripts were used to scrape and extract text from web pages. Here are some of the sources:
+   - **News**: Articles from major Hindi news portals like Amar Ujala, Bhaskar, India TV, Jagran, Live Hindustan, and Patrika.
+   - **Non-News**: Diverse sources including Wikipedia, commoncrawl.org, and other culturally significant content like 'Man ki Baat' from AIR.
 
-More information on training data needed
+2. **Translated Data**: A portion of the Pile dataset, which is a large English dataset used for training AI models, was translated into Hindi using three different translation models. IndicTrans2 (AI4Bharat) was selected as the best model for this purpose based on its accuracy and efficiency.
 
+3. **Dialects**: Data collection for dialects presented a unique challenge due to the limited material available on the internet. Data for major dialects like Maithili, Bhojpuri, Magahi, and Braj Bhasha was collected from multiple sources, including fieldwork where representatives collected old books and other texts, which were then digitized and converted into text data.
 
 ## Training Procedure
 
 <!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
 
-### Preprocessing
+Training involved extensive preprocessing to clean and standardize the text, followed by supervised learning on a high-performance computing setup.
 
-More information needed
+- **Pre-training:** Conducted on a dataset of 22 billion tokens using advanced tokenization techniques.
+- **Fine-Tuning:** Supervised fine-tuning performed with a focus on Indian languages, utilizing datasets specifically tailored for cultural, political, and social contexts.
 
-### Speeds, Sizes, Times
+Below is a table summarizing the datasets used for pre-training and fine-tuning the model:
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+| Phase         | Data Source                             | Tokens    | Notes                                               |
+|---------------|-----------------------------------------|-----------|-----------------------------------------------------|
+| Pre-training  | Cleaned dataset of Hindi and dialects   | 22 billion| Utilized advanced tokenization                      |
+| Fine-tuning   | Custom datasets tailored for Indian languages | Varied    | Focus on cultural, political, and social contexts   |
 
-More information needed
- 
-# Evaluation
+- **Training Infrastructure:** Utilized high-performance computing resources provided by CDAC, featuring NVIDIA A100 GPUs.
+- **Running Infrastructure:** Tested for both GPU (NVIDIA GeForce RTX 3070 or higher) and CPU (Intel Xeon Platinum 8580) environments.
+
+### Preprocessing
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+The collected data underwent several stages of cleaning and preprocessing to ensure high quality and usability for training:
 
-## Testing Data, Factors & Metrics
+- **Cleaning**: The data was cleaned of unwanted text, characters, and personal information like mobile numbers. Transliteration was performed where necessary, and unwanted tags from scraped web pages were removed.
+- **Bias Removal**: A Bias Removal Toolkit was developed to detect and remove biased language from the training data. This toolkit helped in ensuring that the text used for training the model was ethical, correct, and socially responsible.
+- **Tokenization**: The data was tokenized using a custom tokenizer developed specifically for Hindi and its dialects. This tokenizer was based on Byte Pair Encoding (BPE) with additional mechanisms like byte fallback to handle the peculiarities of Hindi script efficiently.
 
-### Testing Data
+#### Summary
 
-<!-- This should link to a Data Card if possible. -->
+The final dataset used for training consisted of:
 
-More information needed
+- **Raw Data Size**: Over 500 GB of raw data collected.
+- **Cleaned and Curated Data**: Approximately 200 GB of clean Hindi and dialect text data.
+- **Tokenization**: Utilized 22 billion tokens created from the cleaned data for pre-training.
 
+This diverse and extensive training data foundation allowed Project Indus LLM to develop robust capabilities for understanding and generating Hindi text, making it a powerful tool for applications requiring Indian language processing.
 
-### Factors
+# Evaluation
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+### Indic LLM Leaderboard Results
 
-More information needed
+Project Indus LLM has been evaluated using the Indic LLM Leaderboard, which employs the `indic_eval` evaluation framework specifically designed for assessing models on Indian language tasks. This framework provides a comprehensive view of model performance across a variety of benchmarks tailored to Indian languages.
 
-### Metrics
+Detailed results from the Indic LLM Leaderboard (α), accessible at [Hugging Face Indic LLM Leaderboard](https://huggingface.co/spaces/Cognitive-Lab/indic_llm_leaderboard), are shown below:
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+|              Task              | Version |  Metric  | Value |   | Stderr |
+|--------------------------------|---------|----------|-------|---|--------|
+| All                            |         | acc      | 0.2891| ± | 0.0109 |
+|                                |         | acc_norm | 0.3013| ± | 0.0112 |
+| indiceval:ARC-Challenge:hindi:10 |    0    | acc      | 0.2167| ± | 0.0120 |
+|                                |         | acc_norm | 0.2474| ± | 0.0126 |
+| indiceval:ARC-Easy:hindi:5     |    0    | acc      | 0.3615| ± | 0.0099 |
+|                                |         | acc_norm | 0.3552| ± | 0.0098 |
 
-More information needed
+These results highlight the model's capabilities in understanding and generating Hindi language text under controlled testing conditions. The standard error values indicate the variance observed during the evaluation, providing insights into the consistency of the model's performance across different evaluation runs.
 
-## Results 
+### Open LLM Leaderboard Evaluation Results
 
-More information needed
+Additionally, Project Indus LLM has been evaluated on the Open LLM Leaderboard, which provides another layer of benchmarking by comparing the model's performance against other state-of-the-art language models. Below are the summarized results from the Open LLM Leaderboard:
 
-# Model Examination
+|             Metric              |Value|
+|---------------------------------|----:|
+|Avg.                             |20.07|
+|AI2 Reasoning Challenge (25-Shot)|22.70|
+|HellaSwag (10-Shot)              |25.04|
+|MMLU (5-Shot)                    |23.12|
+|Winogrande (5-shot)              |49.57|
 
-More information needed
+These benchmark results can be explored further on [Hugging Face Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard).
 
-# Environmental Impact
+### Evaluation Context
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+The evaluation metrics `acc` (accuracy) and `acc_norm` (normalized accuracy) are used to quantify the model's performance. The tasks are differentiated by their difficulty and the specific dataset used, such as the ARC Challenge and ARC Easy sets, both adapted to Hindi language conditions to ensure relevant assessment. This structured evaluation ensures that the Indus LLM not only performs well in generalized text generation tasks but also in more specialized, context-specific scenarios pertinent to the Indian linguistic framework.
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+## Results
 
-- **Hardware Type:** More information needed
-- **Hours used:** More information needed
-- **Cloud Provider:** More information needed
-- **Compute Region:** More information needed
-- **Carbon Emitted:** More information needed
+Project Indus demonstrates competitive performance, particularly in text generation tasks, as evidenced by its scores on standardized benchmarks.
 
-# Technical Specifications [optional]
+# Technical Specifications
 
 ## Model Architecture and Objective
 
-More information needed
+Project Indus LLM is based on a GPT-2.0-like architecture, tailored to handle the complexities of the Hindi language and its dialects. This model was designed to serve as a foundational model that can be fine-tuned for various applications, making it highly versatile and adaptable to different domains within the Indian context.
+
+- **Architecture Details**:
+  - **Layers**: 22 transformer layers, which provide a deep neural network capable of understanding complex language patterns.
+  - **Heads**: 32 attention heads per layer, facilitating a broad attention mechanism across different parts of the input data.
+  - **Embedding Size**: 2048, which allows the model to represent a wide variety of information and nuances in the data.
+  - **Vocabulary Size**: 32,300, tailored to include a comprehensive set of Hindi words and common phrases found in the training data.
+
+The objective of this model is to provide a robust tool for text generation and understanding in Hindi and its dialects, supporting the development of applications that require natural language processing in these languages. It also aims to bridge the gap in technology where Indian languages are underrepresented, providing a platform for further linguistic research and technological inclusion.
 
 ## Compute Infrastructure
 
-More information needed
+##### Hardware
+
+The pre-training and fine-tuning of Project Indus LLM were conducted on high-performance computing infrastructure provided by the Centre for Development of Advanced Computing (CDAC). This setup included:
 
-### Hardware
+- **Nodes and GPUs**: Utilization of six nodes, each equipped with eight NVIDIA A100 GPUs. These GPUs are state-of-the-art for machine learning tasks and provide the necessary computational power to handle the large volumes of data and complex model architectures.
+- **Memory and Storage**: Each node was equipped with ample memory and storage to handle the datasets and model parameters efficiently. Specific configurations included 40 GB of GPU memory per card, essential for training large models.
 
-More information needed
+##### Software
 
-### Software
+The software environment was crucial for efficiently training and running the model. Key components included:
 
-More information needed
+- **Operating System**: Linux, chosen for its stability and support for high-performance computing tasks.
+- **Machine Learning Frameworks**: PyTorch, used for its flexibility and efficiency in training deep learning models. It supports extensive parallel processing and GPU acceleration, which are critical for training large models like Project Indus LLM.
+- **Job Scheduler**: SLURM (Simple Linux Utility for Resource Management) was used to manage and allocate resources effectively across the distributed system. This ensured optimal scheduling of training jobs without resource contention.
 
 # Citation
 
 <!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
 
-**BibTeX:**
+The detailed citation information will help in acknowledging the work and efforts of the team behind Project Indus LLM when it is used or referenced in academic or professional settings.
 
-More information needed
+```bibtex
+@article{malhotra2024projectindus,
+  title={Project Indus: A Foundational Model for Indian Languages},
+  author={Malhotra, Nikhil and Brahme, Nilesh and Mishra, Satish and Sharma, Vinay},
+  journal={Tech Mahindra Makers Lab},
+  year={2024},
+  url={https://www.techmahindra.com/en-in/innovation/the-indus-project/}
+}
+```
 
 **APA:**
+Malhotra, N., Brahme, N., Mishra, S., & Sharma, V. (2024). Project Indus: A Foundational Model for Indian Languages. *Tech Mahindra Makers Lab*. Available at <https://www.techmahindra.com/en-in/innovation/the-indus-project/>
 
-More information needed
-
-# Glossary [optional]
+# Glossary
 
 <!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
 
-More information needed
-
-# More Information [optional]
-
-More information needed
+This glossary section explains key terms used throughout the model documentation and technical details, helping users unfamiliar with certain concepts to better understand the content.
 
-# Model Card Authors [optional]
+- **Transformer Layers**: Part of a neural network architecture that uses self-attention mechanisms to process sequential data such as text. Essential for NLP tasks.
+- **Attention Heads**: Sub-units of a model layer that allow the model to focus on different parts of the input sequence when making predictions.
+- **Embedding Size**: The size of the vector used to represent each token or word in a dense numerical form. Larger embeddings can capture more detailed information.
+- **Block Size**: The maximum length of the input tokens the model can process in one operation.
+- **Vocabulary Size**: The total number of unique words or tokens that the model can understand and generate.
 
-<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
+# More Information
 
-Nikhil Malhotra, Nilesh Brahme, Vinay Sharma, Satish Mishra
+For further details on Project Indus LLM, including additional documentation, tutorials, and community discussions, visit the following resources:
 
-# Model Card Contact
+- **Project Repository**: [Hugging Face Repository](https://huggingface.co/nickmalhotra/ProjectIndus)
+- **Tech Mahindra Makers Lab**: Insights into the research and development behind Project Indus can be found on the [Tech Mahindra Innovation page](https://www.techmahindra.com/en-in/innovation/makers-lab/).
+- **Community Forums**: Engage with the community on [Hugging Face Forums](https://huggingface.co/nickmalhotra/ProjectIndus/discussions?status=open&type=discussion) for support, brainstorming, and sharing of new ideas related to Project Indus.
 
-More information needed
+# Model Card Authors
 
-# How to Get Started with the Model
+<!-- This section provides another layer of transparency and accountability. Whose views is this model card representing? How many voices were included in its construction? Etc. -->
 
-Use the code below to get started with the model.
-# Use a pipeline as a high-level helper
-from transformers import pipeline
-pipe = pipeline("text-generation", model="nickmalhotra/Indus_1.175B")
+The model card and documentation for Project Indus LLM were collaboratively authored by:
 
-# Load model directly
-from transformers import AutoTokenizer, AutoModelForCausalLM
+- **Nikhil Malhotra**: Chief Innovation Officer at Tech Mahindra.
+- **Nilesh Brahme**: Senior AI Research Scientist and one of the primary contributors to the Project Indus development.
+- [**Satish Mishra**](https://huggingface.co/zicsx): AI Architect, whose insights have significantly shaped the model's capabilities.
+- **Vinay Sharma**: LLM Engineer focused on the linguistic data processing and model training aspects of Project Indus.
 
-tokenizer = AutoTokenizer.from_pretrained("nickmalhotra/Indus_1.175B")
+# Model Card Contact
 
-model = AutoModelForCausalLM.from_pretrained("nickmalhotra/Indus_1.175B")
+For inquiries, support, or further information regarding Project Indus LLM, please reach out through the following channels:
 
-<details>
-<summary> Click to expand </summary>
+- **Email**: [projectindus@techmahindra.com](mailto:projectindus@techmahindra.com) - For direct queries and professional engagements.
+- **GitHub Issues**: For technical issues, feature requests, or contributions, please use the Issues section of the [Project Indus GitHub repository](https://github.com/Tech-Mahindra-Makers-Lab/Indus-1.1B).
+- **Hugging Face Spaces**: Questions and discussions related to model implementation and community projects can be posted in our dedicated space on Hugging Face.
 
-More information needed
+# How to Get Started with the Model
 
-</details>
-# [Open LLM Leaderboard Evaluation Results](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard)
-Detailed results can be found [here](https://huggingface.co/datasets/open-llm-leaderboard/details_nickmalhotra__indus_1.175B)
+To begin using Project Indus LLM for your projects, follow these steps to set up and run the model:
 
-|             Metric              |Value|
-|---------------------------------|----:|
-|Avg.                             |20.07|
-|AI2 Reasoning Challenge (25-Shot)|22.70|
-|HellaSwag (10-Shot)              |25.04|
-|MMLU (5-Shot)                    |23.12|
-|TruthfulQA (0-shot)              | 0.00|
-|Winogrande (5-shot)              |49.57|
-|GSM8k (5-shot)                   | 0.00|
+# Load model directly
 
+```python
+from transformers import AutoModel, AutoTokenizer
+
+model = AutoModel.from_pretrained("makers-lab/Indus-1.1B-IT")
+tokenizer = AutoTokenizer.from_pretrained("makers-lab/Indus-1.1B-IT")
+
+# Example inference
+def format_template(user_prompt):
+    messages = [
+        {"role": "user", "content": user_prompt},
+    ]
+    response = tokenizer.apply_chat_template(messages, tokenize=True,  add_generation_prompt=True, return_tensors="pt")
+    return  response
+
+user_prompt = """भारत के वर्तमान प्रधानमंत्री कौन हैं?"""
+
+input_ids = format_template(user_prompt)
+
+# Generate text using the model
+output = model.generate(input_ids,
+                        eos_token_id=tokenizer.eos_token_id,
+                        pad_token_id=tokenizer.eos_token_id,
+                        max_length=1024,
+                        num_beams=5,
+                        do_sample=True,
+                        early_stopping=True,
+                        temperature=0.7,
+                        top_k=50,
+                        top_p=0.95,
+                        repetition_penalty=1.2,
+                        no_repeat_ngram_size=3,
+                        num_return_sequences=1,
+                        )
+print(tokenizer.decode(output[0], skip_special_tokens=False))