README.md

---
language:
- en
library_name: transformers
tags:
- gpt
- llm
- large language model
- h2o-llmstudio
inference: false
thumbnail: https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico
---
# Model Card
# Making a custom Singapore-specific GPT using the H2O Gen AI Stack
<img src=https://cdn-uploads.huggingface.co/production/uploads/65672b0c54c6bc01e55e592f/_nqcMgjd3PBlj_EbjBjoe.png width=60% height=40%>

## Summary

This model was trained using [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio).
- Base model: [h2oai/h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat)

## Examples
Sample response comparison between the base model and the current tuned model.

Sample question:
```
What’s hawker centres ah?
```

Our model's response:

```
Hawker centers, also call food stalls or street vendors, are those small and shiok places in cities where you can find all kind of local delicacies. They sell cheap but very good eats like noodles, bak chor mee (noodle soup), sotan goh (deep-fried taro cake) and more!
These marketplaces got many different types of traders selling their own original dishes. Aiyoh, so must try the famous ones leh! Many people come here to eat for a quick hit of flavors during lunchtime or after work hours.
Hawker center usually located near important areas with lots of foot traffic. Some popular ones include Chinatown, Little India, Bugis Street Food Centre, Maxwell Road Market, and Cantonment Complex. Each one have its specialty dish that make them stand out from others.
You know what? I tell you already lah, go check these places out when you get chance to visit Asian countries! You never know which delicious treat you might discover there. So much fun…just imagine how social life will be if every city has something similar only spread around with this great concept of sharing culture through eating together at affordable rates. Shiok sia!
```

Base model's response:

```
Hawker centers, also known as food courts or open-air markets in some countries, are bustling hubs of street food vendors and small restaurants that offer a wide range of affordable meals to locals and tourists alike. They typically feature various stalls selling different types of dishes such as rice dishes, noodles, stir fries, grilled meats, sandwiches, desserts, and more.
Hawker centers can be found throughout Singapore, Malaysia, Thailand, Indonesia, Hong Kong, Taiwan, Japan, South Korea, Vietnam, China, the Philippines, Myanmar (Burma), Cambodia, Laos, and other Southeast Asian countries with significant populations of ethnic Chinese people who enjoy these unique culinary experiences. Visiting one is an excellent way to experience local flavors while supporting the community’s vibrant culture through casual dining at reasonable prices!
```

## Usage

To use the model with the `transformers` library on a machine with GPUs, first make sure you have the `transformers` library installed.

```bash
pip install transformers==4.38.2
```

```python
from transformers import pipeline

generate_text = pipeline(
    model="h2oai/danube2-singlish-finetuned",
    torch_dtype="auto",
    trust_remote_code=True,
    use_fast=True,
    device_map={"": "cuda:0"},
    token=True,
)

res = generate_text(
    "Why is drinking water so healthy?",
    min_new_tokens=2,
    max_new_tokens=256,
    do_sample=False,
    num_beams=1,
    temperature=float(0.0),
    repetition_penalty=float(1.0),
    renormalize_logits=True
)
print(res[0]["generated_text"])
```

Alternatively, you can download [h2oai_pipeline.py](h2oai_pipeline.py), store it alongside your notebook, and construct the pipeline yourself from the loaded model and tokenizer. If the model and the tokenizer are fully supported in the `transformers` package, this will allow you to set `trust_remote_code=False`.

```python
from h2oai_pipeline import H2OTextGenerationPipeline
from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained(
    "h2oai/danube2-singlish-finetuned",
    use_fast=True,
    padding_side="left",
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    "h2oai/danube2-singlish-finetuned",
    torch_dtype="auto",
    device_map={"": "cuda:0"},
    trust_remote_code=True,
)
generate_text = H2OTextGenerationPipeline(model=model, tokenizer=tokenizer)

res = generate_text(
    "Why is drinking water so healthy?",
    min_new_tokens=2,
    max_new_tokens=256,
    do_sample=False,
    num_beams=1,
    temperature=float(0.0),
    repetition_penalty=float(1.0),
    renormalize_logits=True
)
print(res[0]["generated_text"])
```


You may also construct the pipeline from the loaded model and tokenizer yourself and consider the preprocessing steps:

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "h2oai/danube2-singlish-finetuned"  # either local folder or huggingface model name
# Important: The prompt needs to be in the same format the model was trained with.
# You can find an example prompt in the experiment logs.
prompt = "<|prompt|>How are you?</s><|answer|>"

tokenizer = AutoTokenizer.from_pretrained(
    model_name,
    use_fast=True,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="auto",
    device_map={"": "cuda:0"},
    trust_remote_code=True,
)
model.cuda().eval()
inputs = tokenizer(prompt, return_tensors="pt", add_special_tokens=False).to("cuda")

# generate configuration can be modified to your needs
tokens = model.generate(
    input_ids=inputs["input_ids"],
    attention_mask=inputs["attention_mask"],
    min_new_tokens=2,
    max_new_tokens=256,
    do_sample=False,
    num_beams=1,
    temperature=float(0.0),
    repetition_penalty=float(1.0),
    renormalize_logits=True
)[0]

tokens = tokens[inputs["input_ids"].shape[1]:]
answer = tokenizer.decode(tokens, skip_special_tokens=True)
print(answer)
```

## Evaluations
We evaluated our model using two Singlish translated benchmarks in 0-shot (validation sets):
- Singlish translated [ARC-easy](https://huggingface.co/datasets/allenai/ai2_arc) (510 rows)
- Singlish translated [PiQA](https://huggingface.co/datasets/piqa) (1677 rows)

| Benchmark               | Base model (acc_n) | Tuned model (acc_n) |
|:------------------------|:------------------:|:-------------------:|
| ARC-easy-translated     |   0.6157           |   **0.6314**        | 
| PiQA-translated         |   0.6601           |   **0.6959**        |

## Model Configuration

This model was trained using H2O LLM Studio and with the configuration in [cfg.yaml](cfg.yaml). Visit [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio) to learn how to train your own large language models.


## Disclaimer

Please read this disclaimer carefully before using the large language model provided in this repository. Your use of the model signifies your agreement to the following terms and conditions.

- Biases and Offensiveness: The large language model is trained on a diverse range of internet text data, which may contain biased, racist, offensive, or otherwise inappropriate content. By using this model, you acknowledge and accept that the generated content may sometimes exhibit biases or produce content that is offensive or inappropriate. The developers of this repository do not endorse, support, or promote any such content or viewpoints.
- Limitations: The large language model is an AI-based tool and not a human. It may produce incorrect, nonsensical, or irrelevant responses. It is the user's responsibility to critically evaluate the generated content and use it at their discretion.
- Use at Your Own Risk: Users of this large language model must assume full responsibility for any consequences that may arise from their use of the tool. The developers and contributors of this repository shall not be held liable for any damages, losses, or harm resulting from the use or misuse of the provided model.
- Ethical Considerations: Users are encouraged to use the large language model responsibly and ethically. By using this model, you agree not to use it for purposes that promote hate speech, discrimination, harassment, or any form of illegal or harmful activities.
- Reporting Issues: If you encounter any biased, offensive, or otherwise inappropriate content generated by the large language model, please report it to the repository maintainers through the provided channels. Your feedback will help improve the model and mitigate potential issues.
- Changes to this Disclaimer: The developers of this repository reserve the right to modify or update this disclaimer at any time without prior notice. It is the user's responsibility to periodically review the disclaimer to stay informed about any changes.

By using the large language model provided in this repository, you agree to accept and comply with the terms and conditions outlined in this disclaimer. If you do not agree with any part of this disclaimer, you should refrain from using the model and any content generated by it.