Update README.md

README.md

@@ -11,11 +11,37 @@ inference: false
 thumbnail: https://h2o.ai/etc.clientlibs/h2o/clientlibs/clientlib-site/resources/images/favicon.ico
 ---
 # Model Card
+# Making a custom Singapore-specific GPT using the H2O Gen AI Stack
+<img src=https://cdn-uploads.huggingface.co/production/uploads/65672b0c54c6bc01e55e592f/_nqcMgjd3PBlj_EbjBjoe.png width=60% height=40%>
+
 ## Summary
 
 This model was trained using [H2O LLM Studio](https://github.com/h2oai/h2o-llmstudio).
 - Base model: [h2oai/h2o-danube2-1.8b-chat](https://huggingface.co/h2oai/h2o-danube2-1.8b-chat)
 
+## Examples
+Sample response comparison between the base model and the current tuned model.
+
+Sample question:
+```
+What’s hawker centres ah?
+```
+
+Our model's response:
+
+```
+Hawker centers, also call food stalls or street vendors, are those small and shiok places in cities where you can find all kind of local delicacies. They sell cheap but very good eats like noodles, bak chor mee (noodle soup), sotan goh (deep-fried taro cake) and more!
+These marketplaces got many different types of traders selling their own original dishes. Aiyoh, so must try the famous ones leh! Many people come here to eat for a quick hit of flavors during lunchtime or after work hours.
+Hawker center usually located near important areas with lots of foot traffic. Some popular ones include Chinatown, Little India, Bugis Street Food Centre, Maxwell Road Market, and Cantonment Complex. Each one have its specialty dish that make them stand out from others.
+You know what? I tell you already lah, go check these places out when you get chance to visit Asian countries! You never know which delicious treat you might discover there. So much fun…just imagine how social life will be if every city has something similar only spread around with this great concept of sharing culture through eating together at affordable rates. Shiok sia!
+```
+
+Base model's response:
+
+```
+Hawker centers, also known as food courts or open-air markets in some countries, are bustling hubs of street food vendors and small restaurants that offer a wide range of affordable meals to locals and tourists alike. They typically feature various stalls selling different types of dishes such as rice dishes, noodles, stir fries, grilled meats, sandwiches, desserts, and more.
+Hawker centers can be found throughout Singapore, Malaysia, Thailand, Indonesia, Hong Kong, Taiwan, Japan, South Korea, Vietnam, China, the Philippines, Myanmar (Burma), Cambodia, Laos, and other Southeast Asian countries with significant populations of ethnic Chinese people who enjoy these unique culinary experiences. Visiting one is an excellent way to experience local flavors while supporting the community’s vibrant culture through casual dining at reasonable prices!
+```
 
 ## Usage
 
@@ -25,14 +51,6 @@ To use the model with the `transformers` library on a machine with GPUs, first m
 pip install transformers==4.38.2
 ```
 
-Also make sure you are providing your huggingface token to the pipeline if the model is lying in a private repo.
-    - Either leave `token=True` in the `pipeline` and login to hugginface_hub by running
-        ```python
-        import huggingface_hub
-        huggingface_hub.login(<ACCESS_TOKEN>)
-        ```
-    - Or directly pass your <ACCESS_TOKEN> to `token` in the `pipeline`
-
 ```python
 from transformers import pipeline
 
@@ -58,16 +76,6 @@ res = generate_text(
 print(res[0]["generated_text"])
 ```
 
-You can print a sample prompt after the preprocessing step to see how it is feed to the tokenizer:
-
-```python
-print(generate_text.preprocess("Why is drinking water so healthy?")["prompt_text"])
-```
-
-```bash
-<|prompt|>Why is drinking water so healthy?</s><|answer|>
-```
-
 Alternatively, you can download [h2oai_pipeline.py](h2oai_pipeline.py), store it alongside your notebook, and construct the pipeline yourself from the loaded model and tokenizer. If the model and the tokenizer are fully supported in the `transformers` package, this will allow you to set `trust_remote_code=False`.
 
 ```python
@@ -144,40 +152,15 @@ answer = tokenizer.decode(tokens, skip_special_tokens=True)
 print(answer)
 ```
 
-## Quantization and sharding
-
-You can load the models using quantization by specifying ```load_in_8bit=True``` or ```load_in_4bit=True```. Also, sharding on multiple GPUs is possible by setting ```device_map=auto```.
-
-## Model Architecture
+## Evaluations
+We evaluated our model using two Singlish translated benchmarks in 0-shot (validation sets):
+- Singlish translated [ARC-easy](https://huggingface.co/datasets/allenai/ai2_arc) (510 rows)
+- Singlish translated [PiQA](https://huggingface.co/datasets/piqa) (1677 rows)
 
-```
-MistralForCausalLM(
-  (model): MistralModel(
-    (embed_tokens): Embedding(32000, 2560, padding_idx=0)
-    (layers): ModuleList(
-      (0-23): 24 x MistralDecoderLayer(
-        (self_attn): MistralSdpaAttention(
-          (q_proj): Linear(in_features=2560, out_features=2560, bias=False)
-          (k_proj): Linear(in_features=2560, out_features=640, bias=False)
-          (v_proj): Linear(in_features=2560, out_features=640, bias=False)
-          (o_proj): Linear(in_features=2560, out_features=2560, bias=False)
-          (rotary_emb): MistralRotaryEmbedding()
-        )
-        (mlp): MistralMLP(
-          (gate_proj): Linear(in_features=2560, out_features=6912, bias=False)
-          (up_proj): Linear(in_features=2560, out_features=6912, bias=False)
-          (down_proj): Linear(in_features=6912, out_features=2560, bias=False)
-          (act_fn): SiLU()
-        )
-        (input_layernorm): MistralRMSNorm()
-        (post_attention_layernorm): MistralRMSNorm()
-      )
-    )
-    (norm): MistralRMSNorm()
-  )
-  (lm_head): Linear(in_features=2560, out_features=32000, bias=False)
-)
-```
+| Benchmark               | Base model (acc_n) | Tuned model (acc_n) |
+|:------------------------|:------------------:|:-------------------:|
+| ARC-easy-translated     |   0.6157           |   **0.6314**        | 
+| PiQA-translated         |   0.6601           |   **0.6959**        |
 
 ## Model Configuration