ohwi
/

japanese-stablelm-instruct-gamma-7b-dpo-uf-v0

Text Generation

japanese-stablelm

Inference Endpoints

text-generation-inference

Model card Files Files and versions Community

ohwi commited on Dec 23, 2023

Commit

f731411

•

1 Parent(s): a96fc50

Update README.md

Update benchmark results

Files changed (1) hide show

README.md +6 -6

README.md CHANGED Viewed

@@ -28,18 +28,18 @@ This model is trained with [notus](https://github.com/argilla-io/notus) code bas
 ### Training Datasets
-- [Machine Translated Ultrafeedback dataset](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)
 The dataset is machine translated version of Ultrafeedback. Some samples are missing because of API request failure.
 Will redeem the dataset and train again.
-### Benchmarks (WIP)
-| Model                               | Average   | jcommonsenseqa | jnli | marc_ja   | jsquad (exact) | jaqket_v2 | xlsum_ja  | xwinograd_ja | mgsm |
-|-------------------------------------|-----------|----------------|------|-----------|----------------|-----------|-----------|--------------|------|
-| japanese-stablelm-instruct-gamma-7b |           | 83.47          |      | **95.79** | **76.29**      |           | 21.47     |              |      |
-| this model                          |           | **87.04**      |      | 95.65     | 75.30          |           | **22.25** |              |      |
 These benchmark performances are evaluated by [JP Language Model Evaluation Harness](https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable).

 ### Training Datasets
+- Machine Translated [Ultrafeedback dataset](https://huggingface.co/datasets/argilla/ultrafeedback-binarized-preferences-cleaned)
 The dataset is machine translated version of Ultrafeedback. Some samples are missing because of API request failure.
 Will redeem the dataset and train again.
+### Benchmarks
+| Model                               | Average   | jcommonsenseqa | jnli      | marc_ja   | jsquad    | jaqket_v2 | xlsum_ja  | xwinograd_ja | mgsm      |
+|-------------------------------------|-----------|----------------|-----------|-----------|-----------|-----------|-----------|--------------|-----------|
+| japanese-stablelm-instruct-gamma-7b | 59.86     | 83.47          | 18.65     | **95.79** | **76.29** | **82.13** | 21.47     | 81.44        | 19.60     |
+| this model                          | **63.28** | **87.04**      | **43.84** | 95.65     | 75.30     | 80.24     | **22.25** | **81.54**    | **20.40** |
 These benchmark performances are evaluated by [JP Language Model Evaluation Harness](https://github.com/Stability-AI/lm-evaluation-harness/tree/jp-stable).