|
--- |
|
license: apache-2.0 |
|
datasets: |
|
- BAAI/COIG-PC |
|
- ehartford/dolphin |
|
- emozilla/booksum-summary-analysis_llama-8192 |
|
- OpenLeecher/GPT4-10k |
|
- 0x70DA/stackoverflow-chat-data |
|
- togethercomputer/Long-Data-Collections |
|
--- |
|
|
|
## RWKV 7B world focus on reading comprehension |
|
|
|
This is a experimental model based on RWKV 7B world. |
|
|
|
why this model is special? ===> |
|
remove eod, add special token, change vocabs. |
|
|
|
this model is used to QA in large texts, do some in context learning with knowledge indexed database. |
|
|
|
|
|
## trainning details |
|
|
|
train with this kind of new format, |
|
|
|
```<s>User: <sys>xxxx\n\n</sys>xxxxx</s><s>Assistant: xxxxx\n\n</s><s>User: xxxx\n\n</s><s>Assistant: \n\n</s>``` |
|
|
|
so ,use User Assistant as your prefix names. |
|
and when inference in RWKV runner, just use the following format is fine. |
|
|
|
User: xxxx\n\nAssistant: xxxx\n\n，in which are the test cases used. |
|
|
|
|
|
-------------------------------------------- |
|
to use this model with RWKV runner,some effort needed, copy back-python folder to a new one ,which is in the same folder with rwkv-runner.exe(or the file to run) , then pastin rwkv_vocab_v20230424.txt into rwkv_pip folder to replace the vocabs file |
|
|
|
../py310/python main.py in this new folder, then use RWKV runner setting API to 127.0.0.0.1:8000, and go to 127.0.0.1:8000/docs to switch model using this one |
|
|
|
try different temp and topp , 1.2 0.5 may works. |
|
|
|
|
|
 |
|
|
|
|
|
 |
|
|
|
|
|
|
|
 |
|
|
|
|
|
temp 1.2 topp 0.6 |
|
 |
|
|
