theblackcat102 commited on
Commit
bbce939
1 Parent(s): 6714f10

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +86 -0
README.md CHANGED
@@ -1,3 +1,89 @@
1
  ---
2
  license: apache-2.0
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: apache-2.0
3
+ datasets:
4
+ - samsum
5
+ - wmt19
6
+ - ChristophSchuhmann/essays-with-instructions
7
+ - gsm8k
8
+ - Anthropic/hh-rlhf
9
+ - xsum
10
+ - cnn_dailymail
11
+ - openai/webgpt_comparisons
12
+ - ted_talks_iwslt
13
+ - math_qa
14
+ - allenai/scitldr
15
+ - oa_not_yet_finish_datasets
16
  ---
17
+ # OA conversation 12B model checkpoint 1k
18
+
19
+ I accidentally checkout my config and current code changes, so I kinda forgot the list of datasets and limit I added in all datasets mix. But basically its not a finish product, but kinda do context selection pretty well
20
+
21
+ When I resumed the training with new config, the newer models aren't as great as I am expecting to be, so I just abandoned the resumed progress and decided to just upload this one.
22
+
23
+ How to ~~start a conversation~~ run the model:
24
+
25
+ Go to [Open Assistant's model_training section](https://github.com/LAION-AI/Open-Assistant/tree/main/model/model_training), we have a tool which are used to start a simple conversation locally
26
+
27
+ ```
28
+ python -m tools.model_chat --model_path theblackcat102/pythia-12B-dedup-1000
29
+ ```
30
+
31
+ ```
32
+ > How many continents on Earth
33
+ 1. There are 7 continents on Earth, Antarctica being the only continent not supported by a landmass. They are: Asia, Africa, North America, South America, Europe, Australia, and Antarctica.
34
+ 2. Antarctica is a continent situated on the southern tip of South America, between Africa and Australia. It is the northernmost continent and the coldest inhabited place in the world.
35
+ 3. The continents are a result of plate tectonics, a process whereby the earths crust moves horizontally, and so is able to break and join in order to make new landmasses. The continents are the result of the different plates of the earth moving apart (the Atlantic plate being the oldest, moving away from Africa and the Pacific being the younger, with Europe being the newest.)
36
+ > Whats the population on Earth
37
+ As of 2019, the total population on Earth is estimated to be around 7,600,000,000 people, with 4,500,000,000 living in developed countries, 1,400,000,000 living in developing countries, and 1,000,000,000 living as refugees, displaced, or in exile.
38
+ > 地球上有多少人
39
+ 总人口估计约为7亿六千万人,总人均寿命约为76岁, 所有领域的发展水平分别为7万千人左右,占世界总人口的四
40
+ 分之一,2亿五千万人左右,占世界总人口的十分之一。 目前,全球四、五分之一的人居住在发展中国家,占世界发展中国家
41
+ 比例约为8万亿人,占全球发展中国家和脱浪,脱脂的人比率约为7亿亿
42
+ > 地球上有幾個大陸板塊
43
+ 1. 地球上有7個大陸板塊,包括亞洲,拉丁美洲,北美洲,歐洲,澳洲,和亞洲島。
44
+ 2. 亞洲島是南美洲海岸的島嶼。 它位於亞洲板塊的外部,大約在拉丁美洲海島嶼和歐洲海岸。
45
+ 3. 大陸板塊是由地震、自然流和地球在不斷的移动和分裂、约束和解放而产生的结果。
46
+ > 幫我把這句話翻譯成中文:a brown fox jumps over the lazy dog
47
+ a 黑色的狐跳過一隻睡覺的狗。
48
+ ```
49
+
50
+ Some test in multilingual conversation, each starting question contain no historical conversation
51
+
52
+ ```
53
+ >Siapa ni (malay or indonesian)
54
+ Joi disini bercakap dengan cara terbaik dari pengeluaran pendapatan, pembayaran utang, pinjaman, dan tindakan pembayaran, karena ini adalah cara yang sangat efektif untuk membantu orang berkebimbangan dengan mempertimbangkannya dan mengelola pendapatan.
55
+ > いらっしゃいます (japanese)
56
+ emtpy token
57
+ > 누구세요 (korean)
58
+ You may be able to help me by explaining your symptoms to me.
59
+ > 你好
60
+ 请问你有关“Joi 的实用助手”如何实现某些行为么?
61
+ ```
62
+
63
+ As of now results are very random and may varies
64
+
65
+ # Issues I am trying to tackle here
66
+
67
+ - Language
68
+
69
+ Pythia was trained on the Pile, an English based corpus with sprinkles of non English words in between. So translation plays a huge role in joining the bytes token sets to learn some kind of pattern for non English language.
70
+
71
+ - History context selection
72
+
73
+ Make sure the heads are learning to decide whether to ignore or attend what's relevant to the latest question. I have some ideas and one of them is random prepend irrelevant context to the sequence prefix.
74
+
75
+ FYI: Does anyone has any idea whether pythia series could do CoT?
76
+
77
+ # Warning
78
+
79
+ This is a model that is only 1% complete and has not yet been fully cooked. I am uploading it here for the purpose of comparison, as these models are taking up more space than I would like
80
+
81
+ Model trained only in SFT
82
+
83
+ # Many thanks
84
+
85
+ - [LAION](https://laion.ai/) & EleutherAI : basically a dozen of individuals who band together to make open research possible in this age of conflict and greed
86
+ - [Stability.ai](https://stability.ai/) : this project wouldn't be possible without their compute resource
87
+ - [Teams and contributors at Open Assistant](https://github.com/LAION-AI/Open-Assistant/graphs/contributors) : who put their time after their day job or whatever into this project
88
+ - [Huggingface](https://huggingface.co/) : For the storage and spaces here
89
+