Don Mahurin commited on
Commit
9c30a5b
1 Parent(s): 0543a04

update README

Browse files
Files changed (2) hide show
  1. README.md +31 -0
  2. wtac.py +1 -1
README.md CHANGED
@@ -1,3 +1,34 @@
1
  ---
2
  license: mit
3
  ---
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
  ---
2
  license: mit
3
  ---
4
+
5
+ # Language model trained with reversed words
6
+
7
+ The model weights here were trained using Tiny Stories, but with the words and puncuation reversed.
8
+
9
+ This is described at:
10
+
11
+ https://openright.org/reversed-llm-training/
12
+
13
+ ## Running the reverse inference
14
+
15
+ The model may be ran using these weights with the 'llama2.c' project.
16
+
17
+ The input tokens are expected in reverse, and the output tokens are also in reverse. For example, "This is a test.", becomes ".test a is This".
18
+
19
+ We can simply run with reversed input, and see the reversed output.
20
+
21
+ ```
22
+ ./run revstories15M.bin -s 3 -i '.sky the in butterfly'
23
+ ```
24
+
25
+ > .sky the in butterfly purple beautiful the about family her tell
26
+ > ...
27
+
28
+ But to have the input and output unreversed, we can use the 'wtac' script to reverse the words.
29
+
30
+ ```
31
+ ./run revstories15M.bin -s 3 -i "$(echo 'butterfly in the sky.' | python3 wtac.py)" | python3 ./wtac.py
32
+ ```
33
+
34
+ > Once upon a time, there was a little girl named Lily. She loved to play in the park with her friends. One day, the butterfly landed on Lily's hands and led her to a flower. Lily was very happy and couldn't wait to tell her family about the beautiful purple butterfly in the sky.
wtac.py CHANGED
@@ -1,4 +1,4 @@
1
- #!/usr/bin/python3
2
  # reverses word and punctuation order.
3
  import re
4
  import sys
 
1
+ #!/usr/bin/env python3
2
  # reverses word and punctuation order.
3
  import re
4
  import sys