Create README.md

README.md

@@ -0,0 +1,12 @@
+This is a `distilgpt2` model, finetuned on the Wikitext-103 dataset.
+
+It achieves a perplexity of **18.25** using a "sliding window" context, using the `run_clm.py` script at [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers).
+
+| Base LM:        | `distilgpt2` | `gpt2` | 
+| :---            |    ----:   |     ---: |
+| base perplexity | 18.25      | 14.84    |
+| + kNN-LM          |  15.03     |   12.57  |
+| + RetoMaton       | **14.70**  |  **12.46**    |
+
+This model was released as part of the paper ["Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval"](https://arxiv.org/pdf/2201.12431.pdf) (ICML'2022).
+For more information, see: [https://github.com/neulab/knn-transformers](https://github.com/neulab/knn-transformers)