neulab/distilgpt2-finetuned-wikitext103

This is a distilgpt2 model, finetuned on the Wikitext-103 dataset.

It achieves a perplexity of 18.25 using a "sliding window" context, using the run_clm.py script at https://github.com/neulab/knn-transformers.

Base LM:	distilgpt2	gpt2
base perplexity	18.25	14.84
+ kNN-LM	15.03	12.57
+ RetoMaton	14.70	12.46

This model was released as part of the paper "Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval" (ICML'2022).

For more information, see: https://github.com/neulab/knn-transformers

If you use this model, please cite:

@inproceedings{alon2022neuro,
  title={Neuro-Symbolic Language Modeling with Automaton-augmented Retrieval},
  author={Alon, Uri and Xu, Frank and He, Junxian and Sengupta, Sudipta and Roth, Dan and Neubig, Graham},
  booktitle={International Conference on Machine Learning},
  pages={468--485},
  year={2022},
  organization={PMLR}
}