README.md

---
pipeline_tag: text-generation
license: apache-2.0
---

Artifacts for the GoldFinch paper

Code available at https://github.com/recursal/GoldFinch-paper

Contains checkpoints for various small scale ablations and 1.5B scale models, all trained on minipile

## Bibtex citation

```bibtex
@misc{goldstein2024goldfinchhighperformancerwkvtransformer,
      title={GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compression}, 
      author={Daniel Goldstein and Fares Obeid and Eric Alcaide and Guangyu Song and Eugene Cheah},
      year={2024},
      eprint={2407.12077},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2407.12077}, 
}
```