content: text: Edward Beeching, Clementine Fourrier, Nathan
Habib, Sheon Han, Nathan Lambert, Nazneen
Rajani, Omar Sanseviero, Lewis Tunstall, and
Thomas Wolf. 2023. Open llm leaderboard.
https : //huggingface . co/spaces/
HuggingFaceH4/open_llm_leaderboard.

content: text: Tom Brown, Benjamin Mann, Nick Ryder, Melanie
Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind
Neelakantan, Pranav Shyam, Girish Sastry, Amanda
Askell, et al. 2020. Language models are few-shot
learners. Advances in neural information processing
systems, 33:1877-1901.

content: text: Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot,
Ashish Sabharwal, Carissa Schoenick, and Oyvind
Tafjord. 2018. Think you have solved question an-
swering? try arc, the ai2 reasoning challenge. arXiv
preprint arXiv:1803.05457.

content: text: Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian,
Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias
Plappert, Jerry Tworek, Jacob Hilton, Reiichiro
Nakano, et al. 2021. Training verifiers to solve math
word problems. arXiv preprint arXiv:2110.14168.

content: text: Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao,
Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and
Maosong Sun. 2023. Ultrafeedback: Boosting lan-
guage models with high-quality feedback. arXiv
preprint arXiv:2310.01377.

content: text: Chunyuan Deng, Yilun Zhao, Xiangru Tang, Mark Ger-
stein, and Arman Cohan. 2023. Investigating data
contamination in modern benchmarks for large lan-
guage models. arXiv preprint arXiv:2311.09783.

content: text: Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan,
Shizhe Diao, Jipeng Zhang, Kashun Shum, and
Tong Zhang. 2023. Raft: Reward ranked finetuning
for generative foundation model alignment. arXiv
preprint arXiv:2304.06767.

content: text: Mohammad Fraiwan and Natheer Khasawneh. 2023. A
review of chatgpt applications in education, market-
ing, software engineering, and healthcare: Benefits,
drawbacks, and research directions. arXiv preprint
arXiv:2305.00237.

content: text: Trevor Gale, Deepak Narayanan, Cliff Young, and Matei
Zaharia. 2023. Megablocks: Efficient sparse training
with mixture-of-experts. Proceedings of Machine
Learning and Systems, 5.

content: text: Andrea Gesmundo and Kaitlin Maile. 2023. Compos-
able function-preserving expansions for transformer
architectures. arXiv preprint arXiv:2308.06103.

content: text: Shahriar Golchin and Mihai Surdeanu. 2023. Time
travel in llms: Tracing data contamination in large
language models. arXiv preprint arXiv:2308.08493.

content: text: Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou,
Mantas Mazeika, Dawn Song, and Jacob Steinhardt.
2020. Measuring massive multitask language under-
standing. In International Conference on Learning
Representations.

content: text: Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul
Arora, Steven Basart, Eric Tang, Dawn Song, and Ja-
cob Steinhardt. 2021. Measuring mathematical prob-
lem solving with the math dataset. arXiv preprint
arXiv:2103.03874.

content: text: Danny Hernandez, Jared Kaplan, Tom Henighan, and
Sam McCandlish. 2021. Scaling laws for transfer.
arXiv preprint arXiv:2102.01293.

content: text: Changho Hwang, Wei Cui, Yifan Xiong, Ziyue Yang,
Ze Liu, Han Hu, Zilong Wang, Rafael Salas, Jithin
Jose, Prabhat Ram, et al. 2023. Tutel: Adaptive
mixture-of-experts at scale. Proceedings of Machine
Learning and Systems, 5.

content: text: Intel. 2023. Supervised fine-tuning and direct prefer-
ence optimization on intel gaudi2.

content: text: Hamish Ivison, Yizhong Wang, Valentina Pyatkin,
Nathan Lambert, Matthew Peters, Pradeep Dasigi,
Joel Jang, David Wadden, Noah A. Smith, Iz Belt-
agy, and Hannaneh Hajishirzi. 2023. Camels in a
changing climate: Enhancing lm adaptation with tulu
2.

content: text: Albert Q Jiang, Alexandre Sablayrolles, Arthur Men-
sch, Chris Bamford, Devendra Singh Chaplot, Diego
de las Casas, Florian Bressand, Gianna Lengyel, Guil-
laume Lample, Lucile Saulnier, et al. 2023. Mistral
7b. arXiv preprint arXiv:2310.06825.

content: text: Jean Kaddour, Oscar Key, Piotr Nawrot, Pasquale
Minervini, and Matt J Kusner. 2023. No train no
gain: Revisiting efficient training algorithms for
transformer-based language models. arXiv preprint
arXiv:2307.06440.

content: text: Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B
Brown, Benjamin Chess, Rewon Child, Scott Gray,
Alec Radford, Jeffrey Wu, and Dario Amodei. 2020.
Scaling laws for neural language models. arXiv
preprint arXiv:2001.08361.

content: text: Aran Komatsuzaki, Joan Puigcerver, James Lee-Thorp,
Carlos Riquelme Ruiz, Basil Mustafa, Joshua Ainslie,
Yi Tay, Mostafa Dehghani, and Neil Houlsby.
2022. Sparse upcycling: Training mixture-of-
experts from dense checkpoints. arXiv preprint
arXiv:2212.05055.

content: text: Wing Lian. 2023. https : //huggingface . co/
winglian/omega-3b.

content: text: Stephanie Lin, Jacob Hilton, and Owain Evans. 2022.
Truthfulqa: Measuring how models mimic human
falsehoods. In Proceedings of the 60th Annual Meet-
ing of the Association for Computational Linguistics
(Volume 1: Long Papers), pages 3214-3252.

content: text: Shayne Longpre, Le Hou, Tu Vu, Albert Webson,
Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V
Le, Barret Zoph, Jason Wei, et al. 2023. The flan
collection: Designing data and methods for effective
instruction tuning. arXiv preprint arXiv:2301.13688.