language:
- en
datasets:
- natural_instructions
- the_pile
- cot
- Muennighoff/P3
tags:
- gpt
pipeline_tag: text-generation
inference: true
widget:
- text: 'Where is Zurich? Ans:'
- text: 'What is the highest mountain? Answer:'
TOGETHER
!!! Be careful, this repo is still under construction. The content might change recently. !!!
Model Summary
We present Together-GPT-J-6B-ProxAdam-50x, capable of following human instructions and conduct zero/few-shot inference. The model trained in a decentralized fashion with ProxAdam optimizer, requiring only 2% cross-machine communication compared to vanilla data parallel training.
Quick Start
from transformers import pipeline
pipe = pipeline(model='togethercomputer/Together-gpt-J-6B-ProxAdam-50x')
pipe("Where is Zurich? Ans:")
Training Data
We fine-tune GPT-J-6B on NI, P3, COT, the pile data.
The pile is used to keep the general ability of GPT-J. Others are instruction-tuning datasets.
Hyperparameters
We used AdamW with a learning rate of 1e-5 and global batch size of 64, and train for 5k steps. We used mix-precision training where the activation is in FP16 while the optimizer states are kept in FP32. We truncate the input sequence to 2048 tokens, and for input sequence that contains less than 2048 tokens, we concatenate multiple sequences into one long sequence to improve the data efficiency.
Infrastructure
We used the Together Research Computer to conduct training. Specifically, we used 4 data parallel workers, each containing 2 * A100 80GB GPUs. Together Research Computer connects clusters at Stanford University, ETH Zurich, Open Science Grid, and University of Wisconsin-Madison.