Edit model card

Model Card for Backpack-GPT2

The Backpack-GPT2 language model is an instance of the Backpack architecture, intended to combine strong modeling performance with an interface for interpretability and control. Most details about this model and its training should be accessed in the paper, Backpack Language Models.

See also backpackmodels.science.

A depiction of the Backpack language modeling process, in which each word in the sequence is weighted and summed to predict each word in context.

Table of Contents

Model Details

Model Description

The Backpack-GPT2 is a Backpack-based language model, an architecture intended to combine strong modeling performance with an interface for interpretability and control.

  • Developed by: John Hewitt, John Thickstun, Christopher D. Manning, Percy Liang
  • Shared by [Optional]: More information needed
  • Model type: Language model
  • Language(s) (NLP): en
  • License: apache-2.0
  • Resources for more information:

Uses

This model is intended for use in the study and development of increasingly interpretable methods in natural language processing. It is not directly fit for any production use.

Bias, Risks, and Limitations

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)). Predictions generated by the model may include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. This model in particular is limited in its capabilities, and with a brand new architecture, less is known about its biases than, e.g., Transformer-based models.

How to Get Started with the Model

import torch
from transformers import AutoConfig, AutoModelForCausalLM

model_id = "stanfordnlp/backpack-gpt2"
config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
torch_model = AutoModelForCausalLM.from_pretrained(model_id, config=config, trust_remote_code=True)
torch_model.eval()

input = torch.randint(0, 50264, (1, 512), dtype=torch.long)
torch_out = torch_model(
    input,
    position_ids=None,
)
torch_out = torch.nn.functional.softmax(torch_out.logits, dim=-1)
print(torch_out)

Training Details

Training Data

This model was trained on the OpenWebText corpus.

Training Procedure

This model was trained for 100k gradient steps with a batch size of 512k tokens and a linearly decaying learning rate from 6e-4 to zero, with a linear warmup of 5k steps.

Environmental Impact

  • Hardware Type: 4 A100 GPUs (40G)
  • Hours used: Roughly 4 days.
  • Cloud Provider: Stanford compute.
  • Compute Region: Stanford energy grid.

Model Architecture and Objective

This model was trained to minimize the cross-entropy loss, and is a Backpack language model.

Compute Infrastructure

This model was trained on a slurm cluster.

Hardware

This model was trained on 4 A100s.

Software

This model was trained with FlashAttention and PyTorch

Citation

BibTeX:

@InProceedings{hewitt2023backpack,
  author =      "Hewitt, John and Thickstun, John and Manning, Christopher D. and Liang, Percy",
  title =       "Backpack Language Models",
  booktitle =   "Proceedings of the Association for Computational Linguistics",
  year =        "2023",
  publisher =   "Association for Computational Linguistics",
  location =    "Toronto, Canada",
}

Model Card Authors [optional]

John Hewitt

Model Card Contact

johnhew@cs.stanford.edu

Downloads last month
141
Inference Examples
Inference API (serverless) does not yet support model repos that contain custom code.

Dataset used to train stanfordnlp/backpack-gpt2

Spaces using stanfordnlp/backpack-gpt2 2