metadata

language:
  - en
license: mit
datasets:
  - multi_nli
library_name: transformers
pipeline_tag: zero-shot-classification
tags:
  - t5
  - text-classification
  - mnli
model-index:
  - name: sjrhuschlee/flan-t5-base-mnli
    results:
      - task:
          type: natural-language-inference
          name: Natural Language Inference
        dataset:
          name: MultiNLI-matched
          type: multi_nli
          config: default
          split: validation_matched
        metrics:
          - type: accuracy
            value: 87.468
            name: Accuracy
      - task:
          type: natural-language-inference
          name: Natural Language Inference
        dataset:
          name: MultiNLI-mismatched
          type: multi_nli
          config: default
          split: validation_mismatched
        metrics:
          - type: accuracy
            value: 87.276
            name: Accuracy

flan-t5-base-mnli

flan-t5-base-mnli is the flan-T5 base model fine-tuned on the Multi-Genre Natural Language Inference (MNLI) corpus.

Overview

License: MIT
Language model: flan-t5-base
Language: English
Downstream-task: Zero-shot Classification, Text Classification
Training data: MNLI
Eval data: MNLI (Matched and Mismatched)
Infrastructure: 1x NVIDIA 3070

Model Usage

Use the code below to get started with the model. The model can be loaded with the zero-shot-classification pipeline like so:

from transformers import pipeline
classifier = pipeline(
  'zero-shot-classification',
  model='sjrhuschlee/flan-t5-base-mnli',
  trust_remote_code=True,
)

You can then use this pipeline to classify sequences into any of the class names you specify. For example:

sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)
# {'sequence': 'one day I will see the world',
#  'labels': ['travel', 'cooking', 'dancing'],
#  'scores': [0.7944864630699158, 0.10624771565198898, 0.09926578402519226]}

Metrics

# MNLI
{
    "eval_accuracy": 0.8746816097809476,
    "eval_accuracy_mm": 0.8727624084621644,
    "eval_loss": 0.4271220564842224,
    "eval_loss_mm": 0.4265698492527008,
    "eval_samples": 9815,
    "eval_samples_mm": 9832,
}

Uses

Direct Use

This fine-tuned model can be used for zero-shot classification tasks, including zero-shot sentence-pair classification, and zero-shot sequence classification.

Misuse and Out-of-scope Use

The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

Risks, Limitations and Biases

CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.

Significant research has explored bias and fairness issues with language models (see, e.g., Sheng et al. (2021) and Bender et al. (2021)).

Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:

sequence_to_classify = "The CEO had a strong handshake."
candidate_labels = ['male', 'female']
hypothesis_template = "This text speaks about a {} profession."
classifier(sequence_to_classify, candidate_labels, hypothesis_template=hypothesis_template)

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.