README.md

---
language:
- en
license: mit
datasets:
- multi_nli
library_name: transformers
pipeline_tag: zero-shot-classification
tags:
- t5
- text-classification
- mnli
model-index:
- name: sjrhuschlee/flan-t5-base-mnli
  results:
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: MultiNLI-matched
      type: multi_nli
      config: default
      split: validation_matched
    metrics:
    - type: accuracy
      value: 87.468
      name: Accuracy
  - task:
      type: natural-language-inference
      name: Natural Language Inference
    dataset:
      name: MultiNLI-mismatched
      type: multi_nli
      config: default
      split: validation_mismatched
    metrics:
    - type: accuracy
      value: 87.276
      name: Accuracy
---

# flan-t5-base-mnli

flan-t5-base-mnli is the [flan-T5 base model](https://huggingface.co/google/flan-t5-base) fine-tuned on the [Multi-Genre Natural Language Inference (MNLI)](https://huggingface.co/datasets/multi_nli) corpus. 

## Overview

- **License:** MIT 
- **Language model:** flan-t5-base  
- **Language:** English  
- **Downstream-task:** Zero-shot Classification, Text Classification  
- **Training data:** MNLI  
- **Eval data:** MNLI (Matched and Mismatched)  
- **Infrastructure**: 1x NVIDIA 3070  

## Model Usage

Use the code below to get started with the model. The model can be loaded with the zero-shot-classification pipeline like so:

```python
from transformers import pipeline
classifier = pipeline(
  'zero-shot-classification',
  model='sjrhuschlee/flan-t5-base-mnli',
  trust_remote_code=True,
)
```

You can then use this pipeline to classify sequences into any of the class names you specify. For example:

```python
sequence_to_classify = "one day I will see the world"
candidate_labels = ['travel', 'cooking', 'dancing']
classifier(sequence_to_classify, candidate_labels)
# {'sequence': 'one day I will see the world',
#  'labels': ['travel', 'cooking', 'dancing'],
#  'scores': [0.7944864630699158, 0.10624771565198898, 0.09926578402519226]}
```

## Metrics
```bash
# MNLI
{
    "eval_accuracy": 0.8746816097809476,
    "eval_accuracy_mm": 0.8727624084621644,
    "eval_loss": 0.4271220564842224,
    "eval_loss_mm": 0.4265698492527008,
    "eval_samples": 9815,
    "eval_samples_mm": 9832,
}
```

## Uses

#### Direct Use

This fine-tuned model can be used for zero-shot classification tasks, including zero-shot sentence-pair classification, and zero-shot sequence classification.

#### Misuse and Out-of-scope Use

The model should not be used to intentionally create hostile or alienating environments for people. In addition, the model was not trained to be factual or true representations of people or events, and therefore using the model to generate such content is out-of-scope for the abilities of this model.

## Risks, Limitations and Biases

**CONTENT WARNING: Readers should be aware this section contains content that is disturbing, offensive, and can propagate historical and current stereotypes.**

Significant research has explored bias and fairness issues with language models (see, e.g., [Sheng et al. (2021)](https://aclanthology.org/2021.acl-long.330.pdf) and [Bender et al. (2021)](https://dl.acm.org/doi/pdf/10.1145/3442188.3445922)). 

Predictions generated by the model can include disturbing and harmful stereotypes across protected classes; identity characteristics; and sensitive, social, and occupational groups. For example:

```python
sequence_to_classify = "The CEO had a strong handshake."
candidate_labels = ['male', 'female']
hypothesis_template = "This text speaks about a {} profession."
classifier(sequence_to_classify, candidate_labels, hypothesis_template=hypothesis_template)
```

Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.