This is quantized version of the model MBZUAI/LaMini-Flan-T5-783M.

It was quantized using CTranslate2:

ct2-transformers-converter --model MBZUAI/LaMini-Flan-T5-783M --output_dir lamini-flan-t5-783m-int8_float16 --quantization int8_float16

How to use it?

Clone the model

git lfs install
git clone git@hf.co:saikatkumardey/lamini-flan-t5-783m_int8_float16

Code example

import ctranslate2
import transformers


model_dir = "lamini-flan-t5-783m_int8_float16"
translator = ctranslate2.Translator(
    model_dir, compute_type="auto", inter_threads=4, intra_threads=4
)
tokenizer = transformers.AutoTokenizer.from_pretrained("MBZUAI/LaMini-Flan-T5-783M")


input_prompt = """
instruction: Restrict the answer based on the context only. Be verbose.
context: <P>This is an introduction to pandas categorical data type, including a short comparison with R’s factor.

Categoricals are a pandas data type corresponding to categorical variables in statistics. A categorical variable takes on a limited, and usually fixed, number of possible values (categories; levels in R). Examples are gender, social class, blood type, country affiliation, observation time or rating via Likert scales.

In contrast to statistical categorical variables, categorical data might have an order (e.g. ‘strongly agree’ vs ‘agree’ or ‘first observation’ vs. ‘second observation’), but numerical operations (additions, divisions, …) are not possible.

All values of categorical data are either in categories or np.nan. Order is defined by the order of categories, not lexical order of the values. Internally, the data structure consists of a categories array and an integer array of codes which point to the real value in the categories array.

The categorical data type is useful in the following cases:

- A string variable consisting of only a few different values. Converting such a string variable to a categorical variable will save some memory, see here.
- The lexical order of a variable is not the same as the logical order (“one”, “two”, “three”). By converting to a categorical and specifying an order on the categories, sorting and min/max will use the logical order instead of the lexical order, see here.
- As a signal to other Python libraries that this column should be treated as a categorical variable (e.g. to use suitable statistical methods or plot types).</P>

question: In what cases is the categorical data type useful?
answer:
"""

input_tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(input_prompt))

results = translator.translate_batch(
    [input_tokens],
    beam_size=1,
    sampling_temperature=0.01,
    max_decoding_length=1024,
    batch_type="tokens",
    max_batch_size=64,
)

output_tokens = results[0].hypotheses[0]
output_text = tokenizer.decode(tokenizer.convert_tokens_to_ids(output_tokens))

print(output_text)