msaad02/llama2_7b_brockportgpt_gptq

VERIFIED 8/23/23

BrockportGPT GPTQ LLaMA2 Finetune

This is a quantized version of the original model, found at "msaad02/llama2_7b_brockportgpt"

This model is part of my thesis to make a question-answering bot for SUNY Brockport. The data used for finetuning is a subset of the question database I generated, and is also available on my huggingface account.

For more information, see my github repo for the project.

Examples

Are SAT scores required?

SAT scores are not required for admission, but test scores are used as part of Brockport's holistic review process to make admissions decisions.

Who can I contact for help with financial aid?

You can contact the financial aid office at SUNY Brockport for assistance with financial aid. They are available to answer any questions and help you navigate the financial aid process.

Prompting

This prompting style is what was used for finetuning:

Below is an inquiry related to SUNY Brockport - from academics, admissions, and faculty support to student life. Prioritize accuracy and brevity.

### Instruction:
{question}

### Response:
{response}

Usage

Brief implementation using ctransformers. Note you will need to specially pip install ctransformers[gptq] for exllama support. Originally this library only had support for GGML models using llama.cpp as a backend. However, they updated it not long ago at all to have support for exllama, which is an incredibly fast loader for gptq models. ctransformers is the interface to use exllama, which is the current SoTA loader.

from ctransformers import AutoModelForCausalLM
import textwrap

llm = AutoModelForCausalLM.from_pretrained("msaad02/llama2_brockportgpt_gptq", model_type="gptq")

def qa(text: str, full = False):
    # textwrap.dedent gets rid of indenting at the start of each newline
    text = textwrap.dedent(f"""\
        Below is an inquiry related to SUNY Brockport - from academics, admissions, and faculty support to student life. Prioritize accuracy and brevity.

        ### Instruction:
        {text}

        ### Response:
        """)
    
    response = llm(text, max_new_tokens=256)
    response = (text + response) if full else response

    return response

qa("How do I apply?")
> You can apply to SUNY Brockport by following the application process outlined on their website.

qa("How do I apply?", full=True)
> Below is an inquiry related to SUNY Brockport - from academics, admissions, and faculty support to student life. Prioritize accuracy and brevity.
>
> ### Instruction:
> How do I apply?
>
> ### Response:
> You can apply online via the Common Application or Brockport's own Apply Now!