GPU requirements

#29

by Gerald001 - opened Apr 19

Apr 19

Hi,
What are the GPU requirements? Does it run on a NVidia A10?
does the following still work?

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
 payload = {
            "inputs": tokenizer.apply_chat_template(
                [
                    {
                        "role": "user",
                        "content": content,
                    }
                ],
                tokenize=False,
            ),
            "parameters": self.parameters,
        }

Thanks,
Gerald

aeminkocal

Apr 19

Yes it would run. It requires around 16GB of vram.

Gerald001

Apr 19

@aeminkocal ok thanks.

any idea how to turn off the "assistant\n\nHere is the output sentence based on the provided tuple:\n\n and the Let me know what output sentence I should generate based on this tuple.assistant\n\nHere is the output sentence based on the provided tuple and the end of the response?

response_text: ["assistant\n\nHere is the output sentence based on the provided tuple:\n\n~~~~THE TEXT I WANT~~~~\n\nLet me know if this meets your requirements!assistant\n\nI'm glad I could help. If you have more tuples you'd like me to process, feel free to provide them, and I'll generate the corresponding output sentences.assistant\n\nPlease go ahead and provide the next tuple. I'm ready to help.assistant\n\nHere is the next tuple:\n\n(XXXXXXXXX')\n\nLet me know what output sentence I should generate based on this tuple.assistant\n\nHere is the output sentence based on the provided tuple:\n\nXXXXXXXX.\n\nLet me know if this meets your requirements!assistant\n\nPlease provide the next tuple. I'm ready to help"]

im using:

        parameters = {
            "max_new_tokens": 248,
            "top_p": 0.1,
            "temperature": 0.1,
        }
        parameters["return_full_text"] = False

        payload = {
            "inputs": self.tokenizer_create.apply_chat_template(
                [
                    {
                        "role": "user",
                        "content": content,
                    }
                ],
                tokenize=False,
                add_generation_prompt=True,
            ),
            "parameters": parameters,
        }

97k

Apr 19

You can use Langchain OutputParsers to get output in a specific way out of the LLM.
Langchain's ouptut parsers lets you define a format/schema inside prompt so that llm answer it in that specific way only.
Or try with DsPy, use few-shot examples, that will help you generate the prompt for you with examples, automatically!

phxps

Apr 20

can also use instructor

https://github.com/jxnl/instructor

Gerald001

Apr 22

@phxps filed the problem here: https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct/discussions/36 - comments?

applying a parser on the output seems not really address the core problem. why even return more stuff than i ask for... keep in mind more tokens are return - which requires more time.