google/gemma-2b-it · Inference API Parameters

13 days ago

I want to use gemma-2b-it model using Inference API. I want to know what parameters do we need to pass in the body of the API, and what will be the format to pass the body into the inference API which can give me good results.

Currently I am following this format in the body while using Inference API, but I am not getting good response.

{"input":"[INST]{{prompt}}\n\n{{Context}}\n\n{{Question}}\n\n{{assistant}}\n\n[/INST]}

Do I need to include parameters like temperature, top-p, max new tokens, repetition penalty ?? If yes then please correct me with what body do i need to pass.

Thanks & Regards
Shiv Kumar

datamancer88

5 days ago

Hi @Shivkumar27 , are you using the interface on the model card or HuggingFace's Inference API?

Shivkumar27

5 days ago

Hi @datamancer88 , I am using HuggingFace's Inference API

selamw

Google org 4 days ago

You can adjust inference parameters for models on Hugging Face's Inference API through the model card metadata. This allows you to adjust settings like aggregation_strategy and temperature.

Further reference here on ‘How can I control my model’s widget Inference API parameters?’

For example, you can specify these parameters for text generation:

Prompt: "Write a poem about a lonely robot."
inference:
  parameters:
    aggregation_strategy: "none"
    temperature: 0.7