Logits Extraction
#5
by
Romnags
- opened
I want to use the inference code and extract the logits as well from model.chat function. how can i do so? if not is there any alternate way of inferencing the model and extracting out the logits.
hi, you should revise the forward function to get values of self.llm(
input_ids=None,
position_ids=position_ids,
inputs_embeds=vllm_embedding,
**kwargs
)