Repository: EleutherAI/lm-evaluation-harness
Issue: I want to add a custome model named  mnn in  lm-eval.models,How can I compute the output of the loglikelihood?
Root cause: Documentation Deficiency

I want to add a new model named MNN and evaluaate with the task 'piqa'.
MNN model's forward will return the token_id,logits.
-  token_id is a int 
- logits is a numpy.ndarray ([1,151936]) ,151936 is the vocab size.
just like this:
```
 def forward(self, input_ids, attention_mask, position_ids, past_key_values):
        hidden_states = self.embed(input_ids)
        presents = []
        logits = [6,6,6]
        for i in range(self.block_nums):
            hidden_states, kv = self.blocks[i]([hidden_states, attention_mask, position_ids, past_key_values[i]])
            presents.append(kv)
        #我拿ort推理试一下：
        # onnx_model = "./lm.onnx"
        hidden_states = hidden_states.read() #转化成numpy
        hidden_states = hidden_states.reshape(1,2048) #转化成1*2048
        inputs = {
            'hidden_states' : hidden_states,
        }
        token_id,logits = self.ort_session.run(None, inputs)
        token_id = token_id.item()
        # print(logits)
        self.seq_len += 1
        self.token_len += 1
        return token_id, presents,logits

```
Then I use the forward function to build a lm_response for the evaluation:
```
def lm_response(self,query,tokenizer):
        self.tokenizer = tokenizer
        self.max_length = 50
        prompt = query
        input_ids = self.str_to_ids(prompt)
        self.seq_len = input_ids.size
        self.context_len = self.seq_len - 2
        self.token_len = 0
        past_key_values = [F.placeholder(self.past_kv_shape, dtype=np.float32) for i in range(self.block_nums)]
        token_id = input_ids
        token_id_res = []
        logits_res = []
        while self.token_len < self.max_length:
            attention_mask = self.get_attention_mask()
            position_ids = self.get_position_ids()
            token_id, past_key_values,logits = self.forward(token_id, attention_mask, position_ids, past_key_values)
            # print(logits,"\n")
            if token_id == self.stop_id():
                break
            # word = self.id_to_str(token_id)
            # res += word
            token_id_res.append(token_id)
            logits_res.append(logits)
        return token_id_res,logits_res
```



I'm writing a new class named MNN in lm_eval.model just like this:
```
import logging
import time

import requests
from requests.exceptions import RequestException
from tqdm import tqdm

from lm_eval.api.model import LM
from lm_eval.api.registry import register_model
from typing import List, Optional, Tuple, Type, TypeVar

logger = logging.getLogger(__name__)

#这里就是计算的结果，把这里看明白
def get_result(logprobs, tokens):
    #假设就是贪心
    is_greedy = True
    # offsets = logprobs["text_offset"]
    tokens = tokens
    continuation_logprobs = 0
    tokens_logprobs = logprobs
    for k in range(0,len(tokens)):
        continuation_logprobs += tokens_logprobs[k][0][tokens[k]]
    # for i in range(idx, len(tokens)):
    #     token = tokens[i]
    #     top_tokens = logprobs["top_logprobs"][i]
    #     top_token = max(top_tokens.keys(), key=lambda x: top_tokens[x])
    #     if top_token != token:
    #         is_greedy = False
    #         break
    return continuation_logprobs, is_greedy


@register_model("mnn")
class MNN(LM):
    def __init__(self, mnn_model=None, tokenizer = None, **kwargs):
        super().__init__()
        self.mnn_model = mnn_model
        self.tokenizer = tokenizer

        self.logprobs = 10
        # self.temperature = 0.0
        # self.max_length = max_length

    #构建
    def mnn_completion(
        self, context, continuation=None, stop=None, **kwargs
    ):
        """
        - token_id_res (list): 生成文本的token ID序列。
        - logits_res (list): 对应于生成的token ID的logits序列
        """
        try:
            prompt = context
            if continuation:
                prompt += continuation
            #max_tokens设置多少？
            token_id_res,logits_res = self.mnn_model.lm_response(prompt,self.tokenizer)
            return token_id_res,logits_res
        # 检查响应状态，异常则抛出
        except RequestException as e:
            logger.error(f"RequestException: {e}")
    
    def loglikelihood(self, requests, disable_tqdm: bool = False)-> List[Tuple[float, bool]]:
        """
        输入：Tuple[str,str],
            str1:input string to the LM;
            str2:a target string
        输出：Tuple[float,int]
            ll:似然比率
            is_greedy:0/1
        """
        # 如果请求列表为空，则直接返回空列表
        res = []
        # 使用tqdm为每个请求的参数对进行迭代，计算其对数似然
        for context, continuation in tqdm(
            [req.args for req in requests], disable=disable_tqdm
        ):
            
            # 调用模型完成函数，这里直接调用本地模型得到答案就行。
            tokens,logprobs = self.mnn_completion(context=context, continuation=continuation)
            #Maybe I need a 
            # 计算并获取对数似然和是否贪心的结果
            logprob, is_greedy = get_result(logprobs=logprobs, tokens = tokens)
            res.append((logprob, is_greedy))
        return res
    
    #piqa应该是属于generate_until
    def generate_until(self, requests, disable_tqdm: bool = False):
        raise NotImplementedError(
            "generate not yet supported for GGUF models"
        )
    def loglikelihood_rolling(self, requests, disable_tqdm: bool = False):
        raise NotImplementedError(
            "loglikelihood_rolling not yet supported for GGUF models"
        )

```
Then I read the url https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/model_guide.md.
It suggest me to check out lm_eval/models/huggingface.py and lm_eval.api.model.TemplateLM.

But I still don't know how to compute the output of the  loglikelihood,because the 'lm_eval.api.model.TemplateLM' doesn't show the _loglikelihood_tokens just like this.
```
 def _loglikelihood_tokens(self, requests, **kwargs):
        pass
```
I really want to know how to write the  `mnn_completion` and `get result`.Now they  are still rough draft like you see.
Please give me some advice or demo to refer to.Or give me a piece of pseudocode or a modified version of my code.

My email is 2426827419@qq.com.

Hi! See HFLM._loglikelihood_tokens for a good reference implementation. https://github.com/EleutherAI/lm-evaluation-harness/blob/main/lm_eval/models/huggingface.py#L890

You'll need to: 
- get the outputs from a forward pass, run on `context + continuation` (of shape `((len(context + continuation), vocab_size)` )
- select only the positions where the model is predicting a token in the continuation
- take `log_softmax` along the vocabulary dimension, to get each token's log probability
- get the log probability corresponding to each true continuation token id, at the proper sequence position
- sum these log probabilities for each continuation token, and return them, along with `is_greedy=1` if all of these continuation tokens were the largest logprob at that sequence position (and thus the continuation would be produced by greedy decoding), else return `is_greedy=0`. so `(loglikelihood, is_greedy)`.

Hope this is helpful! I would recommend further referencing the HFLM loglikelihood code, as you'll need to make sure to not avoid off-by-one errors in indexing.