Repository: tatsu-lab/alpaca_eval
Issue: Trouble with custom model hosted on OpenAI compatible endpoint
Root cause: Documentation Deficiency

I'm having a really hard time using alpaca_eval for a seemingly trivial and widespread use case. This benchmark is a frequently referenced metric, and many models have an alpaca_eval score. I have a custom model I'm working on, it's currently hosted on an openai-compatible endpoint. e.g. `mymodelnfererence_provider.com/openai/v1` and I just want to evaluate that model with all defaults. the default evaluator/annotator etc...

I can't seem to find an example of this use case. There are many examples in model_configs. Though, its not clear how the OpenAI base URL is configured differently for the case of the openAI compatible model endpoint, vs. the annotation process where we alwayts call to openai proper. 

```
atomic-0:
  prompt_template: "/Users/tastycode/workspace/nola-ai/evalite/model_configs/atomic-0/prompt.txt"
  fn_completions: "openai_completions"
  completions_kwargs:
    openai_api_base:
      "https://api.runpod.ai/v1/vllm-tmdlg8/openai/v1"
    model_name: SARG-ai/atomic-0
    temperature: 0.7
    top_p:
      1.0
  decoder_kwargs:
    model_name: "gpt-4"
    num_procs: 1
    batch_size: 1
  pretty_name: "Atomic 0"
  link: "https://huggingface.co/SARG-ai/atomic-0"
  ```
  
  unfortnuately, i still ended up having to monkey patch `decorders/openai` so that my custom model name would not be sent to actual openai. 
  
 Once I run the eval with ` OPENAI_CLIENT_CONFIG_PATH=/workspace/nolai/evalite/client_configs/openai_configs.yaml alpaca_eval evaluate_from_model \
  --model_configs '/workspace/nolai/evalite/model_configs/atomic-0'`
  
I'm happy to offer making the following changes:
- Outlining a clear example of evaulate_from_model where the model is accessible via OpenAI compatible endpoint
- Explaining what an annotator is, and how it is different from an evaluator including examples demonstrating why one might choose to define an annotator, when it's best to use a particular combination of evaluator/annotator
- Explain the difference between a decoder and an annotator
- Explain how the decoder, annotator and evaluator and "client" work together to produce the alpaca eval score



If you could share any guidance on the above , it would be quite helpful. The current command fails with 

```
WARNING:root:Unknown error.
 It's likely a rate limit so we are retrying...
INFO:root:Sleeping 2 before retrying to call openai API...
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 404 Not Found"
WARNING:root:OpenAIError: id(self):6076324944 {'api_key': 'sk-my-openai-secret-key', 'project': None, '_version': '1.30.3', '_base_url': URL('https://api.openai.com/v1/'), 'max_retries': 2, 'timeout': Timeout(connect=5.0, read=600.0, write=600.0, pool=600.0), '_limits': Limits(max_connections=1000, max_keepalive_connections=100, keepalive_expiry=5.0), '_proxies': None, '_transport': None, '_custom_headers': {}, '_custom_query': {}, '_strict_response_validation': False, '_idempotency_header': None, '_client': <openai._base_client.SyncHttpxClientWrapper object at 0x17e1c1dd0>, '_default_stream_cls': <class 'openai.Stream'>, 'completions': <openai.resources.completions.Completions object at 0x17fcfa710>, 'chat': <openai.resources.chat.chat.Chat object at 0x2849e8410>, 'embeddings': <openai.resources.embeddings.Embeddings object at 0x2849e85d0>, 'files': <openai.resources.files.Files object at 0x17ea7b050>, 'images': <openai.resources.images.Images object at 0x2849e88d0>, 'audio': <openai.resources.audio.audio.Audio object at 0x2849e8a90>, 'moderations': <openai.resources.moderations.Moderations object at 0x2849e8c50>, 'models': <openai.resources.models.Models object at 0x2849e8e10>, 'fine_tuning': <openai.resources.fine_tuning.fine_tuning.FineTuning object at 0x2849e8fd0>, 'beta': <openai.resources.beta.beta.Beta object at 0x2849e9190>, 'batches': <openai.resources.batches.Batches object at 0x2849e9350>, 'with_raw_response': <openai._client.OpenAIWithRawResponse object at 0x2849e9510>, 'with_streaming_response': <openai._client.OpenAIWithStreamedResponse object at 0x2849ea7d0>} Error code: 404 - {'error': {'message': 'This is not a chat model and thus not supported in the v1/chat/completions endpoint. Did you mean to use v1/completions?', 'type': 'invalid_request_error', 'param': 'model', 'code': None}}.
```

It seems that what you want from AlpacaEval is only the annotations. AlpacaEval also provides the code to generate outputs for your convenience, but if you prefer generating the outputs on your side you should go ahead: you definitely do not need alpaca_eval for that! Given the completions/outputs simply use: `alpaca_eval --model_outputs 'example/outputs.json'`. We provided some code for completions (`alpaca_eval evaluate_from_model  --model_configs 'example'`) to make it easier to use, but you definitely do not need that if you don't find it natural.

To answer your questions:
- see this config for an example of a model that uses openai_client with a different end point: https://github.com/tatsu-lab/alpaca_eval/blob/main/src/alpaca_eval/models_configs/Mistral-7B-Instruct-v0.2/configs.yaml. Why are you setting `decoding_kwargs` and where did you see that? that's not a thing...  Note that one slight challenge is that you will need a different API key for generating your outputs (calling your server) and for calling OpenAI for annotations. Given that use OpenAI client in both cases, that means you need to set OpenAI API key twice to something different. You can do that by setting your [client_configs](https://github.com/tatsu-lab/alpaca_eval/tree/main/client_configs) like that:
```
default:
    - api_key: "<your OpenAI API key here>"

SARG-ai/atomic-0:
    - api_key: "<some_local_api_key>" # use your local key for your model
 ```
- Annotators are the actual object that perform the annotations (i.e the LLM as a judge object). At a high level the input is a dataframe containing the input and output pairs, and the output is the score as evaluated by the model. For historical reasons, decoders = completion functions (e.g. openai's, anthropic's, vllm's), those are helper functions for uniformizing the API to all the different ways of generating text from a model. They are text in text out, and they not specialized at all for AlpacaEval. We use them both inside the annotator (for getting the output given the desired prompt) and for generating outputs.