Repository: truera/trulens
Issue: [BUG] Use Anthropic Claude 3 Opus as Feedback Provider
Root cause: Interface Contract Mismatch

**Bug Description**
TruLens recorder works only with OpenAI() models. When I try Anthropic or TogetherAI APIs exceptions emerges.

**To Reproduce**
```py
import logging
from time import sleep

import numpy as np
from llama_index.core import (
    VectorStoreIndex,
    SimpleDirectoryReader
)

from trulens_eval import TruLlama, Feedback, Tru
from trulens_eval.app import App
from trulens_eval.feedback import Groundedness
from trulens_eval.feedback.provider.litellm import LiteLLM as fLiteLLM
from trulens_eval.feedback.provider.openai import OpenAI as fOpenAI


# load documents
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# build index
index = VectorStoreIndex.from_documents(documents=documents)

# tru = Tru('sqlite:///default.sqlite')
tru = Tru()

llm = fOpenAI()
llm = fLiteLLM(model_engine="claude-3-opus-20240229")
llm = fLiteLLM(model_engine="together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1")

query_engine = index.as_query_engine(
    similarity_top_k=10,
    node_postprocessors=[],
)
context = App.select_context(query_engine)

# grounded = Groundedness(groundedness_provider=fOpenAI())
grounded = Groundedness(groundedness_provider=llm)
f_groundedness = (
    Feedback(grounded.groundedness_measure_with_cot_reasons)
    .on(context.collect())  # collect context chunks into a list
    .on_output()
    .aggregate(grounded.grounded_statements_aggregator)
)

f_qa_relevance = Feedback(llm.relevance).on_input_output()

# Question/statement relevance between question and each context chunk.
f_qs_relevance = (
    Feedback(llm.qs_relevance)
    .on_input()
    .on(context)
    .aggregate(np.mean)
)

logging.info("Creating TruLens Recorder")
app_id = "app_id_01"

tru_rag = TruLlama(
    query_engine,
    app_id=app_id,
    feedbacks=[f_groundedness, f_qa_relevance, f_qs_relevance]
)

logging.info("Computing RAG triad metrics")
for questions in ["How do you do?"]:
    for idx, question in enumerate(questions):
        with tru_rag as recording:
            logging.info(f"Question: {question}")
            response = query_engine.query(question)
            logging.info(f"response {response}")
            msg = "\n".join([
                f"Question {idx}: %s" % question,
                "Answer: %s" % response
            ])
            logging.info(msg)
            sleep(0.1)
            # The record of the app invocation can be retrieved from the `recording`:
            # logging.info(f"recording.records %s" % recording.records)

```

**Expected behavior**
The recorder should log the questions and answers into Sqlite database with calculated metrics. It just logs questions and answers.

**Relevant Logs/Tracebacks**
2024-03-11 17:37:38,066:INFO:Computing RAG triad metrics
2024-03-11 17:37:52,821:INFO:HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
2024-03-11 17:37:53,036:INFO:Context impl SQLiteImpl.
2024-03-11 17:37:53,037:INFO:Will assume non-transactional DDL.
2024-03-11 17:37:53,169:INFO:✅ added record record_hash_beef086a4fb01e5aaa1a58c895e581eb
2024-03-11 17:37:53,175:INFO:Context impl SQLiteImpl.
2024-03-11 17:37:53,175:INFO:Will assume non-transactional DDL.
17:37:53 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:53,370:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
17:37:53 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:53,473:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -

17:37:53 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:53,496:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:53,727:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=3.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:53,751:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=3.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:53,896:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=3.
17:37:55 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:55,733:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
17:37:55 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:55,755:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
17:37:55 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:37:55,901:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:56,111:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=2.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:56,177:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=2.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-11 17:37:56,900:ERROR:litellm request failed <class 'litellm.exceptions.BadRequestError'>=AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}. Retries remaining=2.
17:38:00 - LiteLLM:INFO: 
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:38:00,117:INFO:
POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
2024-03-11 17:38:12,760:INFO:Context impl SQLiteImpl.
2024-03-11 17:38:12,761:INFO:Will assume non-transactional DDL.
2024-03-11 17:38:12,767:INFO:🛑 feedback result qs_relevance FAILED feedback_result_hash_86510e3dc727b3009952db519baa9741



**Environment:**
Hardware Overview:
  Model Name:	MacBook Pro
  Model Identifier:	Mac14,9
  Model Number:	MPHE3LL/A
  Chip:	Apple M2 Pro
  Total Number of Cores:	10 (6 performance and 4 efficiency)
  Memory:	16 GB
  System Firmware Version:	10151.81.1
  OS Loader Version:	10151.81.1

Python 3.11.6 (v3.11.6:8b6ee5ba3b, Oct  2 2023, 11:18:21) [Clang 13.0.0 (clang-1300.0.29.30)] on darwin

faiss_cpu==1.8.0
FlagEmbedding==1.2.5
langchain_community==0.0.24
langchain_core==0.1.28
langchain-together==0.0.2.post1
langchain-anthropic==0.1.4
litellm==1.23.16
trulens_eval==0.25.0
llama_index==0.10.16
llama-index-core=0.10.16
llama-index-llms-litellm==0.1.3
llama-index-llms-anthropic==0.1.5
llama-index-embeddings-huggingface==0.1.4
llama-index-vector-stores-faiss==0.1.2
llama-index-llms-together==0.1.3
llama-index-embeddings-together==0.1.2
llama-index-embeddings-voyageai==0.1.3
llama-index-postprocessor-cohere-rerank==0.1.2
sentence_transformers==2.3.1
nest_asyncio==1.6.0
numpy==1.26.4
openai==1.13.3
torch==2.2.1
torchvision==0.17.1
torchaudio==2.2.1
pypdf==4.0.2
python-dotenv==1.0.1
tiktoken==0.5.2
watchdog==4.0.0
voyageai==0.1.7



<!-- Greeting -->
Hey @aabor, nice to see you back here! Hope you're doing great.

<!-- Answer -->
Based on the error message you're receiving, it seems like the model "claude-3-opus-20240229" is not supported by the API you're using. The error message suggests using the Messages API instead. 

The `litellm` package is trying to use the model "claude-3-opus-20240229" with the API endpoint `https://api.anthropic.com/v1/complete`, but it appears that this model is not supported on this endpoint. 

You might want to check if the model "claude-3-opus-20240229" is indeed supported by the Anthropic API and if it's compatible with the endpoint you're using. If the model is not supported, you might need to use a different model. If the model is supported but not on the endpoint you're using, you might need to switch to the suggested Messages API.

Regarding the TruLens recorder issue, it seems like the recorder is designed to work with OpenAI models. If you're trying to use it with models from Anthropic or TogetherAI APIs, you might encounter exceptions. 

You might want to check if there's a way to make the TruLens recorder compatible with models from other providers. This could involve modifying the recorder's code or using some sort of adapter or wrapper to make the models from other providers compatible with the recorder.

If you're still encountering issues, could you provide more details about how you're using the `trulens_eval` and `litellm` packages, and how the model "claude-3-opus-20240229" is specified in your code? This could help in identifying the cause of the issue and suggesting a more specific solution.

## Sources
- [trulens_eval/trulens_eval/app.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/app.py)

<!-- Dosu Comment Footer -->
<details open>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>

Hi; is the issue coming from litellm? Can you check whether litellm without the use of trulens operates as expected on the models you want to run and whether litellm's cost tracking work on them?

Litellm works separately. For example, the code below gave expected output:

```py

from litellm import completion

messages = [{"role": "user", "content": "Write me a poem about the blue sky"}]

response = completion(model="together_ai/meta-llama/Llama-2-70b-chat-hf", messages=messages)

print(response)

# ModelResponse(id='863850c3ba2befb4-PDX', choices=[Choices(finish_reason='eos', index=0, message=Message(content="  Sure! Here's a short poem about the blue sky:\n\nThe blue sky stretches far and wide,\nA canvas painted by the sun's bright side.\nDotted with clouds, so fluffy and white,\nIt's a sight that's simply out of sight.\n\nThe blue sky brings us joy and cheer,\nA symbol of hope, and a promise clear.\nIt's a reminder of the beauty above,\nA reflection of the love that we have.\n\nSo let's bask in the blue sky's embrace,\nAnd let our spirits soar with grace.\nFor in its depths, we'll find our peace,\nA sense of calm, a world to cease.\n\nThe blue sky's magic is pure and true,\nA treasure that's waiting for me and you.\nSo let's gaze up, and let our hearts sing,\nIn the blue sky's splendor, our spirits will take wing.", role='assistant'))], created=1710294024, model='meta-llama/Llama-2-70b-chat-hf', object='chat.completion', system_fingerprint=None, usage=Usage(completion_tokens=220, prompt_tokens=17, total_tokens=237))

```

@aabor - it looks like the model you’re trying to use with LiteLLM/TruLens is Claude-3-opus. Can you confirm that this model works with LiteLLM? Your test above shows just Llama-2

```py
from litellm import completion

messages = [{"role": "user", "content": "Hey! how's it going?"}]
response = completion(model="claude-3-opus-20240229", messages=messages)
print(response)
```

07:45:17 - LiteLLM:INFO: 

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -H 
-d '{'model': 'claude-3-opus-20240229', 'prompt': "\n\nHuman: Hey! how's it going?\n\nAssistant: ", 'max_tokens_to_sample': 256}'


2024-03-13 07:45:17,915:INFO:

POST Request Sent from LiteLLM:
curl -X POST \
https://api.anthropic.com/v1/complete \
-H 'accept: application/json' -H 'anthropic-version: 2023-06-01' -H 'content-type: application/json' -
-d '{'model': 'claude-3-opus-20240229', 'prompt': "\n\nHuman: Hey! how's it going?\n\nAssistant: ", 'max_tokens_to_sample': 256}'



Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/main.py", line 1020, in completion
    response = anthropic.completion(
               ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/llms/anthropic.py", line 170, in completion
    raise AnthropicError(
litellm.llms.anthropic.AnthropicError: {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/utils.py", line 2481, in wrapper
    raise e
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/utils.py", line 2384, in wrapper
    result = original_function(*args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/main.py", line 1897, in completion
    raise exception_type(
          ^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/utils.py", line 7520, in exception_type
    raise e
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/litellm/utils.py", line 6478, in exception_type
    raise BadRequestError(
litellm.exceptions.BadRequestError: AnthropicException - {"type":"error","error":{"type":"invalid_request_error","message":"\"claude-3-opus-20240229\" is not supported on this API. Please use the Messages API instead."}}

This model works when I use other packages to access, for example, llama_index 
```py
from llama_index.llms.anthropic import Anthropic as LlamaIndexAnthropic
```

Can you try upgrading your LiteLLM version? See the related LiteLLM issue: https://github.com/BerriAI/litellm/issues/2314

I upgraded to the most recent LiteLLM version, which is 1.31.6.
It works well with direct completion function call, also it gives less errors.
But still when I use TruLens to estimate performance the following error appears:

2024-03-13 08:14:01,351:INFO:HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
2024-03-13 08:14:01,566:INFO:Context impl SQLiteImpl.
2024-03-13 08:14:01,566:INFO:Will assume non-transactional DDL.
2024-03-13 08:14:01,706:INFO:✅ added record record_hash_d250bdf66726f7230a6ed74cff868ffc
2024-03-13 08:14:01,711:INFO:Context impl SQLiteImpl.
2024-03-13 08:14:01,712:INFO:Will assume non-transactional DDL.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
2024-03-13 08:14:02,052:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
2024-03-13 08:14:02,061:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
2024-03-13 08:14:02,077:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Can you add `litellm.set_verbose=True` to get more info on the error?

Ok, I set `litellm.set_verbose=True`.

All other environment was the same. 
Make notice that another models "together_ai/meta-llama/Llama-2-70b-chat-hf", "together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1" also gave mistakes when I set it as an engine in the TruLens recorder. Please, test usage of this models with TruLens as well. May be, it is wise to add some automatic tests of this functionality in your CI/CD pipeline.

Here you are the output:

2024-03-14 11:27:28,303:INFO:HTTP Request: POST https://api.anthropic.com/v1/messages "HTTP/1.1 200 OK"
2024-03-14 11:27:28,514:INFO:Context impl SQLiteImpl.
2024-03-14 11:27:28,514:INFO:Will assume non-transactional DDL.
2024-03-14 11:27:28,650:INFO:✅ added record record_hash_77e3c859b487952c27fc1654c61b3930
2024-03-14 11:27:28,654:INFO:Context impl SQLiteImpl.
2024-03-14 11:27:28,654:INFO:Will assume non-transactional DDL.
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a INFORMATION OVERLAP classifier providing the overlap of information between a SOURCE and STATEMENT.\nFor every sentence in the statement, please answer with this template:\n\nTEMPLATE: \nStatement Sentence: <Sentence>, \nSupporting Evidence: <Choose the exact unchanged sentences in the source that can answer the statement, if nothing matches, say NOTHING FOUND>\nScore: <Output a number between 0-10 where 0 is no information overlap and 10 is all information is overlapping>\nGive me the INFORMATION OVERLAP of this SOURCE and STATEMENT.\n\nSOURCE:])
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given RESPONSE to the given PROMPT.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long RESPONSES should score equally well as short RESPONSES.\n\n- Answers that intentionally do not answer the question, such as 'I don't know' and model refusals, should also be counted as the most RELEVANT.\n\n- RESPONSE must be relevant to the entire PROMPT to get a score of 10.\n\n- RELEVANCE score should increase as the RESPONSE provides RELEVANT context to more parts of the PROMPT.\n\n- RESPONSE that is RELEVANT to none of the PROMPT should get a score of 0.\n\n- RESPONSE that is RELEVANT to some of the PROMPT should get as score of 2, 3, or 4. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to most of the PROMPT should get a score between a 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to the entire PROMPT should get a score of 9 or 10.\n\n- RESPONSE that is RELEVANT and answers the entire PROMPT completely should get a score of 10.\n\n- RESPONSE that confidently FALSE should get a score of 0.\n\n- RESPONSE that is only seemingly RELEVANT should get a score of 0.\n\n- Never elaborate.\n\nPROMPT: "}])
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given STATEMENT to the given QUESTION.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long STATEMENTS should score equally well as short STATEMENTS.\n\n- RELEVANCE score should increase as the STATEMENT provides more RELEVANT context to the QUESTION.\n\n- RELEVANCE score should increase as the STATEMENT provides RELEVANT context to more parts of the QUESTION.\n\n- STATEMENT that is RELEVANT to some of the QUESTION should score of 2, 3 or 4. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to most of the QUESTION should get a score of 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to the entire QUESTION should get a score of 9 or 10. Higher score indicates more RELEVANCE.\n\n- STATEMENT must be relevant and helpful for answering the entire QUESTION to get a score of 10.\n\n- Answers that intentionally do not answer the question, such as 'I don't know', should also be counted as the most relevant.\n\n- Never elaborate.\n\nQUESTION: }])
Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
self.failure_callback: []
self.failure_callback: []
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
2024-03-14 11:27:28,987:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
2024-03-14 11:27:28,989:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
2024-03-14 11:27:29,000:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=3.
self.failure_callback: []
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given RESPONSE to the given PROMPT.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long RESPONSES should score equally well as short RESPONSES.\n\n- Answers that intentionally do not answer the question, such as 'I don't know' and model refusals, should also be counted as the most RELEVANT.\n\n- RESPONSE must be relevant to the entire PROMPT to get a score of 10.\n\n- RELEVANCE score should increase as the RESPONSE provides RELEVANT context to more parts of the PROMPT.\n\n- RESPONSE that is RELEVANT to none of the PROMPT should get a score of 0.\n\n- RESPONSE that is RELEVANT to some of the PROMPT should get as score of 2, 3, or 4. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to most of the PROMPT should get a score between a 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to the entire PROMPT should get a score of 9 or 10.\n\n- RESPONSE that is RELEVANT and answers the entire PROMPT completely should get a score of 10.\n\n- RESPONSE that confidently FALSE should get a score of 0.\n\n- RESPONSE that is only seemingly RELEVANT should get a score of 0.\n\n- Never elaborate.\n\nPROMPT: E: "}])
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a INFORMATION OVERLAP classifier providing the overlap of information between a SOURCE and STATEMENT.\nFor every sentence in the statement, please answer with this template:\n\nTEMPLATE: \nStatement Sentence: <Sentence>, \nSupporting Evidence: <Choose the exact unchanged sentences in the source that can answer the statement, if nothing matches, say NOTHING FOUND>\nScore: <Output a number between 0-10 where 0 is no information overlap and 10 is all information is overlapping>\nGive me the INFORMATION OVERLAP of this SOURCE and STATEMENT.\n\nSOURCE: [\n"}])
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/newFinal returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
Request to litellm:
Logging Details LiteLLM-Failure Calllitellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given STATEMENT to the given QUESTION.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long STATEMENTS should score equally well as short STATEMENTS.\n\n- RELEVANCE score should increase as the STATEMENT provides more RELEVANT context to the QUESTION.\n\n- RELEVANCE score should increase as the STATEMENT provides RELEVANT context to more parts of the QUESTION.\n\n- STATEMENT that is RELEVANT to some of the QUESTION should score of 2, 3 or 4. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to most of the QUESTION should get a score of 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to the entire QUESTION should get a score of 9 or 10. Higher score indicates more RELEVANCE.\n\n- STATEMENT must be relevant and helpful for answering the entire QUESTION to get a score of 10.\n\n- Answers that intentionally do not answer the question, such as 'I don't know', should also be counted as the most relevant.\n\n- Never elaborate.\n\nQUESTION: NCE: "}])
Logging Details LiteLLM-Failure Call
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
self.failure_callback: []
self.failure_callback: []
2024-03-14 11:27:31,043:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=2.
2024-03-14 11:27:31,048:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=2.
2024-03-14 11:27:31,051:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=2.
self.failure_callback: []
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a INFORMATION OVERLAP classifier providing the overlap of information between a SOURCE and STATEMENT.\nFor every sentence in the statement, please answer with this template:\n\nTEMPLATE: \nStatement Sentence: <Sentence>, \nSupporting Evidence: <Choose the exact unchanged sentences in the source that can answer the statement, if nothing matches, say NOTHING FOUND>\nScore: <Output a number between 0-10 where 0 is no information overlap and 10 is all information is overlapping>\nGive me the INFORMATION OVERLAP of this SOURCE and STATEMENT.\n\nSOURCE: ["}])
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given RESPONSE to the given PROMPT.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long RESPONSES should score equally well as short RESPONSES.\n\n- Answers that intentionally do not answer the question, such as 'I don't know' and model refusals, should also be counted as the most RELEVANT.\n\n- RESPONSE must be relevant to the entire PROMPT to get a score of 10.\n\n- RELEVANCE score should increase as the RESPONSE provides RELEVANT context to more parts of the PROMPT.\n\n- RESPONSE that is RELEVANT to none of the PROMPT should get a score of 0.\n\n- RESPONSE that is RELEVANT to some of the PROMPT should get as score of 2, 3, or 4. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to most of the PROMPT should get a score between a 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to the entire PROMPT should get a score of 9 or 10.\n\n- RESPONSE that is RELEVANT and answers the entire PROMPT completely should get a score of 10.\n\n- RESPONSE that confidently FALSE should get a score of 0.\n\n- RESPONSE that is only seemingly RELEVANT should get a score of 0.\n\n- Never elaborate.\n\nPROMPT:VANCE: "}])
Request to litellm:self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given STATEMENT to the given QUESTION.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long STATEMENTS should score equally well as short STATEMENTS.\n\n- RELEVANCE score should increase as the STATEMENT provides more RELEVANT context to the QUESTION.\n\n- RELEVANCE score should increase as the STATEMENT provides RELEVANT context to more parts of the QUESTION.\n\n- STATEMENT that is RELEVANT to some of the QUESTION should score of 2, 3 or 4. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to most of the QUESTION should get a score of 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to the entire QUESTION should get a score of 9 or 10. Higher score indicates more RELEVANCE.\n\n- STATEMENT must be relevant and helpful for answering the entire QUESTION to get a score of 10.\n\n- Answers that intentionally do not answer the question, such as 'I don't know', should also be counted as the most relevant.\n\n- Never elaborate.\n\nQUESTION: \n\nRELEVANCE: "}])
Logging Details LiteLLM-Failure Call
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
self.failure_callback: []
self.failure_callback: []
self.failure_callback: []
2024-03-14 11:27:35,107:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=1.
2024-03-14 11:27:35,110:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=1.
2024-03-14 11:27:35,111:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=1.
Request to litellm:
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given RESPONSE to the given PROMPT.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long RESPONSES should score equally well as short RESPONSES.\n\n- Answers that intentionally do not answer the question, such as 'I don't know' and model refusals, should also be counted as the most RELEVANT.\n\n- RESPONSE must be relevant to the entire PROMPT to get a score of 10.\n\n- RELEVANCE score should increase as the RESPONSE provides RELEVANT context to more parts of the PROMPT.\n\n- RESPONSE that is RELEVANT to none of the PROMPT should get a score of 0.\n\n- RESPONSE that is RELEVANT to some of the PROMPT should get as score of 2, 3, or 4. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to most of the PROMPT should get a score between a 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- RESPONSE that is RELEVANT to the entire PROMPT should get a score of 9 or 10.\n\n- RESPONSE that is RELEVANT and answers the entire PROMPT completely should get a score of 10.\n\n- RESPONSE that confidently FALSE should get a score of 0.\n\n- RESPONSE that is only seemingly RELEVANT should get a score of 0.\n\n- Never elaborate.\n\nPROMPT: та Российской Федерации.\n\nRELEVANCE: "}])
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Request to litellm:Final returned optional params: {}
self.optional_params: {}
Request to litellm:
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a RELEVANCE grader; providing the relevance of the given STATEMENT to the given QUESTION.\nRespond only as a number from 0 to 10 where 0 is the least relevant and 10 is the most relevant. \n\nA few additional scoring guidelines:\n\n- Long STATEMENTS should score equally well as short STATEMENTS.\n\n- RELEVANCE score should increase as the STATEMENT provides more RELEVANT context to the QUESTION.\n\n- RELEVANCE score should increase as the STATEMENT provides RELEVANT context to more parts of the QUESTION.\n\n- STATEMENT that is RELEVANT to some of the QUESTION should score of 2, 3 or 4. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to most of the QUESTION should get a score of 5, 6, 7 or 8. Higher score indicates more RELEVANCE.\n\n- STATEMENT that is RELEVANT to the entire QUESTION should get a score of 9 or 10. Higher score indicates more RELEVANCE.\n\n- STATEMENT must be relevant and helpful for answering the entire QUESTION to get a score of 10.\n\n- Answers that intentionally do not answer the question, such as 'I don't know', should also be counted as the most relevant.\n\n- Never elaborate.\n\nQUESTION: .\n\nRELEVANCE: "}])
Logging Details: logger_fn - None | callable(logger_fn) - False
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
litellm.completion(model='claude-3-opus-20240229', messages=[{'role': 'system', 'content': "You are a INFORMATION OVERLAP classifier providing the overlap of information between a SOURCE and STATEMENT.\nFor every sentence in the statement, please answer with this template:\n\nTEMPLATE: \nStatement Sentence: <Sentence>, \nSupporting Evidence: <Choose the exact unchanged sentences in the source that can answer the statement, if nothing matches, say NOTHING FOUND>\nScore: <Output a number between 0-10 where 0 is no information overlap and 10 is all information is overlapping>\nGive me the INFORMATION OVERLAP of this SOURCE and STATEMENT.\n\nSOURCE: ['.\n"}])
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.
Logging Details: logger_fn - None | callable(logger_fn) - False
self.optional_params: {}
kwargs[caching]: False; litellm.cache: None
Final returned optional params: {}
self.optional_params: {}
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.Logging Details LiteLLM-Failure Call
Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
Logging Details LiteLLM-Failure Call
self.failure_callback: []
self.failure_callback: []
self.failure_callback: []
2024-03-14 11:27:43,174:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=0.
2024-03-14 11:27:43,178:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=0.
2024-03-14 11:27:43,178:ERROR:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=list index out of range. Retries remaining=0.
2024-03-14 11:27:43,179:WARNING:Feedback Function exception caught: Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 627, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 496, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 477, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 574, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 347, in relevance
    return self.generate_score(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 156, in generate_score
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 308, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint litellm request failed 4 time(s): 
	list index out of range
	list index out of range
	list index out of range
	list index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 633, in run
    raise RuntimeError(
RuntimeError: Evaluation of relevance failed on inputs: 
{'prompt': 'Сколько раз одно и то же лицо может замещать должность '
           'руководителя одной и той же государственной или.
2024-03-14 11:27:43,181:WARNING:Feedback Function exception caught: Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 627, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 496, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 477, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 574, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/groundedness.py", line 116, in groundedness_measure_with_cot_reasons
    reason = self.groundedness_provider._groundedness_doc_in_out(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 125, in _groundedness_doc_in_out
    return self.endpoint.run_in_pace(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 308, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint litellm request failed 4 time(s): 
	list index out of range
	list index out of range
	list index out of range
	list index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 633, in run
    raise RuntimeError(
RuntimeError: Evaluation of groundedness_measure_with_cot_reasons failed on inputs: 
{'source': ['Одно и то же лицо не может замещать должность руководителя одной '
            'и той же\n'
            'государств.
2024-03-14 11:27:43,182:WARNING:Feedback Function exception caught: Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 627, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 496, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 477, in track_all_costs
    return Endpoint._track_costs(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 574, in _track_costs
    result: T = __func(*args, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 271, in qs_relevance
    return self.generate_score(
           ^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 156, in generate_score
    response = self.endpoint.run_in_pace(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 308, in run_in_pace
    raise RuntimeError(
RuntimeError: Endpoint litellm request failed 4 time(s): 
	list index out of range
	list index out of range
	list index out of range
	list index out of range
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/Users/aabor/projects/rag/venv/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 633, in run
    raise RuntimeError(
RuntimeError: Evaluation of qs_relevance failed on inputs: 
{'question': 'Сколько раз одно и то же лицо может замещать должность '
             'руководителя одной и той же государственной.
2024-03-14 11:27:43,183:INFO:Context impl SQLiteImpl.
2024-03-14 11:27:43,184:INFO:Will assume non-transactional DDL.
2024-03-14 11:27:43,186:INFO:Context impl SQLiteImpl.
2024-03-14 11:27:43,187:INFO:Context impl SQLiteImpl.
2024-03-14 11:27:43,187:INFO:Will assume non-transactional DDL.
2024-03-14 11:27:43,187:INFO:Will assume non-transactional DDL.
2024-03-14 11:27:43,197:INFO:🛑 feedback result relevance FAILED feedback_result_hash_912d1c23ae37b4b0d3e1ef1fed150351
2024-03-14 11:27:43,197:INFO:🛑 feedback result qs_relevance FAILED feedback_result_hash_5284eda989e2faaeb13bf0f87cc8f235
2024-03-14 11:27:43,197:INFO:🛑 feedback result groundedness_measure_with_cot_reasons FAILED feedback_result_hash_7ae47ad25971

Looks like these models don't like having messages with only role of "system" and will complain if no "user" role is given. 

That is, the problem is with the prompt we are giving to evaluate some of these eval metrics. Will look more into it.

Does not work:
```python

input_dict = {'messages': [{'content': 'You are a RELEVANCE grader; providing the relevance '
                          'of the given CONTEXT to the given QUESTION.\n'
                          'Respond only as a number from 0 to 10 where 0 is '
                          'the least relevant and 10 is the most relevant. \n'
                          '\n'
                          'A few additional scoring guidelines:\n'
                          '\n'
                          '- Long CONTEXTS should score equally well as short '
                          'CONTEXTS.\n'
                          '\n'
                          '- RELEVANCE score should increase as the CONTEXTS '
                          'provides more RELEVANT context to the QUESTION.\n'
                          '\n'
                          '- RELEVANCE score should increase as the CONTEXTS '
                          'provides RELEVANT context to more parts of the '
                          'QUESTION.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to some of the QUESTION '
                          'should score of 2, 3 or 4. Higher score indicates '
                          'more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to most of the QUESTION '
                          'should get a score of 5, 6, 7 or 8. Higher score '
                          'indicates more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to the entire QUESTION '
                          'should get a score of 9 or 10. Higher score '
                          'indicates more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT must be relevant and helpful for '
                          'answering the entire QUESTION to get a score of '
                          '10.\n'
                          '\n'
                          '- Never elaborate.\n'
                          '\n'
                          'QUESTION: Where is Poland?\n'
                          '\n'
                          'CONTEXT: Germany is a country in Europe.\n'
                          '\n'
                          'RELEVANCE: ',
               'role': 'system'}],
 'model': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1'}

temp = completion(**input_dict)
```

Works (only change is role)
```python

input_dict = {'messages': [{'content': 'You are a RELEVANCE grader; providing the relevance '
                          'of the given CONTEXT to the given QUESTION.\n'
                          'Respond only as a number from 0 to 10 where 0 is '
                          'the least relevant and 10 is the most relevant. \n'
                          '\n'
                          'A few additional scoring guidelines:\n'
                          '\n'
                          '- Long CONTEXTS should score equally well as short '
                          'CONTEXTS.\n'
                          '\n'
                          '- RELEVANCE score should increase as the CONTEXTS '
                          'provides more RELEVANT context to the QUESTION.\n'
                          '\n'
                          '- RELEVANCE score should increase as the CONTEXTS '
                          'provides RELEVANT context to more parts of the '
                          'QUESTION.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to some of the QUESTION '
                          'should score of 2, 3 or 4. Higher score indicates '
                          'more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to most of the QUESTION '
                          'should get a score of 5, 6, 7 or 8. Higher score '
                          'indicates more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT that is RELEVANT to the entire QUESTION '
                          'should get a score of 9 or 10. Higher score '
                          'indicates more RELEVANCE.\n'
                          '\n'
                          '- CONTEXT must be relevant and helpful for '
                          'answering the entire QUESTION to get a score of '
                          '10.\n'
                          '\n'
                          '- Never elaborate.\n'
                          '\n'
                          'QUESTION: Where is Poland?\n'
                          '\n'
                          'CONTEXT: Germany is a country in Europe.\n'
                          '\n'
                          'RELEVANCE: ',
               'role': 'user'}],
 'model': 'together_ai/mistralai/Mixtral-8x7B-Instruct-v0.1'}

temp = completion(**input_dict)
```

Hey @aabor - this should be fixed by #1018 and merged/released today. Please take a look!

This change also includes new examples to show use of TogetherAI (in the LiteLLM quickstart) and Claude-3 in its own example notebook.

Hey @aabor - this is released in 0.27.0!

Please upgrade your trulens-eval version `pip install -U trulens-eval` and checkout the new examples:
- [LiteLLM Quickstart now shows TogetherAI usage](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/litellm_quickstart.ipynb)
- [Claude-3 Quickstart](https://github.com/truera/trulens/blob/main/trulens_eval/examples/expositional/models/claude3_quickstart.ipynb)