Repository: truera/trulens
Issue: Cannot work out how to use local LLM with api_base for Instrument Chains
Root cause: Documentation Deficiency

Specifically with the quickstart, everything seems to be OpenAI.  

https://github.com/truera/trulens/blob/main/trulens_eval/examples/quickstart/llama_index_quickstart.ipynb

Now, I don't have any problem accessing the local LLM. I can happily query against it using llama_index, and

```
response = query_engine.query("What did the author do growing up?")
print(response)
```

Gives me

```
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
	- Avoid using `tokenizers` before the fork if possible
	- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
The author grew up working on writing short stories and programming on an IBM 1401 computer in school. Additionally, they had access to surplus Xerox Dandelions which allowed them to experiment with computing technology outside of the classroom setting. The author's interest in art began at a young age, but they never imagined it was possible for them to create paintings until later in life when they started exploring their creative side and discovered that art could be learned just like any other skill. After attending Harvard, the author pursued their passion for art and began taking classes to develop their skills.
```

This is the default feedback function initialization looks like:

```
# Initialize provider class
openai = OpenAI()
.
.
.
# Question/answer relevance between overall question and answer.
f_qa_relevance = Feedback(openai.relevance).on_input_output()

```

However, I'm using a local LLM with the OpenAI API.  I believe that I can use LiteLLM for this.

https://truera.com/evaluate-with-any-llm-trulens-litellm/

```
from trulens_eval import LiteLLM
litellm_provider  = LiteLLM(model_engine="gpt-3.5-turbo") # choose any model here!

# Define a relevance function using LiteLLM
relevance = Feedback(litellm_provider.relevance).on_input_output()
```

This unfortunately doesn't work.  I get an error message that says that the LLM has not been defined.

I've tried using LiteLLM, which you seem to support, but I cannot work out how to specifically reference this in your code.

```
import numpy as np

from trulens_eval import LiteLLM
from litellm import completion
import litellm

litellm.set_verbose=True

# Initialize Huggingface-based feedback function collection class:
hugs = feedback.Huggingface()
# Define a language match feedback function using HuggingFace.
f_lang_match = Feedback(hugs.language_match).on_input_output()

litellm_provider = feedback.LiteLLM(model_engine='mymodel', endpoint='http://localhost:1234/v1')
# Question/answer relevance between overall question and answer.
#f_qa_relevance = Feedback(openai.relevance).on_input_output()
f_qa_relevance = Feedback(litellm_provider.qs_relevance).on_input_output()


tru_query_engine = TruLlama(query_engine,
    app_id='LlamaIndex_App1',
#    feedbacks=[f_lang_match, f_qa_relevance, f_qs_relevance])
    feedbacks=[f_qa_relevance])
#    feedbacks=[f_qa_relevance])

# Instrumented query engine can operate like the original:
llm_response = tru_query_engine.query("What did the author do growing up?")

print(llm_response)

```

This is the error that I get:

```
WARNING:trulens_eval.app:Function <method-wrapper '__call__' of method object at 0x2988ff840> has not been instrumented. This may be ok if it will call a function that has been instrumented exactly once. Otherwise unexpected results may follow. You can use `AddInstruments.method` of `trulens_eval.instruments` before you use the `TruLlama` wrapper to make sure `__call__` does get instrumented. `TruLlama` method `print_instrumented` may be used to see methods that have been instrumented. 

`query` will be deprecated soon; To record results of your app's execution, use one of these options to invoke your app:
    (1) Use the `with_` method:
        ```python
        app # your app
        tru_app_recorder: TruLlama = TruLlama(app, ...)
        result = tru_app_recorder.with_(app.query, ...args/kwargs-to-app.query...)
        ```
    (2) Use TruLlama as a context manager: 
        ```python
        app # your app
        tru_app_recorder: TruLlama = TruLlama(app, ...)
        with tru_app_recorder:
            result = app.query(...args/kwargs-to-app.query...)
        
        ```


`query_with_record` will be deprecated soon; To record results of your app's execution, use one of these options to invoke your app:
    (1) Use the `with_record` method:
        ```python
        app # your app
        tru_app_recorder: TruLlama = TruLlama(app, ...)
        result, record = tru_app_recorder.with_record(app.query, ...args/kwargs-to-app.query...)
        ```
    (2) Use TruLlama as a context manager: 
        ```python
        app # your app
        tru_app_recorder: TruLlama = TruLlama(app, ...)
        with tru_app_recorder as records:
            result = app.query(...args/kwargs-to-app.query...)
        record = records.get()
        ```

The author grew up working on writing short stories and programming on an IBM 1401 computer in school. Additionally, they had access to surplus Xerox Dandelions which allowed them to experiment with computing technology outside of the classroom setting. The author's interest in art began at a young age, but they never imagined it was possible for them to create paintings until later in life when they started exploring their creative side and discovered that art could be learned just like any other skill. After attending Harvard, the author pursued their passion for art and began taking classes to develop their skills.
litellm.caching: False; litellm.caching_with_models: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None

Provider List: https://docs.litellm.ai/docs/providers


Give Feedback [/](https://file+.vscode-resource.vscode-cdn.net/) Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
ERROR:trulens_eval.feedback.provider.endpoint.base:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/mymodel',..)` Learn more: https://docs.litellm.ai/docs/providers. Retries remaining=3.
Logging Details LiteLLM-Failure Call
self.failure_callback: []
ERROR:trulens_eval.feedback.provider.endpoint.base:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/mymodel',..)` Learn more: https://docs.litellm.ai/docs/providers. Retries remaining=2.
litellm.caching: False; litellm.caching_with_models: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None

Provider List: https://docs.litellm.ai/docs/providers


Give Feedback [/](https://file+.vscode-resource.vscode-cdn.net/) Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
self.failure_callback: []
ERROR:trulens_eval.feedback.provider.endpoint.base:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/mymodel',..)` Learn more: https://docs.litellm.ai/docs/providers. Retries remaining=1.
litellm.caching: False; litellm.caching_with_models: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None

Provider List: https://docs.litellm.ai/docs/providers


Give Feedback [/](https://file+.vscode-resource.vscode-cdn.net/) Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
self.failure_callback: []
ERROR:trulens_eval.feedback.provider.endpoint.base:litellm request failed <class 'litellm.exceptions.APIConnectionError'>=LLM Provider NOT provided. Pass in the LLM provider you are trying to call. E.g. For 'Huggingface' inference endpoints pass in `completion(model='huggingface/mymodel',..)` Learn more: https://docs.litellm.ai/docs/providers. Retries remaining=0.
WARNING:trulens_eval.feedback.feedback:Feedback Function exception caught: Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 481, in run
    result_and_meta, part_cost = Endpoint.track_all_costs_tally(
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 377, in track_all_costs_tally
    result, cbs = Endpoint.track_all_costs(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 303, in track_all_costs
    return Endpoint._track_costs(thunk, with_endpoints=endpoints)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 480, in _track_costs
    result: T = thunk()
                ^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 482, in <lambda>
    lambda: self.imp(**ins)
            ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/provider/base.py", line 200, in qs_relevance
    self.endpoint.run_me(
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/provider/endpoint/base.py", line 215, in run_me
    raise RuntimeError(
RuntimeError: API litellm request failed 4 time(s).

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/trulens_eval/feedback/feedback.py", line 486, in run
    raise RuntimeError(
RuntimeError: Evaluation of qs_relevance failed on inputs: 
{'question': 'What did the author do growing up?',
 'statement': 'The author grew up working on writing short stories and '
    
API litellm request failed 4 time(s)..

litellm.caching: False; litellm.caching_with_models: False; litellm.cache: None
kwargs[caching]: False; litellm.cache: None

Provider List: https://docs.litellm.ai/docs/providers


Give Feedback [/](https://file+.vscode-resource.vscode-cdn.net/) Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm.set_verbose=True'.

Logging Details: logger_fn - None | callable(logger_fn) - False
Logging Details LiteLLM-Failure Call
self.failure_callback: []


```



Could it be that the issue is that it is failing on the embedding pointing to OpenAI's embedding model?  If so, then I can't work out how to ensure that I reference my own embedding model, like I can do with Llama-Index.  Is this available with TruLens?

This was resolved by providing all of the litellm values to ensure api_base etc were available.
i.e. 
```
litellm.api_base='http://localhost:1234/v1'
litellm.api_key='...'
```


Thanks for sharing back @jonmach - will get this added to docs to make it more clear for others.