zhiminy/EvalEng huggingface dataset:
Repository: truera/trulens
Issue: Default cut to InputAttribution is incorrect
Root cause: Algorithmic Error

The default `cut` argument to `InputAttribution` is set to None:

https://github.com/truera/trulens/blob/c93dbe328fd49be19c60815df566f62d3d882a44/trulens/nn/attribution.py#L465

This is then passed to `InternalInfluence` with a cut `(InputCut(), None)`, and `InternalInfluence` interprets a `None` cut as an `InputCut()`. This is not the intended default behavior, which should be `OutputCut()`.

This causes correctness issues, where attributions are returned identical to the input (tested on tensorflow==2.4.0).
Repository: truera/trulens
Issue: Can it work with a Local LLM?
Root cause: Validation Gap

```
import os
from langchain import PromptTemplate, LLMChain
from langchain.llms import GPT4All
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.vectorstores.faiss import FAISS
from langchain.embeddings import HuggingFaceEmbeddings
from trulens_eval import TruChain, Feedback, Huggingface, Tru


tru = Tru()


os.environ["HUGGINGFACE_API_KEY"] = ""


gpt4all_path = './models/gpt4all-converted.bin'


callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])


embeddings = HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2')
hugs = Huggingface()


f_lang_match = Feedback(hugs.language_match).on_input_output()

llm = GPT4All(model=gpt4all_path,verbose=True,temp=0.2)


index = FAISS.load_local("my_faiss_index", embeddings)


template = """Answer the following question using {context}
Question: {question}
Answer:
"""


def get_best_answer(question):
    matched_docs, sources = similarity_search(question, index, n=4)
    context = "\n".join([doc.page_content for doc in matched_docs])
    prompt = PromptTemplate(template=template, input_variables=["context", "question"]).partial(context=context)
    llm_chain = LLMChain(prompt=prompt, llm=llm)
    truchain = TruChain(llm_chain, app_id='Chain1_ChatApplication', feedbacks=[f_lang_match],tags = "prototype")
    answer = truchain.run(question)
    return answer



def similarity_search(query, index, n=4):
    matched_docs = index.similarity_search(query, k=n)
    sources = []
    for doc in matched_docs:
        sources.append(
            {
                "page_content": doc.page_content,
                "metadata": doc.metadata,
            }
        )
    return matched_docs, sources


while True:

    question = input("Please enter your question (or type 'exit' to close the program): ")


    if question.lower() == "exit":
        break


    answer = get_best_answer(question)


    print("Answer:", answer)

```
gives
```
✅ app Chain1_ChatApplication -> default.sqlite
✅ feedback def. feedback_definition_hash_44c43fe23fdbb98055154e6bb126142a -> default.sqlite
Traceback (most recent call last):
  File "c:\Users\abhishek\Local-LLM\gpt4all_truelens.py", line 75, in <module>
    answer = get_best_answer(question)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\abhishek\Local-LLM\gpt4all_truelens.py", line 48, in get_best_answer
    answer = truchain.run(question)
             ^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'RuntimeError' object is not callable`
```
Repository: truera/trulens
Issue: Testing Local LLMs?
Root cause: Documentation Deficiency

```
import os
from langchain import PromptTemplate, LLMChain
from langchain.llms import GPT4All
from langchain.callbacks.manager import CallbackManager
from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler
from langchain.vectorstores.faiss import FAISS
from langchain.embeddings import HuggingFaceEmbeddings
from trulens_eval import TruChain, Feedback, Huggingface, Tru


tru = Tru()


os.environ["HUGGINGFACE_API_KEY"] = ""


gpt4all_path = './models/gpt4all-converted.bin'


callback_manager = CallbackManager([StreamingStdOutCallbackHandler()])


embeddings = HuggingFaceEmbeddings(model_name='sentence-transformers/all-MiniLM-L6-v2')
hugs = Huggingface()


f_lang_match = Feedback(hugs.language_match).on_input_output()

llm = GPT4All(model=gpt4all_path,verbose=True,temp=0.2)


index = FAISS.load_local("my_faiss_index", embeddings)


template = """Answer the following question using {context}
Question: {question}
Answer:
"""


def get_best_answer(question):
    matched_docs, sources = similarity_search(question, index, n=4)
    context = "\n".join([doc.page_content for doc in matched_docs])
    prompt = PromptTemplate(template=template, input_variables=["context", "question"]).partial(context=context)
    llm_chain = LLMChain(prompt=prompt, llm=llm)
    truchain = TruChain(llm_chain, app_id='Chain1_ChatApplication', feedbacks=[f_lang_match],tags = "prototype")
    answer = truchain.run(question)
    return answer



def similarity_search(query, index, n=4):
    matched_docs = index.similarity_search(query, k=n)
    sources = []
    for doc in matched_docs:
        sources.append(
            {
                "page_content": doc.page_content,
                "metadata": doc.metadata,
            }
        )
    return matched_docs, sources


while True:

    question = input("Please enter your question (or type 'exit' to close the program): ")


    if question.lower() == "exit":
        break


    answer = get_best_answer(question)


    print("Answer:", answer)

```
gives

```
✅ app Chain1_ChatApplication -> default.sqlite
✅ feedback def. feedback_definition_hash_44c43fe23fdbb98055154e6bb126142a -> default.sqlite
Traceback (most recent call last):
  File "c:\Users\abhishek\Local-LLM\gpt4all_truelens.py", line 75, in <module>
    answer = get_best_answer(question)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\Users\abhishek\Local-LLM\gpt4all_truelens.py", line 48, in get_best_answer
    answer = truchain.run(question)
             ^^^^^^^^^^^^^^^^^^^^^^
TypeError: 'RuntimeError' object is not callable`

```

Can you try taking out the `.run` call? The TruChain should be called via `truchain(question)`

Thanks, It works using above.
Repository: truera/trulens
Issue: Support for Llama_Index's query_engine aquery method and chat_engine chat/achat methods
Root cause: Unimplemented Feature Gap

Currently trulens supports LlamaIndex's query method, but doesn't support aquery. Also we need support of chat_engine methods: chat and achat.

Thanks!

Hi @proDOOMman , we are working on a solution to this here https://github.com/truera/trulens/pull/341 . There are some issues with using streaming mode (llama index does not support streaming with async right now).

both aquery and stream_chat,astream_chat were in the last release
Repository: truera/trulens
Issue: Installation fails due to fastavro
Root cause: External Dependency Breakage

Trulens package needs `fastavro==1.7.4`. 
1.7.4 version has an issue and it has been fixed in latest update (1.8.2)
Can you please fix this ?
 

As a workaround I did `pip install --no-build-isolation fastavro==1.7.4` and then `pip install trulens-eval` 
Reference : https://github.com/fastavro/fastavro/issues/701#issuecomment-1639051800

Looks like this dependency is introduced from cohere. Also; they just merged the fix it seems 4 minutes ago
https://github.com/cohere-ai/cohere-python/pull/273
Repository: truera/trulens
Issue: [Bug] Selector `GetItems` crashes with `AttributeError`
Root cause: Validation Gap

### How to reproduce:
```python
from trulens_eval import Select

selector = Select.RecordCalls._call.args.inputs[["key1", "key2"]]

repr(selector)
```
Output:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File ".../.venv/lib/python3.9/site-packages/trulens_eval/util.py", line 951, in __repr__
    return "JSONPath()" + ("".join(map(repr, self.path)))
  File ".../.venv/lib/python3.9/site-packages/trulens_eval/util.py", line 926, in __repr__
    return f"[{','.join(self.indices)}]"
AttributeError: 'GetItems' object has no attribute 'indices'
```

Then obviously this crashes any Feedback that uses this selector

### Suggested fix
On [util.py](https://github.com/truera/trulens/blob/e85e8f18763f38790e5964d2a77e785416f09647/trulens_eval/trulens_eval/util.py#L926)
```python
    def __repr__(self):
        return f"[{','.join(self.items)}]"
```

Thanks @lariel-fernandes for finding, the easy repro and suggested fix! Was able to repro and fix with your suggestion.

@joshreini1 I had meant to open a PR myself, since it's such an easy fix, but I couldn't (GH says `you need to fork this repository to propose changes`)

Is that the standard procedure? (sorry, not very used to contributing to open source projects here 😅 )

That is the standard procedure 😀, feel free to fork and PR in the future!

I just merged this change, so you can expect it in the next release, 0.8.0, or build from main if you want it sooner!
Repository: truera/trulens
Issue: AzureOpenAI fails: missing deployment_id 
Root cause: Validation Gap

Bug when using the new AzureOpenAI wrapper added in PR #242 

### How to reproduce: 
```
azure = AzureOpenAI(model_engine="gpt-35-turbo", deployment_id="gpt-4")
```

### Error: 
```
Traceback (most recent call last):
  File "/home/lab/lab392-rfpvirtualassistant/scripts/langchain/eval.py", line 15, in <module>
    pipeline = QAPipeline(config['AWS']['AWSKendraSearchIndexId'], config['OpenAI']['OpenAIAnswerDeployment'], eval_mode=True, answers=answers)
  File "/home/lab/lab392-rfpvirtualassistant/scripts/langchain/pipeline.py", line 58, in __init__
    azure = AzureOpenAI(model_engine="gpt-35-turbo", deployment_id=openAIDeployment)
  File "/home/lab/anaconda/envs/lab392/lib/python3.10/site-packages/trulens_eval/feedback.py", line 1689, in __init__
    super().__init__(
  File "/home/lab/anaconda/envs/lab392/lib/python3.10/site-packages/trulens_eval/feedback.py", line 1049, in __init__
    super().__init__(
  File "/home/lab/anaconda/envs/lab392/lib/python3.10/site-packages/trulens_eval/feedback.py", line 1020, in __init__
    super().__init__(*args, **kwargs)
  File "/home/lab/anaconda/envs/lab392/lib/python3.10/site-packages/trulens_eval/util.py", line 1660, in __init__
    super().__init__(__tru_class_info=class_info, **kwargs)
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for AzureOpenAI
deployment_id
  field required (type=value_error.missing)
```

I believe the problem is that in class `OpenAI`, the `deployment_id` parameter is lost since it is not assigned to `self_kwargs`. PR incoming with my suggestion to fix this bug. 

Fixed in #338
Repository: truera/trulens
Issue: [FR] Allow passing `dict` as input to feedback function
Root cause: Validation Gap

## Context
Some feedback functions may need to operate on the final formatted prompt, not only on prompt inputs. For example, when using the ReAct pattern the prompt might contain dynamic instructions or examples that need to be taken into account when evaluating the LLM behavior.

One could try to achieve that by passing both the prompt template and the inputs dictionary to the feedback function, then letting it format the prompt before calculating the metric:
```python
class CustomProvider(Provider):
    def my_feedback_func(self, prompt: str, inputs: dict) -> float:
        prompt = prompt.format(**inputs)
        return float(len(prompt))

Feedback(CustomProvider().my_feedback_func) \
    .on(Select.App.app.prompt.template) \
    .on(Select.RecordCalls._call.args.inputs)
```

However, that is not possible, because the [FeedbackCall](https://github.com/truera/trulens/blob/bac458e9f1d3054bb98645fe15ac8c3be49c4e93/trulens_eval/trulens_eval/schema.py#L284) schema doesn't allow a `dict` as argument to the feedback function.
Using the above defined feedback in an app will produce the following error:
```
Feedback Function Exception Caught: Traceback (most recent call last):
  File ".../.venv/lib/python3.9/site-packages/trulens_eval/feedback.py", line 862, in run
    feedback_call = FeedbackCall(
  File "pydantic/main.py", line 341, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 1 validation error for FeedbackCall
args -> inputs
  str type expected (type=type_error.str)
```

## Suggestion
Change the schema of [FeedbackCall](https://github.com/truera/trulens/blob/bac458e9f1d3054bb98645fe15ac8c3be49c4e93/trulens_eval/trulens_eval/schema.py#L284) to the following:
```python
class FeedbackCall(SerialModel):
    args: Dict[str, Union[str, JSON]]
    ret: float
```
Repository: truera/trulens
Issue: Feedback function on Groundedness did not work with AzureOpenAI
Root cause: Configuration Error

The summarize_provider is default to OpenAI, which prevents from instantiating ChatCompletion with AzureOpenAI.

![image](https://github.com/truera/trulens/assets/85856280/659cf885-b1df-434c-96f2-55f8928e56a4)


Hi @tarynduong - I have an open PR to fix this issue [here](https://github.com/truera/trulens/pull/369). Please take a look to see if this solution works for your case.

Thanks!
Repository: truera/trulens
Issue: Add multiple tags using a list or multiple columns called tag1, tag2 to TruChain
Root cause: Documentation Deficiency

This will be helpful if we want to track user ID, and session ID, customer ID and other related information in the session. 

Hey @harkrish1 !

In 0.11.0 we added record-level metadata. You can add the `record_metadata` arg to your application call like so:
```
metadata = dict(user_id="u_xyz", session_id="session_abc", customer_id="cust_123")
tru_chain(input_prompt, record_metadata = metadata)
```

Does this work for you?
Repository: truera/trulens
Issue: Changing the input of `agreement_measure` returns nan
Root cause: Validation Gap

**Usecase:** When changing the input to a custom input, `agreement_measure` returns nan. 

If one changes the input like so: 

```python
f_agreement_measure = Feedback(GroundTruthAgreement(answers, azure).agreement_measure).on(
                Select.Record.calls[0].rets.query).on(
                Select.Record.main_output["answer"]
)
```

You can see that in `agreement_measure_calls`, the `ret` and `meta` are empty because it wasn't able to find the input in the dictionary:

```
[{'args': {'prompt': 'Prompt 1 Modified', 'response': "response 1"}, 'ret': nan, 'meta': {}}]
```

From what I've understood so far, it's because in `GroundTruthAgreement` and `_find_response()`, it is searching for the original query `Prompt 1`, like in `Select.Record.main_input`. Instead, it is searching for a query containing `Prompt 1 Modified`, contained in `Select.Record.calls[0].rets.query`, but can't find it: 

```
self.ground_truth = [{'query': 'Prompt 1', 'response': 'response 1'}]
```

Therefore, it returns nan. 

Hi; what is `answers` populated with in your example?

Hi! `answers = []`, and I don't have them at the moment that I call the `Feedback` function as it is changed in the chain, so I don't see how I could populate it at that stage.... (?)

@Mymoza that's the intended behavior. Essentially if there isn't a ground truth entry, the output is undefined. It also doesn't get picked up by the statistics on the leaderboard.
Repository: truera/trulens
Issue: [Bug] Dashboard breaks with `llama-index>0.7.23`
Root cause: External Dependency Breakage

**Affects releases: 0.9.0**

## Issue
Dashboard breaks if I my Python environment has `llama-index 0.7.24.post1` or later.

Not sure if it's an issue with the new `llama-index` releases or a compatibility issue between that and trulens-eval.

I can only get it to work by setting `llama-index>=0.7.16,<=0.7.23` in `requirements.txt` or `pyproject.toml`

## Reproduce
Update to `llama-index=0.7.24.post1` or later (up to `0.8.2`), then run `Tru().start_dashboard()` and browse to the dashboard.

See dashboard log here: [log.txt](https://github.com/truera/trulens/files/12359341/log.txt)

## Suggestion
A temporary workaround would be pinning the `llama-index` version to `0.7.23` in `requirements.txt` and `setup.py`.

But users of trulens-eval would miss on any future `llama-index` updates while the issue lasts.


Fix in next release. Temporary workaround is discussed here: https://github.com/jerryjliu/llama_index/issues/7244 .

Released in 0.10.0
Repository: truera/trulens
Issue: [Bug] Missing Package and Unicode Encode Error
Root cause: Validation Gap

Hello, you forgot to import os :D 
![image](https://github.com/truera/trulens/assets/85856280/4b8a2594-1bdf-4a3c-9478-025d3f90ceef)
And it seems that these symbols below only work with Streamlit. The Azure Function libs do not support encoding these unicode characters. When I replace them with nothing, my app works. Could you consider removing them? 
<img width="402" alt="Capture" src="https://github.com/truera/trulens/assets/85856280/a6b429a3-e414-4606-b42f-6355579d51c4">

@tarynduong Thanks for the feedback! Could you share a bit more about how you're using Azure Functions and where you're running into errors?

Bump @tarynduong - can you share more about the unicode errors you're seeing with Azure Functions?

Hello,
I have the same problem on a windows machine runnning python 3.10
```
Tru().run_dashboard()
File "C:\Users\lkNGAT\project\venv\lib\site-packages\trulens_eval\tru.py", line 117, in __init__
print(
File "C:\Users\lkNGAT\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode character '\U0001f991' in position 0: character maps to <undefined>
```

There is some discussion here: https://stackoverflow.com/questions/14630288/unicodeencodeerror-charmap-codec-cant-encode-character-maps-to-undefined

@lkNGAT @tarynduong Can you put this in your code before you import or use trulens and let us know if the issue persists:
```python
import sys
if sys.stdout.encoding != 'cp850':
  sys.stdout = codecs.getwriter('cp850')(sys.stdout.buffer, 'strict')
if sys.stderr.encoding != 'cp850':
  sys.stderr = codecs.getwriter('cp850')(sys.stderr.buffer, 'strict')
```
Repository: truera/trulens
Issue: Using it with out the Sqllite db
Root cause: Documentation Deficiency

Hi Teams
The record tracking function is look great！
I would like to ask is it possible to use it with out running a db, I mainly want to use it to get the tracking record for each time of my call.

Thank you very much

Hi @ricoyudog ,
Yes you can! 

Each tru wrapper has a *_with_record method that has this information populated.

for example: https://www.trulens.org/trulens_eval/api/truchain/
```
response = my_llm_app(query)

from trulens_eval import TruChain
truchain = TruChain(
    my_llm_app,
    app_id='Chain1_ChatApplication')

response, tru_record = truchain.call_with_record(query)

latency = tru_record.perf.end_time - tru_record.perf.start_time
```

> 

But it seems like when my pipeline is finished, the terminal is still not close

Are you running the dashboard? That kicks off a long living thread that will prevent terminal closing. You can either stop the dashboard with tru.stop_dashboard() or not run the dashboard

Otherwise let me know if I'm not on the right path in understanding your question
Repository: truera/trulens
Issue: "Display full app json" always shows data for first row on the Dashboard
Root cause: Algorithmic Error

If I open trulens-eval dashboard evaluations -> http://localhost:8501/Evaluations
And have mulitple records,
when I select a different record then the first one in the row,
still the first record's data is shown in the bottom part "Display full app json".
Expected behaviour: the data for the other row should be displayed.

My code to run trulens:
```
truchain = TruChain(chain, app_id="app_id", tags="my tag)
await truchain.acall_with_record(variables)

```
trulens-eval version: 0.11.0

An app json is a specific state of the wrapped app. It is keyed by the app_id. It's likely that if you are expecting differences in the app json, that its probably picking up only 1 version based on the app_id. Can you validate that if you are expecting app json changes, and the app_id is changed, that the json gets updated?

If that is validated; we can think of having a hash key based upon the json value to make sure we pick up changes.
Repository: truera/trulens
Issue: No notebooks for NLP and time-series RNNs .
Root cause: Unimplemented Feature Gap

The notebooks directory contains notebooks named [intro_demo_pytorch.ipynb](https://github.com/truera/trulens/blob/master/notebooks/intro_demo_pytorch.ipynb) and [intro_demo_tf2_keras.ipynb](https://github.com/truera/trulens/blob/master/notebooks/intro_demo_tf2_keras.ipynb) , these notebooks show how trulens can be used for visualizing and explaining an image classification model. I request similar notebooks for NLP and time-series RNNs so that users can easily get started with them.
Thankyou!

Recently added was https://github.com/truera/trulens/blob/master/notebooks/nlp_demo_pytorch.ipynb .
Repository: truera/trulens
Issue: Multiple tags for a record?

I was wondering if it is supported to add multiple tags for a record.

Example code from the documentation for adding tags:
https://www.trulens.org/trulens_eval/langchain_quickstart/#instrument-chain-for-logging-with-trulens
```
truchain = TruChain(chain,
    app_id='Chain1_ChatApplication',
    feedbacks=[f_lang_match],
    tags = "prototype")
```

What if I wanted to add one more tag, e.g. 
`tags = ["tag1","tag2"]`

Is something like this supported?
Currently I get an error, that a single string is expected.

Hey @molntamas !

Right now there's a few options:
For app-level tags, you can pass the `metadata` arg when you wrap your application, like:
```
tru_query_engine = TruLlama(query_engine,
                    metadata={
                        'embed_model':embed_model_name,
                        'top_k':top_k,
                        'chunk_size':chunk_size
                        })
```

Or for record-level information you can add the `record_metadata` arg to your application call like so:
```
metadata = dict(tag1="prototype", field2="segment A")
tru_chain(input_prompt, record_metadata = metadata)
```

Do either of these approaches work for you?