Repository: truera/trulens
Issue: Error when using Streaming mode in trulens for openai chat completion.
Root cause: Unimplemented Feature Gap

**Bug Description**
I am trying to use streaming mode in OpenAI chat completions through the Trulens library. When attempting to instrument my generation completion code with streaming enabled, I received the following error:
```
Could not find usage information in openai response:
<openai.Stream object at 0x00000250663A5270>
Could not find usage information in openai response:
<openai.Stream object at 0x00000250663A5270>
Could not find usage information in openai response:
<openai.Stream object at 0x00000250663A5270>
Could not find usage information in openai response:
<openai.Stream object at 0x00000250663A5270>
D:\miniconda3\envs\openai_venv\lib\site-packages\numpy\core\fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
D:\miniconda3\envs\openai_venv\lib\site-packages\numpy\core\_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
```

**To Reproduce**
```
@instrument
def generate_completion(self, query: str, context_str: str) -> str:
    response = client.chat.completions.create(
        model="gpt-3.5-turbo",
        max_tokens=500,
        messages=[
            {"role": "system", "content": system_prompt()},
            {"role": "user", "content": context_str}
        ],
        stream=True
    )
    collected_messages = []
    for chunk in response:
        token = chunk.choices[0].delta.content
        print(token)
        if isinstance(token, str):
            previous_tokens += token
            collected_messages.append(token)

    response = ''.join([m for m in collected_messages if m is not None])
    return response

```
"When I disable streaming, the code works fine. However, when streaming is enabled, it throws this error. Can anyone help me understand how to resolve this issue?"



<!-- Greeting -->
Hey @Sushmita10062002, great to see you back here! Hope you're doing well.

<!-- Answer -->
Based on the error message you provided, it seems like the Trulens library is having trouble finding usage information in the OpenAI response when streaming mode is enabled. This could be due to a variety of reasons, such as a misconfiguration in the endpoint or a lack of support for streaming mode in the Trulens library.

Here are some general steps you can take to troubleshoot this issue:

1. **Ensure Streaming Mode Support**: Verify that the Trulens library or the specific endpoint wrapper you are using (e.g., `OpenAI`, `AzureOpenAI`, `LiteLLM`) supports streaming mode. This might involve checking the library's documentation or source code.

2. **Update Endpoint Configuration**: If the library supports streaming mode, ensure that the endpoint configuration (e.g., within `LiteLLM` or `OpenAI` classes) includes necessary parameters for streaming. This might involve setting flags or parameters that enable streaming mode.

3. **Handle Stream Chunks**: Modify the cost tracking logic in `_test_llm_provider_endpoint` or a similar method to correctly account for `n_stream_chunks` when streaming mode is used. This might involve capturing and tallying the number of chunks received during a streaming session.

4. **Check API Version Compatibility**: Ensure that the `OPENAI_API_VERSION` environment variable is set to a version that supports streaming mode. The provided code sets this to `"2023-07-01-preview"`, which should be verified against OpenAI's API documentation for streaming support.

5. **Review Error Messages**: If you are encountering specific error messages, review them for hints on what might be misconfigured or unsupported. This can guide specific adjustments in your code or configuration.

6. **Consult Documentation**: Review the Trulens library and OpenAI API documentation for any updates or guidance on using streaming mode. There might be specific instructions or examples on how to enable and use streaming mode effectively.

If you're still encountering issues after trying these steps, it would be helpful if you could provide more information about your setup. Specifically, it would be useful to know:

- The version of the Trulens library you're using
- The version of the OpenAI API you're using
- Any modifications you've made to the Trulens library or your endpoint configuration to support streaming mode

This additional information will help in diagnosing the issue and providing a more specific solution.

## Sources
- [trulens_eval/tests/e2e/test_endpoints.py](https://github.com/truera/trulens/blob/main/trulens_eval/tests/e2e/test_endpoints.py)
- [trulens_eval/trulens_eval/feedback/provider/endpoint/bedrock.py](https://github.com/truera/trulens/blob/main/trulens_eval/trulens_eval/feedback/provider/endpoint/bedrock.py)

<!-- Dosu Comment Footer -->
<details>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>


@piotrm0 can you take a look at this issue?

<!-- Answer -->
Hi @joshreini1, thank you for getting back to us. We appreciate your response. We'll go ahead and close this issue now. If you have any further questions or need additional assistance, feel free to open a new issue. Thank you again!


<!-- Dosu Comment Footer -->
<details>
<summary>

#### Details
</summary>



**About Dosu**
This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.
</details>

*To continue the conversation, mention @dosu.*


@joshreini1 Not sure why Dosu closed this issue - we are looking into it. Sorry for the confusion!