Repository: langchain
Issue: ChatDatabricks() unable to stream special characters
Labels: 🤖:bug,investigate

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

## Config rag_chain_config.yaml

```yml
databricks_resources:
  llm_endpoint_name: gpt-4o-mini
  vector_search_endpoint_name: VS_ENDPOINT
input_example:
  messages:
  - content: Hvilken vinterrug sort havde mest udbytte i 2023?
    role: user
  - content: Det største udbytte i årets landsforsøg med sorter af vinterrug i 2023
      var 112,1 hkg pr. ha, og det blev høstet i sorten KWS Cursor.
    role: assistant
  - content: Hvilken vinterhvede sort havde mest udbytte i 2023?
    role: user
llm_config:
  llm_parameters:
    max_tokens: 1500
    temperature: 0.01
  llm_system_prompt_template: "\n    You are a trusted assistant that helps answer\
    \ questions based only on the provided information. If you do not know the answer\
    \ to a question, you truthfully say you do not know.  Here is some context which\
    \ might or might not help you answer: {context}.  Answer directly, do not repeat\
    \ the question, do not start with something like: the answer to the question,\
    \ do not add AI in front of your answer, do not say: here is the answer, do not\
    \ mention the context or the question. You must provide your answer in the Danish\
    \ language or in the detected language if the queried question from the user differs\
    \ from Danish.\n    If the answer contains multiple steps or points, provide the\
    \ answer in a bullet format. \n    Based on this context, answer this question:\
    \ {question}\n"
retriever_config:
  chunk_template: 'Passage: {chunk_text}

    '
  data_pipeline_tag: poc
  parameters:
    k: 5
    query_type: hybrid
  schema:
    chunk_text: chunked_text
    document_uri: path
    primary_key: chunk_id
  vector_search_index: VS_INDEX

```

## Code example
```python
from operator import itemgetter
import mlflow
import os

from databricks.vector_search.client import VectorSearchClient

from langchain_community.chat_models import ChatDatabricks
from langchain_community.vectorstores import DatabricksVectorSearch

from langchain_core.runnables import RunnableLambda
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain_core.prompts import (
    PromptTemplate,
    ChatPromptTemplate,
)

## Enable MLflow Tracing
mlflow.langchain.autolog()


############
# Helper functions
############
# Return the string contents of the most recent message from the user
def extract_user_query_string(chat_messages_array):
    return chat_messages_array[-1]["content"]


# Return the chat history, which is is everything before the last question
def extract_chat_history(chat_messages_array):
    return chat_messages_array[:-1]


# FIT AND FINISH: We should not require a value here.
model_config = mlflow.models.ModelConfig(development_config='rag_chain_config.yaml')

databricks_resources = model_config.get("databricks_resources")
retriever_config = model_config.get("retriever_config")
llm_config = model_config.get("llm_config")

############
# Connect to the Vector Search Index
############
vs_client = VectorSearchClient(disable_notice=True)
vs_index = vs_client.get_index(
    endpoint_name=databricks_resources.get("vector_search_endpoint_name"),
    index_name=retriever_config.get("vector_search_index"),
)
vector_search_schema = retriever_config.get("schema")

############
# Turn the Vector Search index into a LangChain retriever
############
vector_search_as_retriever = DatabricksVectorSearch(
    vs_index,
    text_column=vector_search_schema.get("chunk_text"),
    columns=[
        vector_search_schema.get("primary_key"),
        vector_search_schema.get("chunk_text"),
        vector_search_schema.get("document_uri"),
    ],
).as_retriever(search_kwargs=retriever_config.get("parameters"))

############
# Required to:
# 1. Enable the RAG Studio Review App to properly display retrieved chunks
# 2. Enable evaluation suite to measure the retriever
############

mlflow.models.set_retriever_schema(
    primary_key=vector_search_schema.get("primary_key"),
    text_column=vector_search_schema.get("chunk_text"),
    doc_uri=vector_search_schema.get(
        "document_uri"
    ),  # Review App uses `doc_uri` to display chunks from the same document in a single view
)


############
# Method to format the docs returned by the retriever into the prompt
############
def format_context(docs):
    chunk_template = retriever_config.get("chunk_template")
    chunk_contents = [
        chunk_template.format(
            chunk_text=d.page_content,
            document_uri=d.metadata[vector_search_schema.get("document_uri")],
        )
        for d in docs
    ]
    return "".join(chunk_contents)

############
# Prompt Template for generation
############
prompt = ChatPromptTemplate.from_messages(
    [
        (  # System prompt contains the instructions
            "system",
            llm_config.get("llm_system_prompt_template"),
        ),
        # User's question
        ("user", "{question}"),
    ]
)

############
# FM for generation
############
model = ChatDatabricks(
    endpoint=databricks_resources.get("llm_endpoint_name"),
    extra_params=llm_config.get("llm_parameters")
)

############
# RAG Chain
############
chain = (
    {
        "question": itemgetter("messages") | RunnableLambda(extract_user_query_string),
        "context": itemgetter("messages")
        | RunnableLambda(extract_user_query_string)
        | vector_search_as_retriever
        | RunnableLambda(format_context),
    }
    | prompt
    | model
    | StrOutputParser()
)

## Tell MLflow logging where to find your chain.
# `mlflow.models.set_model(model=...)` function specifies the LangChain chain to use for evaluation and deployment.  This is required to log this chain to MLflow with `mlflow.langchain.log_model(...)`.

mlflow.models.set_model(model=chain)

# Streaming example output
chain_input = {
    "messages": [
        {
            "role": "user",
            "content": "Hvordan bekæmper man væselhale?",
        }
    ]
}

for chunk in chain.stream(chain_input):
    print(chunk, end="|", flush=True)
```

### Error Message and Stack Trace (if applicable)

## Code example streams the following strange formatting of special Danish characters "æ, ø, å": 
```
|-| Bek|Ã¦|mp|else| skal| ske| sÃ¥| tid|ligt| som| muligt| efter| frem|spiring|.
|-| An|vend|else| af| M|aten|o| Duo| |600| SC| er| effektiv|,| isÃ¦r| ved| tid|lig| behandling|.
|-| Be|handling| med| |0|,|7| l| M|aten|o| Duo| pr|.| ha| den| |20|.| august| har| g|ivet| de| bedste| result|ater|.
|-| T|ils|Ã¦t|ning| af| Boxer| kan| forbed|re| bek|Ã¦|mp|elsen|,| men| kan| vÃ¦re| problem|at|isk| pÃ¥| grund| af| ford|amp|ning|.
|-| Det| er| vigtigt| at| reduc|ere| vÃ¦|sel|hale| til| et| niveau|,| hvor| de| ikke| pÃ¥vir|ker| op|form|ering| og| sp|red|ning|.||

```

However, when manually correcting the output:

```python
for chunk in chain.stream(chain_input):
    # Decode using latin1 (or the incorrect encoding) and re-encode in UTF-8
    corrected_chunk = chunk.encode('latin1').decode('utf-8')
    print(corrected_chunk, end="|", flush=True)

```

```
|-| Bek|æ|mp|else| skal| ske| så| tid|ligt| som| muligt| efter| frem|spiring|.
|-| An|vend|else| af| M|aten|o| Duo| |600| SC| er| effektiv|,| især| ved| tid|lig| behandling|.
|-| Be|handling| med| |0|,|7| l| M|aten|o| Duo| pr|.| ha| den| |20|.| august| har| vist| sig| at| være| den| bedste| metode|.
|-| T|ils|æt|ning| af| Boxer| kan| forbed|re| bek|æ|mp|elsen|,| men| kan| være| problem|at|isk| at| anv|ende| tid|ligt| på| grund| af| ford|amp|ning|.
|-| Det| er| vigtigt| at| reduc|ere| væ|sel|hale| til| et| niveau|,| hvor| de| ikke| påvir|ker| op|form|ering| og| sp|red|ning|.||
```

## Error clearly happens in ChatDatabricks()
![image](https://github.com/user-attachments/assets/b70857e0-ecc5-4f15-9b69-3641acb5e3ef)

## Workaround is currently to turn off streaming
```python 
model = ChatDatabricks(
    endpoint=databricks_resources.get("llm_endpoint_name"),
    extra_params=llm_config.get("llm_parameters"),
    disable_streaming=True
)
```
![image](https://github.com/user-attachments/assets/72be3796-134f-41d9-9724-9f2a34a55bdb)


### Description

from langchain_community.chat_models import ChatDatabricks unable to stream special Danish characters.

I expect to see "æ", "ø" and "å".

Instead I see strange representations formatted like: "Ã¦t" and "Ã¥"

### System Info

anyio==3.5.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asttokens==2.0.5
attrs==22.1.0
backcall==0.2.0
beautifulsoup4==4.11.1
black==22.6.0
bleach==4.1.0
blinker==1.4
boto3==1.24.28
botocore==1.27.96
certifi==2022.12.7
cffi==1.15.1
chardet==4.0.0
charset-normalizer==2.0.4
click==8.0.4
comm==0.1.2
contourpy==1.0.5
cryptography==39.0.1
cycler==0.11.0
Cython==0.29.32
databricks-sdk==0.1.6
dbus-python==1.2.18
debugpy==1.6.7
decorator==5.1.1
defusedxml==0.7.1
distlib==0.3.7
distro==1.7.0
distro-info==1.1+ubuntu0.2
docstring-to-markdown==0.11
entrypoints==0.4
executing==0.8.3
facets-overview==1.1.1
fastjsonschema==2.19.1
filelock==3.13.1
fonttools==4.25.0
googleapis-common-protos==1.62.0
grpcio==1.48.2
grpcio-status==1.48.1
httplib2==0.20.2
idna==3.4
importlib-metadata==4.6.4
ipykernel==6.25.0
ipython==8.14.0
ipython-genutils==0.2.0
ipywidgets==7.7.2
jedi==0.18.1
jeepney==0.7.1
Jinja2==3.1.2
jmespath==0.10.0
joblib==1.2.0
jsonschema==4.17.3
jupyter-client==7.3.4
jupyter-server==1.23.4
jupyter_core==5.2.0
jupyterlab-pygments==0.1.2
jupyterlab-widgets==1.0.0
keyring==23.5.0
kiwisolver==1.4.4
launchpadlib==1.10.16
lazr.restfulclient==0.14.4
lazr.uri==1.0.6
lxml==4.9.1
MarkupSafe==2.1.1
matplotlib==3.7.0
matplotlib-inline==0.1.6
mccabe==0.7.0
mistune==0.8.4
more-itertools==8.10.0
mypy-extensions==0.4.3
nbclassic==0.5.2
nbclient==0.5.13
nbconvert==6.5.4
nbformat==5.7.0
nest-asyncio==1.5.6
nodeenv==1.8.0
notebook==6.5.2
notebook_shim==0.2.2
numpy==1.23.5
oauthlib==3.2.0
packaging==23.2
pandas==1.5.3
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.10.3
patsy==0.5.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==2.5.2
plotly==5.9.0
pluggy==1.0.0
prometheus-client==0.14.1
prompt-toolkit==3.0.36
protobuf==4.24.0
psutil==5.9.0
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==8.0.0
pyarrow-hotfix==0.5
pycparser==2.21
pydantic==1.10.6
pyflakes==3.1.0
Pygments==2.11.2
PyGObject==3.42.1
PyJWT==2.3.0
pyodbc==4.0.32
pyparsing==3.0.9
pyright==1.1.294
pyrsistent==0.18.0
python-apt==2.4.0+ubuntu4
python-dateutil==2.8.2
python-lsp-jsonrpc==1.1.1
python-lsp-server==1.8.0
pytoolconfig==1.2.5
pytz==2022.7
pyzmq==23.2.0
requests==2.28.1
rope==1.7.0
s3transfer==0.6.2
scikit-learn==1.1.1
scipy==1.10.0
seaborn==0.12.2
SecretStorage==3.3.1
Send2Trash==1.8.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2.post1
ssh-import-id==5.11
stack-data==0.2.0
statsmodels==0.13.5
tenacity==8.1.0
terminado==0.17.1
threadpoolctl==2.2.0
tinycss2==1.2.1
tokenize-rt==4.2.1
tomli==2.0.1
tornado==6.1
traitlets==5.7.1
typing_extensions==4.4.0
ujson==5.4.0
unattended-upgrades==0.1
urllib3==1.26.14
virtualenv==20.16.7
wadllib==1.3.6
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==0.58.0
whatthepatch==1.0.2
widgetsnbextension==3.6.1
yapf==0.33.0
zipp==1.0.0