Repository: langchain
Issue: langchain-huggingface: Using ChatHuggingFace requires hf token for local TGI using localhost HuggingFaceEndpoint
Labels: 🤖:improvement,investigate

### Checked other resources

- [X] I added a very descriptive title to this issue.
- [X] I searched the LangChain documentation with the integrated search.
- [X] I used the GitHub search to find a similar question and didn't find it.
- [X] I am sure that this is a bug in LangChain rather than my code.
- [X] The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

### Example Code

```python
from langchain_huggingface import HuggingFaceEndpoint, ChatHuggingFace

# This part works as expected
llm = HuggingFaceEndpoint(endpoint_url="http://127.0.0.1:8080")

# This part raises huggingface_hub.errors.LocalTokenNotFoundError
chat_llm = ChatHuggingFace(llm=llm)
```

### Error Message and Stack Trace (if applicable)

Traceback (most recent call last):
  .venv/lib/python3.10/site-packages/langchain_huggingface/chat_models/huggingface.py", line 320, in __init__
    self._resolve_model_id()

  .venv/lib/python3.10/site-packages/langchain_huggingface/chat_models/huggingface.py", line 458, in _resolve_model_id
    available_endpoints = list_inference_endpoints("*")

  .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 7081, in list_inference_endpoints
    user = self.whoami(token=token)

  .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)

  .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 1390, in whoami
    headers=self._build_hf_headers(

  .venv/lib/python3.10/site-packages/huggingface_hub/hf_api.py", line 8448, in _build_hf_headers
    return build_hf_headers(

  .venv/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 114, in _inner_fn
    return fn(*args, **kwargs)

  .venv/lib/python3.10/site-packages/huggingface_hub/utils/_headers.py", line 124, in build_hf_headers
    token_to_send = get_token_to_send(token)

  .venv/lib/python3.10/site-packages/huggingface_hub/utils/_headers.py", line 158, in get_token_to_send
    raise LocalTokenNotFoundError(

huggingface_hub.errors.LocalTokenNotFoundError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.

### Description

- I am trying to use `langchain_huggingface` library to connect to a TGI instance served locally. The problem is when wrapping a `HuggingFaceEndpoint` into `ChatHuggingFace`, it raises error requesting user token to be provided when it shouldn't be necessary a token when the model has already being downloaded and is serving locally.
- There is a similar issue #23872 but the fix they mentioned doesn't work because adding the `model_id` parameter to the `ChatHuggingFace` doesn't avoid falling in the following case:

```python
class ChatHuggingFace(BaseChatModel):
    """Hugging Face LLM's as ChatModels.
    ...
    """  # noqa: E501
    ...
    def __init__(self, **kwargs: Any):
        super().__init__(**kwargs)

        from transformers import AutoTokenizer  # type: ignore[import]

        self._resolve_model_id()  # ---> Even when providing the model_id it will enter here

        self.tokenizer = (
            AutoTokenizer.from_pretrained(self.model_id)
            if self.tokenizer is None
            else self.tokenizer
        )
    ...
    def _resolve_model_id(self) -> None:
        """Resolve the model_id from the LLM's inference_server_url"""

        from huggingface_hub import list_inference_endpoints  # type: ignore[import]

        if _is_huggingface_hub(self.llm) or (
            hasattr(self.llm, "repo_id") and self.llm.repo_id
        ):
            self.model_id = self.llm.repo_id
            return
        elif _is_huggingface_textgen_inference(self.llm):
            endpoint_url: Optional[str] = self.llm.inference_server_url
        elif _is_huggingface_pipeline(self.llm):
            self.model_id = self.llm.model_id
            return
        else: # This is the case we are in when _is_huggingface_endpoint() is True
            endpoint_url = self.llm.endpoint_url
        available_endpoints = list_inference_endpoints("*")  # ---> This line raises the error if we don't provide the hf token
        for endpoint in available_endpoints:
            if endpoint.url == endpoint_url:
                self.model_id = endpoint.repository

        if not self.model_id:
            raise ValueError(
                "Failed to resolve model_id:"
                f"Could not find model id for inference server: {endpoint_url}"
                "Make sure that your Hugging Face token has access to the endpoint."
            )
```

I was able to solve the issue by modifying the constructor method so when providing the `model_id` it doesn't resolve it:
```python
class ChatHuggingFace(BaseChatModel):
    """Hugging Face LLM's as ChatModels.
    ...
    """  # noqa: E501

    ...
    def __init__(self, **kwargs: Any):
        super().__init__(**kwargs)

        from transformers import AutoTokenizer  # type: ignore[import]

        self.model_id or self._resolve_model_id()  # ---> Not a good solution because if model_id is invalid then the tokenizer instantiation will fail only if the tokinizer is not provided and also won't check other hf_hub inference cases

        self.tokenizer = (
            AutoTokenizer.from_pretrained(self.model_id)
            if self.tokenizer is None
            else self.tokenizer
        )
```

I imagine there is a better way to solve this, for example by adding some logic to check if the `endpoint_url` is a valid ip to request or if it is served with TGI or simply by checking if it's localhost:
```python
class ChatHuggingFace(BaseChatModel):
    """Hugging Face LLM's as ChatModels.
    ...
    """  # noqa: E501

    ...
    def _resolve_model_id(self) -> None:
        """Resolve the model_id from the LLM's inference_server_url"""

        from huggingface_hub import list_inference_endpoints  # type: ignore[import]

        if _is_huggingface_hub(self.llm) or (
            hasattr(self.llm, "repo_id") and self.llm.repo_id
        ):
            self.model_id = self.llm.repo_id
            return
        elif _is_huggingface_textgen_inference(self.llm):
            endpoint_url: Optional[str] = self.llm.inference_server_url
        elif _is_huggingface_pipeline(self.llm):
            self.model_id = self.llm.model_id
            return
        elif _is_huggingface_endpoint(self.llm):  # ---> New case added to check url
            ...  # Take the following code with a grain of salt
            if is_tgi_hosted(self.llm.endpoint_url):
                if not self.model_id and not self.tokenizer:
                    raise ValueError("You must provide valid model id or a valid tokenizer")
                return
            ...
            endpoint_url = self.llm.endpoint_url
        else:  # ---> New last case in which no valid huggingface interface was provided
            raise TypeError("llm must be `HuggingFaceTextGenInference`, `HuggingFaceEndpoint`, `HuggingFaceHub`, or `HuggingFacePipeline`.") 
        available_endpoints = list_inference_endpoints("*")
        for endpoint in available_endpoints:
            if endpoint.url == endpoint_url:
                self.model_id = endpoint.repository

        if not self.model_id:
            raise ValueError(
                "Failed to resolve model_id:"
                f"Could not find model id for inference server: {endpoint_url}"
                "Make sure that your Hugging Face token has access to the endpoint."
            )
```


### System Info

System Information
------------------
> OS:  Linux
> OS Version:  #126-Ubuntu SMP Mon Jul 1 10:14:24 UTC 2024
> Python Version:  3.10.14 (main, Jul 18 2024, 23:22:54) [GCC 11.4.0]

Package Information
-------------------
> langchain_core: 0.2.22
> langchain: 0.2.10
> langchain_community: 0.2.9
> langsmith: 0.1.93
> langchain_google_community: 1.0.7
> langchain_huggingface: 0.0.3
> langchain_openai: 0.1.17
> langchain_text_splitters: 0.2.2