Repository: nomic-ai/gpt4all
File: gpt4all-bindings/python/gpt4all/pyllmodel.py (Python)

Code before:

        def run_llmodel_prompt(prompt: str, callback: ResponseCallbackType, **kwargs):
            self.prompt_model(prompt, callback, **kwargs)
            output_queue.put(TERMINATING_SYMBOL)

        # Kick off llmodel_prompt in separate thread so we can return generator
        # immediately
        thread = threading.Thread(
            target=run_llmodel_prompt,
            args=(
                prompt, 
                _generator_callback_wrapper(callback)
            ),
            kwargs=kwargs,
        )
        thread.start()

        # Generator
        while True:
            response = output_queue.get()
            if response is TERMINATING_SYMBOL:
                break
            yield response

    def _callback_decoder(self, callback: RawResponseCallbackType) -> ResponseCallbackType:
        def _callback(token_id: int, response: str) -> bool:
            nonlocal callback
            return callback(token_id, response.decode("utf-8", "replace"))

        return _callback

    # Empty prompt callback
    @staticmethod
    def _prompt_callback(token_id: int) -> bool:
        return True

    # Empty recalculate callback
    @staticmethod
    def _recalculate_callback(is_recalculating: bool) -> bool:
        return is_recalculating


Review comment (suggestion):
Looks like there's a small mistake here. It should be:
```py
    def _callback_decoder(self, callback: ResponseCallbackType) -> RawResponseCallbackType:
        def _callback(token_id: int, response: bytes) -> bool:
            nonlocal callback
            return callback(token_id, response.decode("utf-8", "replace"))

        return _callback
```
Note the types and `response: bytes`.

The `_callback` is what gets passed into `ResponseCallback` and must have `bytes` as second argument. Consequently, it has the `RawResponseCallbackType`.

In my previous comment, I was thinking about renaming this one, first of all. Because it does the decoding.

Then it runs the `callback`, which gets the decoded bytes as second argument, i.e. a string. So the `callback` is of type `ResponseCallbackType`.

When in doubt, pip-install and run `mypy` over it. (Yes, it doesn't look intuitive.)
<br>

I think otherwise the PR looks great now! (Although I hope I haven't missed anything.)
👌

Code after:
            ResponseCallback(self._callback_decoder(callback)),
            RecalculateCallback(self._recalculate_callback),
            self.context,
        )

    def prompt_model_streaming(
        self,
        prompt: str,
        callback: ResponseCallbackType = empty_response_callback,
        **kwargs
    ) -> Iterable[str]:
        # Symbol to terminate from generator
        TERMINATING_SYMBOL = object()

        output_queue = queue.Queue()

        # Put response tokens into an output queue
        def _generator_callback_wrapper(callback: ResponseCallbackType) -> ResponseCallbackType:
            def _generator_callback(token_id: int, response: str):
                nonlocal callback

                if callback(token_id, response):
                    output_queue.put(response)
                    return True

                return False

            return _generator_callback

        def run_llmodel_prompt(prompt: str, callback: ResponseCallbackType, **kwargs):
            self.prompt_model(prompt, callback, **kwargs)
            output_queue.put(TERMINATING_SYMBOL)

        # Kick off llmodel_prompt in separate thread so we can return generator
        # immediately
        thread = threading.Thread(
            target=run_llmodel_prompt,
            args=(
                prompt, 
                _generator_callback_wrapper(callback)
            ),
            kwargs=kwargs,
        )
        thread.start()

        # Generator
        while True:
            response = output_queue.get()
            if response is TERMINATING_SYMBOL:
                break
            yield response