furhat-realtime-api 0.1.3
A Python client for the Furhat Realtime API.
Requires Python: >=3.9
Keywords: api, client, furhat, robotics, websocket

# Python Furhat Realtime API Client

Python implementation of Furhat Realtime API client. 

There are two different clients:

- `FurhatClient`: A synchronous (blocking) client for simple, sequential command execution.
- `AsyncFurhatClient`: An asynchronous client for advanced, concurrent, or event-driven interactions.

## Installation

Install the module with pip:

```
pip install furhat-realtime-api
```

Then include it your code:

```python
from furhat_realtime_api import *
```

## Synchronous Event client

The synchronous client (`FurhatClient`) is a wrapper around the asynchronous client that allows for blocking calls in a standard (non-async) Python context. This is useful for simple use cases where you want to send one command at a time and wait for the result before proceeding. Note that it is not suitable for advanced scenarios requiring concurrent or overlapping commands.

```python
from furhat_realtime_api import FurhatClient
import logging

furhat = FurhatClient("127.0.0.1")  # Add authentication key as second argument if needed
furhat.set_logging_level(logging.INFO)  # Use logging.DEBUG for more details
furhat.connect()
furhat.request_speak_text("Hello world, I am Furhat.")
furhat.disconnect()
```

## Asyncronous Event client

The asynchronous client is based on the Python [asyncio](https://docs.python.org/3/library/asyncio.html) library. This allows for much more flexible use cases, where requests can be aborted and responses can be streamed while actions are being executed. 

```python
from furhat_realtime_api import AsyncFurhatClient
import asyncio
import logging

furhat = AsyncFurhatClient("127.0.0.1")

async def run_example():
    await furhat.connect()
    await furhat.request_speak_text("Hello world, I am Furhat.", wait=True)
    await furhat.disconnect()

asyncio.run(run_example())
```

Note that one important difference when making requests in the asynchronous API is that they need to be called with the `await` keyword.

If you want to mix the use of the synchronous and asynchronous clients, you should not instantiate both, but you can access it like this:

```python
from furhat_realtime_api import FurhatClient
import logging

furhat = FurhatClient("127.0.0.1")  
furhat_async = furhat.async_client
furhat.connect()

async def run_example():
    await furhat_async.request_speak_text("Hello world, I am Furhat.", wait=True)

asyncio.run(run_example())
```

### Reacting to events

With the asynchronous client, you can react to events from the robot. This allows your application to react to specific events from the the robot, such as when the robot starts or stops speaking, hears something, or detects a user.

Here is an example of how to react to events related to the robot speaking:

```python
from furhat_realtime_api import AsyncFurhatClient, Events
import asyncio

furhat = AsyncFurhatClient("127.0.0.1")

async def on_speak_start(event):
    print("Furhat started speaking:", event)

async def on_speak_end(event):
    print("Furhat finished speaking:", event)

async def main():
    await furhat.connect()
    # Register handlers
    furhat.add_handler(Events.response_speak_start, on_speak_start)
    furhat.add_handler(Events.response_speak_end, on_speak_end)
    
    await furhat.request_speak_text("Hello! Please say something.")
    await furhat.disconnect()

asyncio.run(main())
```

## API reference

Here is a summary of the main methods. All of them are available in the asynchronous API, but only some of them in the synchronous wrapper API. 

`connect()` / `disconnect()`

* Establishes or closes the connection to the robot.

### Speaking (text-to-speech)

`request_speak_text(text: str, wait: bool, abort: bool)`

* Sync and Async: Makes the robot speak the given text. If `wait` is True, the method blocks until speaking is finished. If `abort` is True, any ongoing speech is stopped before starting the new one.

`request_speak_audio(url: str, wait: bool, abort: bool, text: str, lipsync: bool = True)`

* Sync and Async: Makes the robot play the given audio (from url). Same `wait` and `abort` logic as `speak_text`. `text` is just for logging purposes and can be omitted. `lipsync` denotes whether automatic lipsync should be added.

`request_speak_stop()`

* Sync and Async: Immediately stops any ongoing speech or audio playback.

#### Event callbacks (Async only)

`response_speak_start`

* The robot started to speak
* Parameters:
  - `text` (string): The synthesized text
  - `gen_time` (number): The time it took to synthesize (in ms)

`response_speak_end` 

* The robot stopped speaking
* Parameters:
  - `text` (string): The synthesized text (what was actually said if aborted)
  - `aborted` (boolean, optional): Whether the speech was stopped prematurely (default: false)
  - `failed` (boolean, optional): Whether the speech synthesis failed (default: false)

`response_speak_word`

* Sent while speaking, if monitor_words is enabled in the speak request
* Parameters:
  - `word` (string): The word spoken
  - `index` (number): The index of the word spoken

`response_speak_audio_buffer`

* Audio buffer status update during streaming audio playback
* Parameters:
  - `played` (int): The number of bytes of audio data played
  - `received` (int): The number of bytes of audio data received

### Speaking (streaming audio to the robot)

You can send audio to the robot which will be played through the robot's speakers. You can choose whether to add automatic lipsync or not. 

`request_speak_audio_start(sample_rate: int = 24000, lipsync: bool = False)`

* Async only: Start streaming audio data to the robot for real-time speech synthesis.

`request_speak_audio_data(audio: str)`

* Async only: Send audio data chunk to the robot (used with `request_speak_audio_start`).

`request_speak_audio_end()`
 
* Async only: End the audio streaming session started with `request_speak_audio_start`.

### Listening (speech-to-text)

`request_listen_config(languages: list = ["en-US"], phrases: list = None)`

* Sync and Async: Configure speech recognition languages and optional phrase hints for better recognition.

`request_listen_start(partial: bool = False, concat: bool = True, stop_no_speech: bool = True, stop_robot_start: bool = True, stop_user_end: bool = True, resume_robot_end: bool = False, no_speech_timeout: float = 8.0, end_speech_timeout: float = 1.0)`

* Starts listening for user speech with various configuration options for when to start/stop and timeout behavior.
* Sync: Will return the result as a string
* Async: You need to add event callbacks to get results

`request_listen_stop()`

* Async only: Force the robot to stop listening for speech.

#### Event callbacks (Async only)

You can add the following events handlers to receive results and monitor the speaking activity:

`response_listen_start`

* The robot started to listen

`response_listen_end` 

* The robot stopped listening
* Parameters:
  - `cause` (string): The reason for stopping, can be either 'stopped', 'robot_speak', 'speech_end', 'silence_timeout'

`response_hear_start`

* The robot started to hear speech

`response_hear_end` 

* The robot stopped hearing speech, with a speech recognition result
* Parameters:
  - `text` (string): Recognized speech

`response_hear_partial`

* The robot detected a partial speech recognition result
* Parameters:
  - `text` (string): Recognized speech

### Voice  

`request_voice_status()`

* Sync and Async: Returns the current voice configuration and status of the robot.

`request_voice_config(voice_id: str, language: str, provider: str, gender: str, name: str, input_language: bool = True)`

* Sync and Async: Configure the robot's voice according to the parameters (one or several can be used to add constraints).

### Attention

`request_attend_user(user_id: str)`

* Sync and Async: Makes the robot turn its attention to a specific user (by user ID). Use the string `closest` to attend to the closest user. 

`request_attend_location(x: float, y: float, z: float)`

* Sync and Async: Makes the robot attend to a specific location.

### Gestures 

`request_gesture_start(name: str, intensity: float = 1.0, duration: float = 1.0, wait: bool = False)`

* Sync and Async: Makes the robot perform a gesture. You can control the intensity and duration. If `wait` is True, the method blocks until the gesture is finished.

#### Event callbacks (Async only)

`response_gesture_start`

* The robot started performing a gesture

`response_gesture_end`

* The robot finished performing a gesture

### Face  

`request_face_params(params: dict)`

* Sync and Async: Sets facial animation parameters directly. The `params` dict should specify the desired facial features and their values.

`request_face_headpose(yaw: float, pitch: float, roll: float, relative: bool)`

* Sync and Async: Directly controls the head pose of the robot. Set `relative` to True for movement relative to current gaze (attention) target, or False for absolute positioning.

`request_face_config(face_id: str = None, visibility: bool = None, microexpressions: bool = None)`

* Sync and Async: Sets the current mask and character (face), and/or face visibility and microexpressions.

`request_face_status(face_id: bool = True, face_list: bool = True)`

* Sync and Async: Gets the current and available masks and characters. Returns information about the face configuration.

`request_face_reset()`

* Sync and Async: Resets all facial parameters to their default values.

### LED  

`request_led_set(color: str)`

* Sync and Async: Sets the color of the robot's LED. The `color` can be a color name (e.g., "red") or a hex code (e.g., "#FF0000").

### Users

`request_users_once()`

* Sync and Async: Get the current user status and detected users as a one-time request. Returns the user object.

`request_users_start()`
  
* Async only: Start continuous monitoring of users in the robot's field of view. You must register an event callback to receive the user objects.

`request_users_stop()`

* Async only: Stop the continuous user monitoring started with `request_users_start`.

#### Event callbacks (Async only)

`response_users_data`

* Provides continuous updates about detected users in the robot's field of view, including user positions, IDs, and status information
* Parameters:
  - `users` (list): List of user objects, sorted by how close they are to the robot

### Audio Streaming

`request_audio_start(sample_rate: int = 16000, microphone: bool = True, speaker: bool = False)`

* Async only: Start streaming audio data from the robot's microphone and/or to the robot's speaker. You must register an event callback to receive the audio.

`request_audio_stop()`

* Async only: Stop the audio streaming session started with `request_audio_start`.

#### Event callbacks (Async only)

`response_audio_data`

* Provides streaming audio data from the robot's microphone when audio streaming is active
* Parameters:
  - `microphone` (string, optional): Base64-encoded audio data, 16-bit, mono, little-endian
  - `speaker` (string, optional): Base64-encoded audio data, 16-bit, mono, little-endian

### Camera

`request_camera_once()`

* Sync and Async: Get a one-time snapshot from the robot's camera. Returns the image data.

`request_camera_start()`

* Async only: Start streaming camera frames from the robot's camera. You must register an event callback to receive the images.

`request_camera_stop()`

* Async only: Stop the camera streaming started with `request_camera_start`.

#### Event callbacks (Async only)

`response_camera_data`

* Provides streaming camera frames from the robot's camera when camera streaming is active
* Parameters:
  - `image` (string): Base64-encoded jpeg image

### Utility Methods

`set_logging_level(level: int)`

* Sync and Async: Set the logging level for the client (e.g., `logging.DEBUG`, `logging.INFO`).

`add_handler(event_type: str | list[str], handler: Callable)`

* Async only: Register an event handler for one or more event types.

`remove_handler(event_type: str | list[str], handler: Callable)`

* Async only: Remove a previously registered event handler.

`wait_for_event(event_type: str | list[str], timeout: float = 5.0, request_id: str = None)`

* Async only: Wait for a specific event type and return its data.

`send_event(event: Dict)`

* Async only: Send a raw event dictionary to the websocket.

`send_event_and_wait(event: Dict, return_type: str | list[str], timeout: float = 5.0)`

* Async only: Send an event and wait for a specific response type.