hiro-graph-client 5.3.3
Hiro Client for Graph REST API of HIRO 7
License: MIT
Requires Python: >=3.7
Keywords: arago HIRO7 API REST WebSocket

# HIRO Graph API Client

This is a client library to access data of the [HIRO Graph](#graph-client-hirograph).

This library also contains classes for handling the [WebSockets](#websockets) `event-ws` and `action-ws` API.

To process large batches of data, take a look at "hiro-batch-client". - (
PyPI: [hiro-batch-client](https://pypi.org/project/hiro-batch-client/),
GitHub: [hiro-batch-client-python](https://github.com/arago/hiro-batch-client-python))

For more information about HIRO Automation, look at https://www.almato.com/de/loesungen/hiro-ai.

For more information about the APIs this library covers, see https://dev-portal.engine.datagroup.de/7.0/.

Currently, implemented are

* `HiroApp` for `app`
* `HiroAuth` for `auth`
* `HiroGraph` for `graph`
* `HiroIam` for `iam`
* `HiroKi` for `ki`
* `HiroAuthz` for `authz`
* `HiroVariables` for `variables`

and for the websockets

* `AbstractEventWebSocketHandler` for `event-ws`
* `AbstractActionWebSocketHandler` for `action-ws`

## Quickstart

To use this library, you will need an account at https://desktop.engine.datagroup.de/ and access to an OAuth Client-Id and Client-Secret
to access the HIRO Graph. See also https://dev-portal.engine.datagroup.de/7.0/.

Most of the documentation is done in the sourcecode.

### HiroGraph Example

Example to use the straightforward graph api client:

```python
from hiro_graph_client import PasswordAuthTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=PasswordAuthTokenApiHandler(
        root_url="https://core.engine.datagroup.de",
        username='',
        password='',
        client_id='',
        client_secret=''
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)
```

## TokenApiHandler

Authorization against the HIRO Graph is done via tokens. These tokens are handled by classes of
type `AbstractTokenApiHandler` in this library. Each of the Hiro-Client-Object (`HiroGraph`, , `HiroApp`, etc.) need to
have some kind of TokenApiHandler at construction.

This TokenApiHandler is also responsible to determine the most up-to-date endpoints for the API calls. You can supply a
custom list of endpoints by using the dict parameter `custom_endpoints=` on construction.

A custom list of headers can also be set via the dict parameter `headers=` in the constructor. These would update the
internal headers. Header names can be supplied in any upper/lower-case.

This library supplies the following TokenApiHandlers:

---

### FixedTokenApiHandler

A simple TokenApiHandler that is generated with a preset-token at construction. Cannot update its token.

---

### EnvironmentTokenApiHandler

A TokenApiHandler that reads an environment variable (default is `HIRO_TOKEN`) from the runtime environment. Will only
update its token when the environment variable changes externally.

---

### PasswordAuthTokenApiHandler

This TokenApiHandler logs into the HiroAuth backend and obtains a token from login credentials. This is also the only
TokenApiHandler (so far) that automatically tries to renew a token from the backend when it has expired.

---

All code examples in this documentation can use these TokenApiHandlers interchangeably, depending on how such a token is
provided.

The HiroGraph example from above with another customized TokenApiHandler:

```python
from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.engine.datagroup.de"
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)
```

Example with additional parameters:

```python
from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.engine.datagroup.de",
        env_var='_TOKEN',
        headers={
            'X-Custom-Header': 'My custom value'
        },
        custom_endpoints={
            "graph": "/api/graph/7.2",
            "auth": "/api/auth/6.2"
        },
        client_name="HiroGraph (testing)"  # Will be used in the header 'User-Agent'
    )
)

# The commands of the Graph API are methods of the class HIROGraph.
# The next line executes a vertex query for instance. 
query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)
```

## Token Handler sharing

When you need to access multiple APIs of HIRO, share the TokenApiHandler between the API objects to avoid unnecessary
requests for token- and version-information against HIRO. The TokenApiHandler will share a `requests.Session`, token-
and version-request-handling, and the token string itself between them.

```python
from hiro_graph_client import HiroGraph, HiroApp, PasswordAuthTokenApiHandler

hiro_api_handler = PasswordAuthTokenApiHandler(
    root_url="https://core.engine.datagroup.de",
    username='',
    password='',
    client_id='',
    client_secret=''
)

hiro_client: HiroGraph = HiroGraph(
    api_handler=hiro_api_handler
)

hiro_app_client: HiroApp = HiroApp(
    api_handler=hiro_api_handler
)
```

## Connection sharing

You can also let TokenApiHandlers share a common connection session instead of letting each of them create their own.
This might prove useful in a multithreading environment where tokens have to be set externally or change often (i.e.
one token per user per thread). This also ensures, that version-requests happen only once when the connection is
initialized.

Use the parameters `pool_maxsize` and `pool_block` to further tune the connection parameters for parallel access to 
the backend. See [requests Session Objects](https://docs.python-requests.org/en/latest/user/advanced/#session-objects)
and the Python documentation of `requests.adapters.HTTPAdapter` and `GraphConnectionHandler`.

```python
from hiro_graph_client import HiroGraph, HiroApp, FixedTokenApiHandler, GraphConnectionHandler

connection_handler = GraphConnectionHandler(
    root_url="https://core.engine.datagroup.de",
    pool_maxsize=200,                     # Optional: Max pool of cached connections for this connection session
    pool_block=True,                      # Optional: Do not allow more parallel connections than pool_maxsize
    client_name="Your Graph Client 0.0.1" # Optional: Will be used in the header 'User-Agent'
)

# Work with token of user 1

user1_client: HiroGraph = HiroGraph(
    api_handler=FixedTokenApiHandler(
        connection_handler=connection_handler,
        token='token user 1'
    )
)

# Work with token of user 2 (Shared Token Handler)

user2_api_handler = FixedTokenApiHandler(
    connection_handler=connection_handler,
    token='token user 2'
)

user2_client: HiroGraph = HiroGraph(
    api_handler=user2_api_handler
)

hiro_app_client: HiroApp = HiroApp(
    api_handler=user2_api_handler
)

```

Everything written in [Token Handler Sharing](#token-handler-sharing) still applies.

## SSL Configuration

SSL parameters are configured using the class `SSLConfig`. This class translates the parameters given to the required
fields for the `requests` library of Python (parameters `cert` and `verify` there). This configuration is given to the
TokenApiHandlers and will be used by the clients attached to it as well.

If this is not set, the default settings of the library `requests` will be used, which is to verify any server
certificates by using system defaults.

#### Example: Disable verification

```python
from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph, SSLConfig

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.engine.datagroup.de",
        # Disable any verification.
        ssl_config=SSLConfig(verify=False)
    )
)

query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)
```

#### Example: Set custom SSL certificates

```python
from hiro_graph_client import EnvironmentTokenApiHandler, HiroGraph, SSLConfig

hiro_client: HiroGraph = HiroGraph(
    api_handler=EnvironmentTokenApiHandler(
        root_url="https://core.engine.datagroup.de",
        # Set custom certification files. If any of them are omitted, system defaults will be used.
        ssl_config=SSLConfig(
            cert_file="<path to client certificate file>",
            key_file="<path to key file for the client certificate>",
            ca_bundle_file="<path to the ca_bundle to verify the server certificate>"
        )
    )
)

query_result = hiro_client.query('ogit\\/_type:"ogit/MARS/Machine"')

print(query_result)
```

## Graph Client "HiroGraph"

The Graph Client is mostly straightforward to use, since all public methods of this class represent an API call in the
[Graph API](https://core.engine.datagroup.de/help/specs/?url=definitions/graph.yaml). Documentation is available in source code as
well. Some calls are a bit more complicated though and explained in more detail below:

### Attachments

To upload data to such a vertex, use `HiroGraph.post_attachment(data=...)`. The parameter `data=` will be given directly
to the call of the Python library `requests` as  `requests.post(data=...)`. To stream data, set `data` to an object of
type `IO`. See the documentation of the Python library `requests` for more details.

Downloading an attachment is done in chunks, so huge blobs of data in memory can be avoided when streaming this data.
Each chunk is 64k by default.

To stream such an attachment to a file, see the example below:

```python
ogit_id = '<ogit/_id of a vertex>'
data_iter = hiro_client.get_attachment(ogit_id)

with io.start("attachment.bin", "wb") as file:
    for chunk in data_iter:
        file.write(chunk)
```

To read the complete data in memory, see this example:

```python
ogit_id = '<ogit/_id of a vertex>'
data_iter = hiro_client.get_attachment(ogit_id)

attachment = b''.join(data_iter)
```

## WebSockets

This library contains classes that make using HIRO WebSocket protocols easier. They handle authentication, exceptions
and much more.

The classes do not handle buffering of messages, so it is the duty of the programmer to ensure, that incoming messages
are either handled quickly or being buffered to avoid clogging the websocket. The classes are thread-safe, so it is
possible to handle each incoming message asynchronously in its own thread and have those threads send results back if
needed (See multithreaded example in [Action WebSocket](#action-websocket)).

### Closing WebSockets

To shut these WebSockets (_ws_) down cleanly, please consider these scenarios:

#### Default behaviour

The library reacts on *KeyboardInterrupt* (SIGINT) and closes the WebSocket cleanly with closing message to the sever
when such an interrupt is received. If another signal (like SIGTERM) is received, the program will stop immediately
without any closing message back to the server.

#### Signal handling

When installing signal handlers, you need to use *ws.signal_stop()* to shut the WebSocket down. Do NOT use
*ws.stop()* or the closing process will deadlock. This is because the signal interrupt is executed in the same thread
as *ws.run_forever()*.

Example:

```python
import signal

[...]

with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    def signal_handler(signum, handler):
        ws.signal_stop()


    signal.signal(signal.SIGINT, signal_handler)
    signal.signal(signal.SIGTERM, signal_handler)
    signal.signal(signal.SIGHUP, signal_handler)  # This signal might not be available on MS Windows 

    ws.run_forever()
```

#### Closing from a separate thread

When closing the WebSocket from another thread than the one running *ws.run_forever()*, you should use *ws.stop()*. This
ensures, that the shutdown is synchronized and *ws.stop()* will return after the WebSocket has been closed.

### Event WebSocket

This websocket receives notifications about changes to vertices that match a certain filter.

See also [API description of event-ws](https://core.engine.datagroup.de/help/specs/?url=definitions/events-ws.yaml)

Example:

```python
import threading

from hiro_graph_client.clientlib import FixedTokenApiHandler
from hiro_graph_client.eventswebsocket import AbstractEventsWebSocketHandler, EventMessage, EventsFilter


class EventsWebSocket(AbstractEventsWebSocketHandler):

    def on_create(self, message: EventMessage):
        """ Vertex has been created """
        print("Create:\n" + str(message))

    def on_update(self, message: EventMessage):
        """ Vertex has been updated """
        print("Update:\n" + str(message))

    def on_delete(self, message: EventMessage):
        """ Vertex has been removed """
        print("Delete:\n" + str(message))


events_filter = EventsFilter(filter_id='testfilter', filter_content="(element.ogit/_type=ogit/MARS/Machine)")

with EventsWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN'),
                     events_filters=[events_filter],
                     query_params={"allscopes": "false", "delta": "false"}) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

```

If you do not set the parameter `scope=`, the default scope of your account will be used. If you need to set the scope
by hand, use the following:

```python
[...]

api_handler = FixedTokenApiHandler('HIRO_TOKEN')

default_scope = api_handler.decode_token()['data']['default-scope']

with EventsWebSocket(api_handler=api_handler,
                     events_filters=[events_filter],
                     scopes=[default_scope],
                     query_params={"allscopes": "false", "delta": "false"}) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

```

### Action WebSocket

This websocket receives notifications about actions that have been triggered within a KI. Use this to write your own
custom action handler.

See also [API description of action-ws](https://core.engine.datagroup.de/help/specs/?url=definitions/action-ws.yaml)

Simple example:

```python
import threading

from hiro_graph_client.actionwebsocket import AbstractActionWebSocketHandler
from hiro_graph_client.clientlib import FixedTokenApiHandler


class ActionWebSocket(AbstractActionWebSocketHandler):

    def on_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Message *submitAction* has been received """

        # Handle the message
        print(f"ID: {action_id}, Capability: {capability}, Parameters: {str(parameters)}")

        # Send back message *sendActionResult*
        self.send_action_result(action_id, "Everything went fine.")

    def on_config_changed(self):
        """ The configuration of the ActionHandler has changed """
        pass


with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

```

Multithreading example using a thread executor:

```python
import threading
import concurrent.futures

from hiro_graph_client.actionwebsocket import AbstractActionWebSocketHandler
from hiro_graph_client.clientlib import FixedTokenApiHandler, AbstractTokenApiHandler


class ActionWebSocket(AbstractActionWebSocketHandler):

    def __init__(self, api_handler: AbstractTokenApiHandler):
        """ Initialize properties """
        super().__init__(api_handler)
        self._executor = None

    def start(self) -> None:
        """ Initialize the executor """
        super().start()
        self._executor = concurrent.futures.ThreadPoolExecutor()

    def stop(self, timeout: int = None) -> None:
        """ Shut the executor down """
        if self._executor:
            self._executor.shutdown()
        self._executor = None
        super().stop(timeout)

    def handle_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Runs asynchronously in its own thread. """
        print(f"ID: {action_id}, Capability: {capability}, Parameters: {str(parameters)}")
        self.send_action_result(action_id, "Everything went fine.")

    def on_submit_action(self, action_id: str, capability: str, parameters: dict):
        """ Message *submitAction* has been received. Message is handled in thread executor. """
        if not self._executor:
            raise RuntimeError('ActionWebSocket has not been started.')
        self._executor.submit(ActionWebSocket.handle_submit_action, self, action_id, capability, parameters)

    def on_config_changed(self):
        """ The configuration of the ActionHandler has changed """
        pass


with ActionWebSocket(api_handler=FixedTokenApiHandler('HIRO_TOKEN')) as ws:
    ws.run_forever()  # Use KeyboardInterrupt (Ctrl-C) to exit. 

```

---
# CHANGELOG
---
# v5.3.2

* Point to https://core.engine.datagroup.de instead of https://core.arago.co.
* Add `HiroGraph.escaped_query` method.

# v5.3.0

* Decode any token via static `decode_token_ext()`.

Changes for /api/auth/6.6:

* Ability to revoke a token.
* Recognize `access_token` and `_TOKEN` in token results.
* Remove obsolete method `fresh()` from class TokenInfo because each token refresh issues a new refresh_token by default
  now.

# v5.2.5

* [bugfix] Calculation within *clientlib.TokenInfo.expired()* now returns *False* as expected when no *expires_at* 
  is available for a token. Previous versions would raise an exception.

# v5.2.4

* Merge headers in AbstractAPI instead of replacing them.

# v5.2.3

* Remove accept_all_certs for good. Directly use the flag in SSLConfig.

# v5.2.2

* Debugged python code documentation.
* Fixed copyright notices.

# v5.2.1

* Update because of wrong documentation on https://pypi.org/project/hiro-graph-client/

# v5.2.0

* Separate connection sessions from token handlers. Introducing class `GraphConnectonHandler` which can be shared
  between `TokenApiHandler`s and API-Classes.
* Updated documentation.

# v5.1.0

* Use connection pooling via requests.Session.
    * Set `pool_connections=1`, `pool_maxsize=10` by default (the latter can be changed).
    * Option `pool_block` to block any connections that would exceed the `pool_maxsize`.
* Removed obsolete `accept_all_certs()`.
* Fixed pytest in Makefile target.
* Added information about `requests.Session` and the `pool_maxsize` to documentation.

# v5.0.0

* Externalize `batchclient.py` as its own project at https://github.com/arago/hiro-batch-client-python.

# v4.9.0

* Added batch handling of issues. `handle_vertices_combined` recognizes `_issue_data` now for issues directly linked to
  vertices.
* Updated documentation.

# v4.8.0

* Refactoring of `websocketlib.py`:

    * Removed unnecessary extra thread.
    * Debugged shutdown of WebSockets via signals and threads.
    * Updated documentation:
        * On signal handling.
        * On clean shutdown of WebSockets.
        * Adjusted the example code for WebSockets.


* Allow `-` or `_` as build number separator on test tags, i.e. `t4.8.0_0` and `t4.8.0-0` are handled identical now.

# v4.7.5

* [budgfix] Handle `self.submitStore` and `self.resultStore` properly on shutdown in AbstractActionWebSocketHandler.
  They might not have been initialized at all when startup fails.

# v4.7.4

* [bugfix] Catch 401 when using PasswordAuthTokenHandler and refresh_token. Raise TokenUnauthorizedError when this
  happens to trigger a retry.

# v4.7.3

* Refactoring of `HiroGraphBatch`. Removed obsolete methods.

# v4.7.2

* Fixed bad call of patch() methods in IAM API.

# v4.7.1

* Updated some code documentation in `HiroGraph` that became obsolete.

* Updated graph queries in `HiroGraph`:

    * For vertex and gremlin queries: Skip unnecessary fields in query payload.
    * Add parameter `count` to vertex query.

* Renamed default _client_name to `hiro-graph-client` (name of the lib on PyPI).

# v4.7.0

* Content- / Media-Type handling

    * Throw `WrongContentTypeError` when an unexpected (or missing) Content-Type is returned.
    * Check for Content-Type of error results for constructing HTTPError.

* Error handling

    * Separate error message extraction, so it can be overwritten per API.
    * Try to handle different message formats by guessing, where their error message might be.
    * Intercept special error messages for API `ki`.
    * Flag `log_communication_on_error` to enable logging of communication when an error is detected in the response.

* Add parameter `max_tries` to TokenHandlers.

# v4.6.2

* Added documentation of new APIs

# v4.6.1

* Updated CHANGELOG.md

# v4.6.0

* Added the following APIS:
    * KI
    * AuthZ
    * Variables
* Adjust return value typing with API methods that return lists of dicts.

# v4.5.2

BUGFIX

* Previous versions incorrectly translated True and False to "True" and
  "False" whereas "true" and "false" (lowercase) are needed in URL queries. Fixed.

# v4.5.1

* Add `decode_token()` to decode the information embedded in a HIRO token.

# v4.5.0

* GitHub repository got renamed from `python-hiro-clients` to `hiro-client-python`. No other technical changes.

# v4.4.0

* Added IAM client
* Updated Graph client and Auth client
* put_binary is allowed to return simple strings now. (i.e. for avatar image updates).

# v4.3.0

* Adding SSL configuration

# v4.2.14

* Removed bug with reversed operator in websocketlib.
* Updated installation instructions in README.md.

# v4.2.13

* You need to explicitly set `query_params={'allscopes': 'true'}` if you want to enable it for EventWebSockets. If this
  is left out of the query_params, it will be added as 'allscopes': 'false'.

# v4.2.12

* Use typing to make sure, that `query_params=` for WebSockets is of type `Dict[str, str]`.
* Set `query_params={'allscopes': 'false'}` as default for the EventWebSocket.

# v4.2.11

* Debugged EventWebSocket handling.
* Abort connection when setting scope or filters failed.

# v4.2.10

* Adding scopes to EventWebSockets.

# v4.2.9

* Documentation of feature in v4.2.8

# v4.2.8

* WebSockets have new option `query_params` to add arbitrary query parameters to the initial websocket request.

# v4.2.7

Changes to `AbstractAuthenticatedWebSocketHandler`:

* Introducing `run_forever()` which will return after the reader thread has been joined. This can happen when another
  thread calls `stop()` (normal exit) or an internal error occurs, either directly when the connection is attempted, a
  token got invalid and could not be refreshed, or any other exception that has been thrown in the internal reader
  thread.

  This ensures, that the main thread does not continue when the websocket reader thread is not there.

* Enable parallel executions of `send()` across multiple threads.

* Make sure, that only one thread triggers a restart by a call to `restart()`.

* Check for active websocket reader thread via `is_active()`.

* Update examples for websockets in README.md.

Generic

* Update README.md to show usage of `client_name`.

# v4.2.6

* Do not require package uuid - it is already supplied with python

# v4.2.5

* Send valid close messages to backend.
* Introduced parameter `client_name` to give connections a name and also set header `User-Agent` more easily.

# v4.2.4

* Updated CHANGELOG.md.

# v4.2.3

* Hardening of clientlib. Removed some None-Value-Errors.

# v4.2.2

* Introduce parameter `remote_exit_codes` to `AbstractAuthenticatedWebSocketHandler`.

# v4.2.1

* Avoid blocking thread in `_backoff()` by not using `sleep()` but `threading.Condition.wait()`.

# v4.2.0

* Implement websocket protocols
    * event-ws
    * action-ws

# v4.1.3

* Use yield from instead of return

# v4.1.2

* Removed a bug with double yields on binary data

# v4.1.1

* Only log request/responses when logging.DEBUG is enabled

# v4.1.0

* Added timeseries handling to command `handle_vertices_combined`

# v4.0.0

* `AbstractTokenApiHandler`
    * Better token handling.
    * Resolve graph api endpoints via calling /api/version.
        * Ability to customize headers. Headers are handled case-insensitively and are submitted to requests
          capitalized.
        * Ability to override internal endpoints.


* AbstractIOCarrier works with `with` statements now.
* Added `BasicFileIOCarrier`.


* Removed `ApiConfig`.
* Renamed several internal classes.
* Better error messages.
* HTTP secure protocol logging.
* Fixed timestamp creation for tokens.


* Joe's suggestions - thanks Joe!

# v3.1.0

* Separation of APIs in client libraries. Currently, supported APIs are:
    * HiroGraph: https://core.arago.co/help/specs/?url=definitions/graph.yaml
    * HiroAuth: https://core.arago.co/help/specs/?url=definitions/auth.yaml
    * HiroApp: https://core.arago.co/help/specs/?url=definitions/app.yaml
* Use correct headers with binary transfers.
* Added gremlin and multi-id queries to HiroGraph.

# v3.0.0

* Renamed classes to match documentation elsewhere (i.e. Graphit -> HiroGraph, GraphitBatch -> HiroGraphBatch).
* Catch token expired error when refresh_token has expired.
* Documentation with examples

# v2.4.2

Added VERSION to package_data in setup.py

# v2.4.1

Added documentation for PyPI

# v2.4.0

Initial release after split from https://github.com/arago/hiro-clients