tigerbeetle 0.16.73
The TigerBeetle client for Python.
Requires Python: >=3.7

<!-- This file is generated by [/src/scripts/client_readmes.zig](/src/scripts/client_readmes.zig). -->
# tigerbeetle-python

The TigerBeetle client for Python.

## Prerequisites

Linux >= 5.6 is the only production environment we
support. But for ease of development we also support macOS and Windows.
* Python (or PyPy, etc) >= `3.7`

## Setup

First, create a directory for your project and `cd` into the directory.

Then, install the TigerBeetle client:

```console
pip install tigerbeetle
```

Now, create `main.py` and copy this into it:

```python
import os

import tigerbeetle as tb

print("Import OK!")

# To enable debug logging, via Python's built in logging module:
# logging.basicConfig(level=logging.DEBUG)
# tb.configure_logging(debug=True)
```

Finally, build and run:

```console
python3 main.py
```

Now that all prerequisites and dependencies are correctly set
up, let's dig into using TigerBeetle.

## Sample projects

This document is primarily a reference guide to
the client. Below are various sample projects demonstrating
features of TigerBeetle.

* [Basic](/src/clients/python/samples/basic/): Create two accounts and transfer an amount between them.
* [Two-Phase Transfer](/src/clients/python/samples/two-phase/): Create two accounts and start a pending transfer between
them, then post the transfer.
* [Many Two-Phase Transfers](/src/clients/python/samples/two-phase-many/): Create two accounts and start a number of pending transfers
between them, posting and voiding alternating transfers.
## Creating a Client

A client is created with a cluster ID and replica
addresses for all replicas in the cluster. The cluster
ID and replica addresses are both chosen by the system that
starts the TigerBeetle cluster.

Clients are thread-safe and a single instance should be shared
between multiple concurrent tasks. This allows events to be
[automatically batched](https://docs.tigerbeetle.com/coding/requests/#batching-events).

Multiple clients are useful when connecting to more than
one TigerBeetle cluster.

In this example the cluster ID is `0` and there is one
replica. The address is read from the `TB_ADDRESS`
environment variable and defaults to port `3000`.

```python
with tb.ClientSync(cluster_id=0, replica_addresses=os.getenv("TB_ADDRESS", "3000")) as client:
    # Use the client.
    pass

# Alternatively:
async with tb.ClientAsync(cluster_id=0, replica_addresses=os.getenv("TB_ADDRESS", "3000")) as client:
    # Use the client, async!
    pass
```

The following are valid addresses:
* `3000` (interpreted as `127.0.0.1:3000`)
* `127.0.0.1:3000` (interpreted as `127.0.0.1:3000`)
* `127.0.0.1` (interpreted as `127.0.0.1:3001`, `3001` is the default port)

## Creating Accounts

See details for account fields in the [Accounts
reference](https://docs.tigerbeetle.com/reference/account).

```python
account = tb.Account(
    id=tb.id(), # TigerBeetle time-based ID.
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=718,
    flags=0,
    timestamp=0,
)

account_errors = client.create_accounts([account])
# Error handling omitted.
```

See details for the recommended ID scheme in
[time-based identifiers](https://docs.tigerbeetle.com/coding/data-modeling#tigerbeetle-time-based-identifiers-recommended).

### Account Flags

The account flags value is a bitfield. See details for
these flags in the [Accounts
reference](https://docs.tigerbeetle.com/reference/account#flags).

To toggle behavior for an account, combine enum values stored in the
`AccountFlags` object (it's an `enum.IntFlag`) with bitwise-or:

* `AccountFlags.linked`
* `AccountFlags.debits_must_not_exceed_credits`
* `AccountFlags.credits_must_not_exceed_credits`
* `AccountFlags.history`


For example, to link two accounts where the first account
additionally has the `debits_must_not_exceed_credits` constraint:

```python
account0 = tb.Account(
    id=100,
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=1,
    timestamp=0,
    flags=tb.AccountFlags.LINKED | tb.AccountFlags.DEBITS_MUST_NOT_EXCEED_CREDITS,
)
account1 = tb.Account(
    id=101,
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=1,
    timestamp=0,
    flags=tb.AccountFlags.HISTORY,
)

account_errors = client.create_accounts([account0, account1])
# Error handling omitted.
```

### Response and Errors

The response is an empty array if all accounts were
created successfully. If the response is non-empty, each
object in the response array contains error information
for an account that failed. The error object contains an
error code and the index of the account in the request
batch.

See all error conditions in the [create_accounts
reference](https://docs.tigerbeetle.com/reference/requests/create_accounts).

```python
account0 = tb.Account(
    id=102,
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=1,
    timestamp=0,
    flags=0,
)
account1 = tb.Account(
    id=103,
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=1,
    timestamp=0,
    flags=0,
)
account2 = tb.Account(
    id=104,
    debits_pending=0,
    debits_posted=0,
    credits_pending=0,
    credits_posted=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    ledger=1,
    code=1,
    timestamp=0,
    flags=0,
)

account_errors = client.create_accounts([account0, account1, account2])
for error in account_errors:
    if error.result == tb.CreateAccountResult.EXISTS:
        print(f"Batch account at {error.index} already exists.")
    else:
        print(f"Batch account at ${error.index} failed to create: {error.result}.")
```

To handle errors you can compare the result code returned
from `client.create_accounts` with enum values in the
`CreateAccountResult` object.

## Account Lookup

Account lookup is batched, like account creation. Pass
in all IDs to fetch. The account for each matched ID is returned.

If no account matches an ID, no object is returned for
that account. So the order of accounts in the response is
not necessarily the same as the order of IDs in the
request. You can refer to the ID field in the response to
distinguish accounts.

```python
accounts = client.lookup_accounts([100, 101])
```

## Create Transfers

This creates a journal entry between two accounts.

See details for transfer fields in the [Transfers
reference](https://docs.tigerbeetle.com/reference/transfer).

```python
transfers = [tb.Transfer(
    id=tb.id(), # TigerBeetle time-based ID.
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=0,
    timestamp=0,
)]

transfer_errors = client.create_transfers(transfers)
# Error handling omitted.
```

See details for the recommended ID scheme in
[time-based identifiers](https://docs.tigerbeetle.com/coding/data-modeling#tigerbeetle-time-based-identifiers-recommended).

### Response and Errors

The response is an empty array if all transfers were created
successfully. If the response is non-empty, each object in the
response array contains error information for a transfer that
failed. The error object contains an error code and the index of the
transfer in the request batch.

See all error conditions in the [create_transfers
reference](https://docs.tigerbeetle.com/reference/requests/create_transfers).

```python
batch = [tb.Transfer(
    id=1,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=0,
    timestamp=0,
),
    tb.Transfer(
    id=2,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=0,
    timestamp=0,
),
    tb.Transfer(
    id=3,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=0,
    timestamp=0,
)]

transfer_errors = client.create_transfers(batch)
for error in transfer_errors:
    if error.result == tb.CreateTransferResult.EXISTS:
        print(f"Batch transfer at {error.index} already exists.")
    else:
        print(f"Batch transfer at {error.index} failed to create: {error.result}.")
```

To handle errors you can compare the result code returned
from `client.create_transfers` with enum values in the
`CreateTransferResult` object.

## Batching

TigerBeetle performance is maximized when you batch
API requests.
A client instance shared across multiple threads/tasks can automatically
batch concurrent requests, but the application must still send as many events
as possible in a single call.
For example, if you insert 1 million transfers sequentially, one at a time,
the insert rate will be a *fraction* of the potential, because the client will
wait for a reply between each one.

```python
batch = [] # Array of transfer to create.
for transfer in batch:
    transfer_errors = client.create_transfers([transfer])
    # Error handling omitted.
```

Instead, **always batch as much as you can**.
The maximum batch size is set in the TigerBeetle server. The default is 8189.

```python
batch = [] # Array of transfer to create.
BATCH_SIZE = 8189 #FIXME
for i in range(0, len(batch), BATCH_SIZE):
    transfer_errors = client.create_transfers(
        batch[i:min(len(batch), i + BATCH_SIZE)],
    )
    # Error handling omitted.
```

### Queues and Workers

If you are making requests to TigerBeetle from workers
pulling jobs from a queue, you can batch requests to
TigerBeetle by having the worker act on multiple jobs from
the queue at once rather than one at a time. i.e. pulling
multiple jobs from the queue rather than just one.

## Transfer Flags

The transfer `flags` value is a bitfield. See details for these flags in
the [Transfers
reference](https://docs.tigerbeetle.com/reference/transfer#flags).

To toggle behavior for a transfer, combine enum values stored in the
`TransferFlags` object (it's an `enum.IntFlag`) with bitwise-or:

* `TransferFlags.linked`
* `TransferFlags.pending`
* `TransferFlags.post_pending_transfer`
* `TransferFlags.void_pending_transfer`

For example, to link `transfer0` and `transfer1`:

```python
transfer0 = tb.Transfer(
    id=4,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=tb.TransferFlags.LINKED,
    timestamp=0,
)
transfer1 = tb.Transfer(
    id=5,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=0,
    timestamp=0,
)

# Create the transfer
transfer_errors = client.create_transfers([transfer0, transfer1])
# Error handling omitted.
```

### Two-Phase Transfers

Two-phase transfers are supported natively by toggling the appropriate
flag. TigerBeetle will then adjust the `credits_pending` and
`debits_pending` fields of the appropriate accounts. A corresponding
post pending transfer then needs to be sent to post or void the
transfer.

#### Post a Pending Transfer

With `flags` set to `post_pending_transfer`,
TigerBeetle will post the transfer. TigerBeetle will atomically roll
back the changes to `debits_pending` and `credits_pending` of the
appropriate accounts and apply them to the `debits_posted` and
`credits_posted` balances.

```python
transfer0 = tb.Transfer(
    id=6,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=tb.TransferFlags.PENDING,
    timestamp=0,
)

transfer_errors = client.create_transfers([transfer0])
# Error handling omitted.

transfer1 = tb.Transfer(
    id=7,
    debit_account_id=102,
    credit_account_id=103,
    # Post the entire pending amount.
    amount=tb.amount_max,
    pending_id=6,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=tb.TransferFlags.POST_PENDING_TRANSFER,
    timestamp=0,
)

transfer_errors = client.create_transfers([transfer1])
# Error handling omitted.
```

#### Void a Pending Transfer

In contrast, with `flags` set to `void_pending_transfer`,
TigerBeetle will void the transfer. TigerBeetle will roll
back the changes to `debits_pending` and `credits_pending` of the
appropriate accounts and **not** apply them to the `debits_posted` and
`credits_posted` balances.

```python
transfer0 = tb.Transfer(
    id=8,
    debit_account_id=102,
    credit_account_id=103,
    amount=10,
    pending_id=0,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=tb.TransferFlags.PENDING,
    timestamp=0,
)

transfer_errors = client.create_transfers([transfer0])
# Error handling omitted.

transfer1 = tb.Transfer(
    id=9,
    debit_account_id=102,
    credit_account_id=103,
    amount=0,
    pending_id=8,
    user_data_128=0,
    user_data_64=0,
    user_data_32=0,
    timeout=0,
    ledger=1,
    code=720,
    flags=tb.TransferFlags.VOID_PENDING_TRANSFER,
    timestamp=0,
)

transfer_errors = client.create_transfers([transfer1])
# Error handling omitted.
```

## Transfer Lookup

NOTE: While transfer lookup exists, it is not a flexible query API. We
are developing query APIs and there will be new methods for querying
transfers in the future.

Transfer lookup is batched, like transfer creation. Pass in all `id`s to
fetch, and matched transfers are returned.

If no transfer matches an `id`, no object is returned for that
transfer. So the order of transfers in the response is not necessarily
the same as the order of `id`s in the request. You can refer to the
`id` field in the response to distinguish transfers.

```python
transfers = client.lookup_transfers([1, 2])
```

## Get Account Transfers

NOTE: This is a preview API that is subject to breaking changes once we have
a stable querying API.

Fetches the transfers involving a given account, allowing basic filter and pagination
capabilities.

The transfers in the response are sorted by `timestamp` in chronological or
reverse-chronological order.

```python
filter = tb.AccountFilter(
    account_id=2,
    user_data_128=0, # No filter by UserData.
    user_data_64=0,
    user_data_32=0,
    code=0, # No filter by Code.
    timestamp_min=0, # No filter by Timestamp.
    timestamp_max=0, # No filter by Timestamp.
    limit=10, # Limit to ten transfers at most.
    flags=tb.AccountFilterFlags.DEBITS | # Include transfer from the debit side.
    tb.AccountFilterFlags.CREDITS | # Include transfer from the credit side.
    tb.AccountFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
)

account_transfers = client.get_account_transfers(filter)
```

## Get Account Balances

NOTE: This is a preview API that is subject to breaking changes once we have
a stable querying API.

Fetches the point-in-time balances of a given account, allowing basic filter and
pagination capabilities.

Only accounts created with the flag
[`history`](https://docs.tigerbeetle.com/reference/account#flagshistory) set retain
[historical balances](https://docs.tigerbeetle.com/reference/requests/get_account_balances).

The balances in the response are sorted by `timestamp` in chronological or
reverse-chronological order.

```python
filter = tb.AccountFilter(
    account_id=2,
    user_data_128=0, # No filter by UserData.
    user_data_64=0,
    user_data_32=0,
    code=0, # No filter by Code.
    timestamp_min=0, # No filter by Timestamp.
    timestamp_max=0, # No filter by Timestamp.
    limit=10, # Limit to ten balances at most.
    flags=tb.AccountFilterFlags.DEBITS | # Include transfer from the debit side.
    tb.AccountFilterFlags.CREDITS | # Include transfer from the credit side.
    tb.AccountFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
)

account_balances = client.get_account_balances(filter)
```

## Query Accounts

NOTE: This is a preview API that is subject to breaking changes once we have
a stable querying API.

Query accounts by the intersection of some fields and by timestamp range.

The accounts in the response are sorted by `timestamp` in chronological or
reverse-chronological order.

```python
query_filter = tb.QueryFilter(
    user_data_128=1000, # Filter by UserData.
    user_data_64=100,
    user_data_32=10,
    code=1, # Filter by Code.
    ledger=0, # No filter by Ledger.
    timestamp_min=0, # No filter by Timestamp.
    timestamp_max=0, # No filter by Timestamp.
    limit=10, # Limit to ten accounts at most.
    flags=tb.QueryFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
)

query_accounts = client.query_accounts(query_filter)
```

## Query Transfers

NOTE: This is a preview API that is subject to breaking changes once we have
a stable querying API.

Query transfers by the intersection of some fields and by timestamp range.

The transfers in the response are sorted by `timestamp` in chronological or
reverse-chronological order.

```python
query_filter = tb.QueryFilter(
    user_data_128=1000, # Filter by UserData.
    user_data_64=100,
    user_data_32=10,
    code=1, # Filter by Code.
    ledger=0, # No filter by Ledger.
    timestamp_min=0, # No filter by Timestamp.
    timestamp_max=0, # No filter by Timestamp.
    limit=10, # Limit to ten transfers at most.
    flags=tb.QueryFilterFlags.REVERSED, # Sort by timestamp in reverse-chronological order.
)

query_transfers = client.query_transfers(query_filter)
```

## Linked Events

When the `linked` flag is specified for an account when creating accounts or
a transfer when creating transfers, it links that event with the next event in the
batch, to create a chain of events, of arbitrary length, which all
succeed or fail together. The tail of a chain is denoted by the first
event without this flag. The last event in a batch may therefore never
have the `linked` flag set as this would leave a chain
open-ended. Multiple chains or individual events may coexist within a
batch to succeed or fail independently.

Events within a chain are executed within order, or are rolled back on
error, so that the effect of each event in the chain is visible to the
next, and so that the chain is either visible or invisible as a unit
to subsequent events after the chain. The event that was the first to
break the chain will have a unique error result. Other events in the
chain will have their error result set to `linked_event_failed`.

```python
batch = [] # List of tb.Transfers to create.
linkedFlag = 0
linkedFlag |= tb.TransferFlags.LINKED

# An individual transfer (successful):
batch.append(tb.Transfer(id=1))

# A chain of 4 transfers (the last transfer in the chain closes the chain with linked=false):
batch.append(tb.Transfer(id=2, flags=linkedFlag)) # Commit/rollback.
batch.append(tb.Transfer(id=3, flags=linkedFlag)) # Commit/rollback.
batch.append(tb.Transfer(id=2, flags=linkedFlag)) # Fail with exists
batch.append(tb.Transfer(id=4, flags=0)) # Fail without committing.

# An individual transfer (successful):
# This should not see any effect from the failed chain above.
batch.append(tb.Transfer(id=2, flags=0 ))

# A chain of 2 transfers (the first transfer fails the chain):
batch.append(tb.Transfer(id=2, flags=linkedFlag))
batch.append(tb.Transfer(id=3, flags=0))

# A chain of 2 transfers (successful):
batch.append(tb.Transfer(id=3, flags=linkedFlag))
batch.append(tb.Transfer(id=4, flags=0))

transfer_errors = client.create_transfers(batch)
# Error handling omitted.
```

## Imported Events

When the `imported` flag is specified for an account when creating accounts or
a transfer when creating transfers, it allows importing historical events with
a user-defined timestamp.

The entire batch of events must be set with the flag `imported`.

It's recommended to submit the whole batch as a `linked` chain of events, ensuring that
if any event fails, none of them are committed, preserving the last timestamp unchanged.
This approach gives the application a chance to correct failed imported events, re-submitting
the batch again with the same user-defined timestamps.

```python
# External source of time.
historical_timestamp = 0
# Events loaded from an external source.
historical_accounts = [] # Loaded from an external source.
historical_transfers = [] # Loaded from an external source.

# First, load and import all accounts with their timestamps from the historical source.
accounts = []
for index, account in enumerate(historical_accounts):
    # Set a unique and strictly increasing timestamp.
    historical_timestamp += 1
    account.timestamp = historical_timestamp
    # Set the account as `imported`.
    account.flags = tb.AccountFlags.IMPORTED
    # To ensure atomicity, the entire batch (except the last event in the chain)
    # must be `linked`.
    if index < len(historical_accounts) - 1:
        account.flags |= tb.AccountFlags.LINKED

    accounts.append(account)

account_errors = client.create_accounts(accounts)
# Error handling omitted.

# The, load and import all transfers with their timestamps from the historical source.
transfers = []
for index, transfer in enumerate(historical_transfers):
    # Set a unique and strictly increasing timestamp.
    historical_timestamp += 1
    transfer.timestamp = historical_timestamp
    # Set the account as `imported`.
    transfer.flags = tb.TransferFlags.IMPORTED
    # To ensure atomicity, the entire batch (except the last event in the chain)
    # must be `linked`.
    if index < len(historical_transfers) - 1:
        transfer.flags |= tb.AccountFlags.LINKED

    transfers.append(transfer)

transfer_errors = client.create_transfers(transfers)
# Error handling omitted.

# Since it is a linked chain, in case of any error the entire batch is rolled back and can be retried
# with the same historical timestamps without regressing the cluster timestamp.
```

## Timeouts And Cancellation

The Client retries indefinitely and doesn't impose any per-request timeout. Cancellation is
provided as a mechanism, and the specific cancellation policy is left to the
application. A Client instance can be closed at any time. On close, all in-flight
requests are canceled and return an error to the caller. Even if an error is returned,
a request might still be processed by the TigerBeetle server.
[Reliable transaction submission](https://docs.tigerbeetle.com/coding/reliable-transaction-submission/)
explains how to make transfers retry-proof using IDs for end-to-end idempotency.