dump-things-pyclient 0.2.12
A client library and CLI commands for dump-things-services
Requires Python: >=3.11

# Dump Things Python Client

A simple client library and some CLI tools for dump-things-server in Python

The tools are in an early state and not automatically tested. Do not use them on
collections with valuable data unless you have a backup, or are very brave,
or quite reckless.


## Tech Stack

- Python >= 3.11

- [uv](https://astral.sh/uv) for dependency management


## Installation

The tools are published as pypi-project `dump-things-pyclient`. Install it, e.g.,
via `pid` (preferably in a virtual environment):

```
pip install dump-things-pyclient
```


## The commands

This project provides the CLI command `dtc`. `dtc` has a number of subcommands:

- auto-curate: automatically move records from inboxes to the curated area of a collection
- clean-incoming: delete all records from an inbox of a collection
- delete-records: delete records from an inbox or the curated area of a collection.
- export: export a collection to the file system
- get-records: get records from a dump-things collection
- import: import a collection from a file system dump (created by "export")
- list-incoming: list records in inboxes of a collection
- maintenance: activate or deactivate maintenance mode on a collection
- post-records: post records to an inbox or the curated area of a collection
- read-pages: read records from collection, curated area of a collection, or specific inboxes

Most commands require a token, all commands accept a token. Tokens are provided
to `dtc` with the option `--token` proceeding the subcommand.

This is the help message of `dtc`, which lists all available subcommands

```
 Usage: dtc [OPTIONS] COMMAND [ARGS]...                                                                                                                                                                                             
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --token  TEXT  provide a token on the command line, NOTE: on multiuser systems you should use the environment variable DTC_TOKEN instead                                                                                         │
│ --debug        show debug output                                                                                                                                                                                                 │
│ --help         Show this message and exit.                                                                                                                                                                                       │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ auto-curate                             Move records from inbox to curate area of a collection                                                                                                                                   │
│ clean-incoming                          Remove records from an inbox of a dump-things collection                                                                                                                                 │
│ delete-records                          Delete records from a dump-things collection                                                                                                                                             │
│ export                                  Export a collection to the file system                                                                                                                                                   │
│ get-records                             Get records from a dump-things collection                                                                                                                                                │
│ import                                  Import a collection from a file system                                                                                                                                                   │
│ list-incoming                           List inboxes of a dump-things collection                                                                                                                                                 │
│ maintenance                             Activate or deactivate maintenance mode on a collection                                                                                                                                  │
│ post-records                            Post records to an inbox or the curated area of a dump-things collection                                                                                                                 │
│ read-pages                              Read records from paginated dump-things endpoints                                                                                                                                        │
│ version                                 Show the version of `dtc`                                                                                                                                                                │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


The following sections show the help message for those `dtc`-subcommands


#### auto-create

Move records from inbox to curate area of a collection

``` 
 Usage: dtc auto-curate [OPTIONS] SERVICE_URL COLLECTION                                                                                                                                                                            
                                                                                                                                                                                                                                    
 Automatically move records from the incoming areas of the collection COLLECTION in the service SERVICE_URL to the curated area of the same collection, or to the curated area of another collection, possibly on another service.  
 A token is required and will be used to authenticate the requests. The token must have curator-rights.                                                                                                                             
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --destination-service-url      DEST_SERVICE_URL  select a different dump-thing-service, i.e. not SERVICE_URL, as destination for auto-curated records (the default is SERVICE_URL)                                               │
│ --destination-collection       DEST_COLLECTION   select a different collection, i.e. not COLLECTION, as destination for auto-curated records                                                                                     │
│ --destination-token            DEST_TOKEN        if provided, this token will be used the authenticate against DEST_SERVICE_URL, which defaults to SERVICE_URL (the default is the token provided via --token)                   │
│ --pid                      -p  PID               if provided, process only records that match the given PIDs. NOTE: matching does not involve CURIE-resolution                                                                   │
│ --exclude                  -e  TEXT              exclude an inbox on the source collection (repeatable)                                                                                                                          │
│ --include                  -i  TEXT              process only the given inbox, all other inboxes are ignored (repeatable, -e/--exclude is applied after inclusion)                                                               │
│ --list-labels              -l                    list the inbox labels of the given source collection, do not perform any curation                                                                                               │
│ --list-records             -r                    list records in the inboxes of the given source collection, do not perform any curation                                                                                         │
│ --dry-run                  -d                    if provided, do not alter any data, instead print what would be done                                                                                                            │
│ --help                                           Show this message and exit.                                                                                                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### clean-incoming

Usage: dtc clean-incoming [OPTIONS] SERVICE_URL COLLECTION INBOX_LABEL

Remove all records from an incoming areas of a collection on a dump-things-service                                                                                                                                                 
This command removes all records from the inbox with label INBOX_LABEL in the collection COLLECTION on the dump-things service given by SERVICE_URL.                                                                               
A token with curator rights has to be provided.

```
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --list-only  -l  only list records in the inbox, do not remove them                                                                                                                                                              │
│ --help           Show this message and exit.                                                                                                                                                                                     │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```


#### delete-records

Delete records from a collection on a dump-things-server

```
Usage: dtc delete-records [OPTIONS] SERVICE_URL COLLECTION PIDS

Delete records from a collection on a dump-things-service                                                                                                                                                                          
This command delete the records given by PIDS from the collection COLLECTION of the dump-things service SERVICE_URL. If no pids are provided on the command line, the pid that should be deleted are read from stdin (one pid per  
line, lines are stripped).                                                                                                                                                                                                         
By default, the records will be deleted from the inbox associated with the token. If the option `-c/--curated` is given, the records are deleted from the curated area of the collection (this requires a token with curator       
rights). If the option `-i/--incoming LABEL` is given, the records are deleted from the inbox specified by `LABEL` (this requires a token with curator rights).

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --curated              -c         delete record from the curated area of the collection. (Note: requires a token with curator rights)                                                                                            │
│ --incoming             -i  LABEL  delete from the collection's inbox with label LABEL, if LABEL is "-", return labels of all collection inboxes and exit                                                                         │
│ --ignore-errors                   ignore errors when deleting a pid and continue with remaining pids                                                                                                                             │
│ --class                -C  CLASS  delete ALL records of class CLASS from the collection's incoming area that is associated with the token. Can be combined with `-i/--incoming LABEL` or `-c/--curated` to delete all records of │
│                                   class CLASS from the incoming area `LABEL` or from the curated area. Note: if neither `-c/--curated` nor `-i/--incoming LABEL` is specified, the command cannot reliably determine which       │
│                                   records are stored in the incoming area associated with a token and which records are stored in the curated area of the collection. This can lead to warnings about records that cannot be     │
│                                   deleted. The command will print a list of all PIDs that could not be deleted.                                                                                                                  │
│ --json-error-messages             if this flag is given, output information about failed delete operations to stdout. The format is JSONL (JSON lines), each JSON record contains the detailed error message, the PID of the     │
│                                   record that could not be deleted.                                                                                                                                                              │
│ --help                            Show this message and exit.                                                                                                                                                                    │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### export 

Export the curated area and the inboxes of a collection to the file system.

```
 Usage: dtc export [OPTIONS] SERVICE_URL COLLECTION DESTINATION_DIR                                                                                                                                                                 
                                                                                                                                                                                                                                    
 Export a collection to disk                                                                                                                                                                                                        
 This command exports all records that are stored in curated area and in the incoming areas of collection COLLECTION of the dump-things service SERVICE_URL.                                                                        
 Exported records are written to the directory DESTINATION_DIR. DESTINATION_DIR must not exist, `export` will create it.                                                                                                            
 A token with curator rights has to be provided.                                                                                                                                                                                    
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --format               -f  [json|yaml]  select output format for the exported records (default: json)                                                                                                                            │
│ --ignore-errors                         ignore records with missing `schema_type` instead of raising an error                                                                                                                    │
│ --keep-schema-type     -k               keep `schema_type`-attribute in records on file-system. By default the schema_type-attribute is removed because the class is encoded in the storage path of the records.                 │
│ --json-error-messages                   if this flag is given, output information about failed read or write operations to stdout. The format is JSONL (JSON lines), each JSON record contains the operation type (read, write), │
│                                         a detailed error message, and additional context dependent information, e.g., the PID of the record that could not be written to the file system.                                        │
│ --help                                  Show this message and exit.                                                                                                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### get-records

```
 Usage: dtc get-records [OPTIONS] SERVICE_URL COLLECTION                                                                                                                                                                            
                                                                                                                                                                                                                                    
 Get records from a collection on a dump-things-service                                                                                                                                                                             
 This command lists records that are stored in collection COLLECTION of the dump-things service SERVICE_URL. By default, all records that are readable with the given token, or the default token, will be displayed. The output    
 format is JSONL (JSON lines), where every line contains a record or a record with paging information.  If `ttl` is chosen as format of the output records, the record content will be a string that contains a TTL-documents.      
 The command supports reading from the curated area only, reading from incoming areas, or reading a record with a given PID.                                                                                                        
 Pagination information is returned for paginated results, when requested with `-P/--pagination`. All results are paginated except "get a record with a given PID" and "get the list of incoming zone labels".                      
 For reading from curated or incoming areas, a token with curator rights has to be provided.                                                                                                                                        
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --class       -C  TEXT                       only read records of this class, ignored if "--pid" is provided                                                                                                                     │
│ --format      -f  [json|ttl]                 request records in a specific format. (NOTE: not all endpoints support the "format"-parameter)                                                                                      │
│ --pid         -p  TEXT                       the pid of the record that should be read                                                                                                                                           │
│ --incoming    -i  LABEL                      read from the collection's inbox with label LABEL, if LABEL is "-", print labels of all collection inboxes and exit                                                                 │
│ --curated     -c                             read from the curated area of the collection. (Note: requires a token with curator rights)                                                                                          │
│ --matching    -m  TEXT                       return only records that have a matching value (use % as wildcard). Ignored if "--pid" is provided. (Note: not all endpoints and backends support matching)                         │
│ --page-size   -s  INTEGER RANGE [1<=x<=100]  set the page size (default: 100). (ignored if "--pid" is provided)                                                                                                                  │
│ --first-page  -F  INTEGER                    the first page to return (default: 1). (ignored if "--pid" is provided)                                                                                                             │
│ --last-page   -l  INTEGER                    the last page to return, if not given, all pages will be returned. (ignored if "--pid" is provided)                                                                                 │
│ --stats                                      show the number of records and pages and exit. (ignored if "--pid" is provided)                                                                                                     │
│ --pagination  -P                             show pagination information (each record from an paginated endpoint is returned as [<record>, <current page number>, <total number of pages>, <page size>, <total number of         │
│                                              items>]. (ignored if "--pid" is provided)                                                                                                                                           │
│ --help                                       Show this message and exit.                                                                                                                                                         │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### import

```
 Usage: dtc import [OPTIONS] SOURCE_DIR                                                                                                                                                                                             
                                                                                                                                                                                                                                    
 Import a collection from disk                                                                                                                                                                                                      
 This command imports all records that are stored on disk in the directory SOURCE_DIR in the format that is created by `dtc export`. The records are stored in the dump-things service and the collection that are recorded in      
 `SOURCE_DIR/description.json`.                                                                                                                                                                                                     
 A token with curator rights has to be provided.                                                                                                                                                                                    
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --service-url          -s  SERVICE_URL  use the service SERVICE_URL instead of the service URL that is stored in `SOURCE_DIR/description.json`                                                                                   │
│ --collection           -c  COLLECTION   use the collection name COLLECTION instead of the collection name that is stored in `SOURCE_DIR/description.json`                                                                        │
│ --ignore-errors                         log errors an continue import instead of raising an exception                                                                                                                            │
│ --json-error-messages                   if this flag is given, output information about failed read or write operations to stdout. The format is JSONL (JSON lines), each JSON record contains the operation type (read, write), │
│                                         a detailed error message, and additional context dependent information, e.g., the PID of the record that could not be posted to the collection.                                          │
│ --help                                  Show this message and exit.                                                                                                                                                              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### list-incoming

```
Usage: dtc list-incoming [OPTIONS] SERVICE_URL COLLECTION

List labels of incoming areas of a collection on a dump-things-service                                                                                                                                                             
This command lists the labels of the incoming areas of the collection COLLECTION on the dump-things service given by SERVICE_URL.                                                                                                  
A token with curator rights has to be provided.

╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ --show-records  -s  list records in inboxes                                                                                                                                                                                      │
│ --help              Show this message and exit.                                                                                                                                                                                  │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
```


#### maintenance

```
 Usage: dtc maintenance [OPTIONS] SERVICE_URL COLLECTION ACTIVE                                                                                                                                                                     
                                                                                                                                                                                                                                    
 Activate or deactivate maintenance mode on collection COLLECTION on the service SERVICE_URL. The argument ACTIVE should be either `On` or `Off` (case-insensitive).                                                                
 A token with curator rights is required.                                                                                                                                                                                           
 This command expects a server version >= 5.4.0                                                                                                                                                                                     
                                                                                                                                                                                                                                    
╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────