python-irodsclient 3.2.0
A python API for iRODS
License: BSD
Keywords: irods

Python iRODS Client (PRC)
=========================

[iRODS](https://www.irods.org) is an open source distributed data management system. This is a client API implemented in Python.

Currently supported:

-   Python 3.9 or later
-   Establish a (secure) connection to iRODS
-   Authenticate via password, GSI, PAM
-   GenQuery and Specific Queries
-   GenQuery2
-   Manage collections, data objects, and permissions
    -   Checksum data objects
    -   Replicate data objects
    -   Parallel PUT/GET data objects
    -   Read, write, and seek operations
    -   Register files and directories
-   Manage users and groups
-   Manage resources
-   Manage and execute iRODS rules
-   Manage metadata
-   Manage ticket-based access

Installing
----------

Install via pip:

    pip install python-irodsclient

or:

    pip install git+https://github.com/irods/python-irodsclient.git[@branch|@commit|@tag]

Uninstalling
------------

    pip uninstall python-irodsclient

Establishing a (secure) connection
----------------------------------

One way of starting a session is to pass iRODS credentials as keyword
arguments:

```python
>>> from irods.session import iRODSSession
>>> with iRODSSession(host='localhost', port=1247, user='bob', password='1234', zone='tempZone') as session:
...      # workload
...
>>>
```

If you're an administrator acting on behalf of another user:

```python
>>> from irods.session import iRODSSession
>>> with iRODSSession(host='localhost', port=1247, user='rods', password='1234', zone='tempZone', client_user='bob',
           client_zone='possibly_another_zone') as session:
...      # workload
...
>>>
```

If no `client_zone` is provided, the `zone` parameter is used in its place.

Using environment files (including any SSL settings) in `~/.irods/`:

```python
>>> import os
>>> import ssl
>>> from irods.session import iRODSSession
>>> try:
...     env_file = os.environ['IRODS_ENVIRONMENT_FILE']
... except KeyError:
...     env_file = os.path.expanduser('~/.irods/irods_environment.json')
...
>>> ssl_settings = {} # Or, optionally: {'ssl_context': <user_customized_SSLContext>}
>>> with iRODSSession(irods_env_file=env_file, **ssl_settings) as session:
...     # workload
...
>>>
```

In the above example, an SSL connection can be made even if no
'ssl_context' argument is specified, in which case the Python client
internally generates its own SSLContext object to best match the iRODS
SSL configuration parameters (such as
"irods_ssl_ca_certificate_file", etc.) used to initialize the
iRODSSession. Those parameters can be given either in the environment
file, or in the iRODSSession constructor call as shown by the next
example.

A pure Python SSL session (without a local `env_file` requires a few more things defined:

```python
>>> import ssl
>>> from irods.session import iRODSSession
>>> ssl_settings = {'client_server_negotiation': 'request_server_negotiation',
...                'client_server_policy': 'CS_NEG_REQUIRE',
...                'encryption_algorithm': 'AES-256-CBC',
...                'encryption_key_size': 32,
...                'encryption_num_hash_rounds': 16,
...                'encryption_salt_size': 8,
...                'ssl_context': ssl_context
...                'ssl_verify_server': 'cert',
...                'ssl_ca_certificate_file': '/etc/irods/ssl/irods.crt'
... }
```

If necessary, a user may provide a custom SSLContext object; although,
as of release v1.1.6, this will rarely be required:

```python
>>> ssl_settings ['ssl_context'] = ssl.create_default_context(purpose=ssl.Purpose.SERVER_AUTH, # ... other options
... )
```

At this point, we are ready to instantiate and use the session:

```python
>>> with iRODSSession(host='irods-provider', port=1247, user='bob', password='1234', zone='tempZone', **ssl_settings) as session:
...	# workload
```

Note that the `irods_` prefix is unnecessary when providing
the `encryption_*` and `ssl_*` options
directly to the constructor as keyword arguments, even though it is
required when they are placed in the environment file.

Creating a PAM or Native Authentication File
--------------------------------------------

The following free functions may be used to create the authentication secrets files (called
`.irodsA` per the convention of iRODS's iCommands):
   - `irods.client_init.write_native_irodsA_file`
   - `irods.client_init.write_pam_irodsA_file`

These functions can roughly be described as duplicating the "authentication" functionality of `iinit`,
provided that a valid `irods_environment.json` has already been created.

Each of the above functions can take a cleartext password and write an appropriately encoded
version of it into an authentication file in the appropriate location.  That location is
`~/.irods/.irodsA` unless the environment variable IRODS_AUTHENTICATION_FILE has been set
in the command shell to dictate an alternative file path.

As an example, here we write a native `.irodsA` file using the first of the two functions.  We
provide the one required argument, a password string which is entered interactively at the
terminal.

```bash
$ echo '{ "irods_user_name":"rods",
          ... # other parameters as needed
        }'> ~/.irods/irods_environment.json
$ python -c "import irods.client_init, getpass
irods.client_init.write_native_irodsA_file(getpass.getpass('Enter iRODS password -> '))"
```

By default, when an `.irodsA` file already exists, it will be overwritten. If however the
`overwrite` parameter is set to `False`, an exception of type `irods.client_init.irodsA_already_exists`
is raised to warn of any older `.irodsA` file that might otherwise have been overwritten.

Equivalently to the above, we can issue the following command.

```bash
$ prc_write_irodsA.py native <<<"${MY_CURRENT_IRODS_PASSWORD}"
```

The redirect may of course be left off, in which case the user is prompted for the iRODS password
and echo of the keyboard input will be suppressed, in the style of `iinit`.  Regardless of
which technique is used, no password will be visible on the terminal during or after input.

For the `pam_password` scheme, typically SSL/TLS must first be enabled to avoid sending data related
to the password - or even sending the raw password itself - over a network connection in the clear.

Thus, for `pam_password` authentication to work well, we should first ensure, when setting up the
client environment, to include within `irods_environment.json` the appropriate SSL/TLS connection
parameters.  In a pinch, `iinit` can be used to verify this prerequisite is fulfilled,
as its invocation would then create a valid `.irodsA` from merely prompting the user for their PAM password.

Once again, this can also be done using the free function directly:

```python
irods.client_init.write_pam_irodsA_file(getpass.getpass('Enter current PAM password -> '))
```

or from the Bash command shell:

```bash
$ prc_write_irodsA.py pam_password <<<"${MY_CURRENT_PAM_PASSWORD}"
```

As a final note, in the `pam_password` scheme, the default SSL requirement can be disabled.
**Warning:** Disabling the SSL requirement may cause user passwords to be sent over the network
in the clear. This should only be done for purposes of testing. Here's how to do it:

```python
from irods.auth.pam_password import ENSURE_SSL_IS_ACTIVE

session = irods.session.iRODSSession(host = "localhost", port = 1247,
                                     user = "alice", password = "test123", zone="tempZone",
                                     authentication_scheme = "pam_password")

session.set_auth_option_for_scheme('pam_password', ENSURE_SSL_IS_ACTIVE, False)

# Do something with the session:
home = session.collections.get('/tempZone/home/alice')
```

Note, however, in future releases of iRODS it is possible that extra SSL checking could be
implemented server-side, at which point the above code could not be guaranteed to work.

Legacy (iRODS 4.2-compatible) PAM authentication
------------------------------------------------

Since v2.0.0, the Python iRODS Client is able to authenticate via PAM using the same file-based client environment as the
iCommands.

Caveat for iRODS 4.3+: when upgrading from 4.2, the "irods_authentication_scheme" setting must be changed from "pam" to "pam_password" in
`~/.irods/irods_environment.json` for all file-based client environments.

To use the PRC PAM login credentials update function for the client login environment, we can set these two configuration variables:

```
legacy_auth.pam.password_for_auto_renew "my_pam_password"
legacy_auth.pam.store_password_to_environment True
```

Optionally, the `legacy_auth.pam.time_to_live_in_hours` may also be set to determine the time-to-live for the new password.
Leaving it at the default value defers this decision to the server.

Maintaining a connection
------------------------

The default library timeout for a connection to an iRODS Server is 120 seconds.

This can be overridden by changing the session `connection_timeout` immediately after creation of the
session object:

```python
>>> session.connection_timeout = 300
```

This will set the timeout to five minutes for any associated connections.

Timeouts are either a positive `int` or `float` with units of seconds, or `None`, all in accordance with their
meaning in the context of the socket module.  A value of `None` indicates timeouts are effectively
infinite in value, i.e. turned off.  Setting a session's `connection_timeout` value to 0 is disallowed
because this would cause the socket to enter non-blocking mode.

Session objects and cleanup
---------------------------

When iRODSSession objects are kept as state in an application, spurious
SYS_HEADER_READ_LEN_ERR errors can sometimes be seen in the
connected iRODS server's log file. This is frequently seen at program
exit because socket connections are terminated without having been
closed out by the session object's cleanup() method.

Since v0.9.0, code has been included in the session
object's `__del__` method to call cleanup(), properly closing out
network connections. However, `__del__`  is not guaranteed to run as
expected, so an alternative may be to call `session.cleanup()`
on any session variable which will not be used again.

Simple PUTs and GETs
--------------------

We can use the just-created session object to put files to (or get them
from) iRODS.

```python
>>> logical_path = "/{0.zone}/home/{0.username}/{1}".format(session,"myfile.dat")
>>> session.data_objects.put("myfile.dat", logical_path)
>>> session.data_objects.get(logical_path, "/tmp/myfile.dat.copy")
```

Note that local file paths may be relative, but iRODS data objects must
always be referred to by their absolute paths. This is in contrast to
the `iput` and `iget` icommands, which keep track of the current working
collection (as modified by `icd`) for the unix shell.

Note also that PRC `put()` is actually using the `open`, `write`, and `close` APIs, rather than the
iRODS PUT API directly.  This is transparent to the caller, but an administrator
should take note as this affects which policy enforcement points (PEPs) are executed
on the iRODS server.

Release v3.1.1 introduces the optional behavior of preventing a call to data object manager's `put()` or `create()`
method from succeeding if the requested data path already exists.  This will become the default in some future release,
but for now the following code can be used to enable it for the duration of
an application's run:

```python
import irods.client_configuration as config
# Prevent accidental data object overwrites in put() and create().
config.data_objects.force_create_by_default = False
config.data_objects.force_put_by_default = False
```

A more ad-hoc solution to defeat the "True" defaults is by passing `**{irods.keywords.FORCE_FLAG_KW:False}` among
the options in an individual call to `put`, or using force=False in a call to `create`.

Both solutions will be unnecessary, but still remain effective, when True-defaulting "force flag" style data overwrites
become non-default behavior and/or deprecated in a future release.

Parallel Transfer
-----------------

Since v0.9.0, data object transfers using `put()` and `get()`
will spawn a number of threads in order to optimize performance for
iRODS server versions 4.2.9+ and file sizes larger than a default
threshold value of 32 Megabytes.

Progress bars
-------------

The PRC supports progress bars which function on the basis of
an "update" callback function.  In the case of a tqdm progress bar (see https://github.com/tqdm/tqdm), you can always just
pass the update method of the progress bar instance directly to the data_object
`put` or `get` method:

```python
   pbar = tqdm.tqdm(total = data_obj.size)
   session.data_objects.get(file_name, data_obj.path, updatables = pbar.update)
```

The updatables parameter can be a list or tuple of update-enabling objects and/or bound methods.

Alternatively, the tqdm progress bar object itself can be passed in, if an adapting
function such as the following is first registered:

```python
    def adapt_tqdm(pbar, l = threading.Lock()):
        def _update(n):
            with l:
                pbar.update(n)
        return _update
    irods.manager.data_objects_manager.register_update_type( adapt_tqdm )
    session.data_objects.put( file, logical_path, updatables = [tqdm_1,tqdm_2] ) # update two tqdm's simultaneously
```

Other progress bars may be included in an updatables parameter, but may require more extensive adaptation.
For example, the ProgressBar object (from the progressbar module) also has an update method, but it
takes an up-to-date cumulative byte-count, instead of the size of an individual transfer in bytes,
as its sole parameter.  There can be other complications:  e.g. a ProgressBar instance does not allow a weak
reference to itself to be formed, which interferes with the Python iRODS Client's internal scheme of accounting
for progress bar instances "still in progress" while also preventing resource leaks.

In such cases, it is probably best to implement a wrapper for the progress
bar in question, and submit the wrapper instance as the updatable parameter.  Whether
a wrapper or the progress bar object itself is thus employed, it is recommended that the user take steps to
ensure the lifetime of the updatable instance extends beyond the time needed for the transfer to complete.

See `irods/test/data_obj_test.py` for examples of these and other subtleties of progress bar usage.

Working with collections (directories)
--------------------------------------

```python
>>> coll = session.collections.get("/tempZone/home/rods")

>>> coll.id
45798

>>> coll.path
/tempZone/home/rods

>>> for col in coll.subcollections:
>>>   print(col)
<iRODSCollection /tempZone/home/rods/subcol1>
<iRODSCollection /tempZone/home/rods/subcol2>

>>> for obj in coll.data_objects:
>>>   print(obj)
<iRODSDataObject /tempZone/home/rods/file.txt>
<iRODSDataObject /tempZone/home/rods/file2.txt>
```

Create a new collection:

```python
>>> coll = session.collections.create("/tempZone/home/rods/testdir")
>>> coll.id
45799
```

Working with data objects (files)
---------------------------------

Create a new data object:

```python
>>> obj = session.data_objects.create("/tempZone/home/rods/test1")
<iRODSDataObject /tempZone/home/rods/test1>
```

Get an existing data object:

```python
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> obj.id 12345

>>> obj.name
test1
>>> obj.collection
<iRODSCollection /tempZone/home/rods>

>>> for replica in obj.replicas:
...     print(replica.resource_name)
...     print(replica.number)
...     print(replica.path)
...     print(replica.status)
...
demoResc
0
/var/lib/irods/Vault/home/rods/test1
1
```

Using the `put()` method rather than the `create()` method will trigger different policy enforcement points (PEPs) on the server.

Put an existing file as a new data object:

```python
>>> session.data_objects.put("test.txt", "/tempZone/home/rods/test2")
>>> obj2 = session.data_objects.get("/tempZone/home/rods/test2")
>>> obj2.id
56789
```

Specifying paths
----------------

Path strings for collection and data objects are usually expected to be
absolute in most contexts in the PRC. They must also be normalized to a
form including single slashes separating path elements and no slashes at
the string's end. If there is any doubt that a path string fulfills
this requirement, the wrapper class `irods.path.iRODSPath` (a subclass of `str`) may be used to normalize it:

    if not session.collections.exists( iRODSPath( potentially_unnormalized_path )): #....

The wrapper serves also as a path joiner; thus:

    iRODSPath( zone, "home", user )

may replace:

    "/".join(["", zone, "home", user])

`iRODSPath` has been available since v1.1.2.

Reading and writing files
-------------------------

PRC provides [file-like
objects](https://docs.python.org/3/glossary.html#term-file-object) for reading and writing files.

```python
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> with obj.open('r+') as f:
...   f.write('foonbarn')
...   f.seek(0,0)
...   for line in f:
...      print(line)
...
foo
bar
```

Since v1.1.9, there is also an auto-close configuration setting for data
objects, set to `False` by default, which may be assigned
the value `True` for guaranteed auto-closing of open data
object handles at the proper time.

In a small but illustrative example, the following Python session does
not require an explicit call to `f.close()`:

```python
>>> import irods.client_configuration as config, irods.helpers as helpers
>>> config.data_objects.auto_close = True
>>> session = helpers.make_session()
>>> f = session.data_objects.open('/{0.zone}/home/{0.username}/new_object.txt'.format(session),'w')
>>> f.write(b'new content.')
```

This may be useful for Python programs in which frequent flushing of
write updates to data objects is undesirable -- with descriptors on such
objects possibly being held open for indeterminately long lifetimes --
yet the eventual application of those updates prior to the teardown of
the Python interpreter is required.

The current value of the setting is global in scope (i.e. applies to all
sessions, whenever created) and is always consulted for the creation of
any data object handle to govern that handle's cleanup behavior.

Also, alternatively, the client may opt into a special "redirect" behavior
in which data objects' `open()` method makes a new connection directly to whichever
iRODS server is found to host the selected replica.  Data reads and
writes will therefore happen on that alternate network route, instead of
through the originally-connected server.  Though not the client's default
behavior, this approach can turn out to be more efficient, particularly
if several concurrent data uploads ("puts") and downloads ("gets") are 
happening which might increase traffic on the client's main communication
route with the server.  (See, in [Python iRODS Client Settings File](#python-irods-client-settings-file),
the client configuration setting `data_objects.allow_redirect`, which may be
set to True to designate the opt-in.)

Python iRODS Client Settings File
---------------------------------

Since v1.1.9, Python iRODS client configuration can be saved in, and
loaded from, a settings file.

If the settings file exists, each of its lines contains (a) a dotted
name identifying a particular configuration setting to be assigned
within the PRC, potentially changing its runtime behavior; and (b) the
specific value, in Python "repr"-style format, that should be assigned
into it.

An example follows:

```
data_objects.auto_close True
```

New dotted names may be created following the example of the one valid
example created thus far, `data_objects.auto_close]`,
initialized in `irods/client_configuration/__init__.py`.
Each such name should correspond to a globally set value which the PRC
routinely checks when performing the affected library function.

The use of a settings file can be indicated, and the path to that file
determined, by setting the environment variable:
`PYTHON_IRODSCLIENT_CONFIGURATION_PATH`. If this variable
is present but empty, this denotes use of a default settings file path
of `~/.python-irodsclient`; if the variable's value is of
non-zero length, the value should be an absolute path to the desired settings
file location. Also, if the variable is set, auto-load of
settings will be performed, meaning that the act of importing
`irods` or any of its submodules will cause the automatic
loading of the settings from the settings file, assuming it exists.
(Failure to find the file at the indicated path will be logged as a
warning.)

If the process of loading configuration settings generates an unhandled exception,
the `irods` module will consult the value of the environment variable
`PYTHON_IRODSCLIENT_CONFIGURATION_LOAD_ERRORS_FATAL` and either (if `True`) incur
a fatal error or (if `False`) simply log it as a warning.  By default this
environment variable is taken as `False` if not set, although that is likely to
change to `True` in the future.
This decision indicates a desire by the client library's designers to flag such
errors as fatal by aborting the process if possible, rather than aquiescing silently
to an unintended consequence.  It is recommended that all client library users
set this environment variable to `True` if their applications load settings
from the environment or configuration files.

Settings can be saved and loaded manually using the `save()` and
`load()` functions in the `irods.client_configuration`
module. Each of these functions accepts an optional `file`
parameter which, if set to a non-empty string, will override the
settings file path currently "in force" (i.e., the
CONFIG_DEFAULT_PATH, as optionally overridden by the environment
variable PYTHON_IRODSCLIENT_CONFIGURATION_PATH).

Configuration settings may also be individually overridden by defining
certain environment variables:

-   Setting: Auto-close option for all data objects.
    -   Dotted Name: `data_objects.auto_close`
    -   Type: `bool`
    -   Default Value: `False`
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__AUTO_CLOSE`

-   Setting: Let a call to data object open() redirect to the server whose storage resource hosts the given object's preferred replica.
    -   Dotted Name: `data_objects.allow_redirect`
    -   Type: `bool`
    -   Default Value: `False`
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__ALLOW_REDIRECT`

-   Setting: Allow `put()` to overwrite an already existing data object by default.
    -   Dotted Name: `data_objects.force_put_by_default`
    -   Type: `bool`
    -   Default Value: `True` (as of v3.1.1, but not into perpetuity.)
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__FORCE_PUT_BY_DEFAULT`

-   Setting: Allow `create()` to overwrite an already existing data object by default.
    -   Dotted Name: `data_objects.force_create_by_default`
    -   Type: `bool`
    -   Default Value: `True` (as of v3.1.1, but not into perpetuity.)
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__DATA_OBJECTS__FORCE_CREATE_BY_DEFAULT`

-   Setting: Whether to use legacy authentication despite the iRODS server supporting the 4.3 authentication plugin framework.
    - Dotted Name: `legacy_auth.force_legacy_auth`
    - Type: `bool`
    - Default Value: `False` (Meaning: allow the version of the connected server to determine the type of authentication used.)
    - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__FORCE_LEGACY_AUTH`

-   Setting: Number of hours to request for the new password entry's TTL (Time To Live) when auto-renewing PAM-authenticated sessions.
    - Dotted Name: `legacy_auth.pam.time_to_live_in_hours`
    - Type: `int`
    - Default Value: `0` (Meaning: conform to server's default TTL value.)
    - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__TIME_TO_LIVE_IN_HOURS`

-   Setting: Plaintext PAM password value, to be used when auto-renewing PAM-authenticated sessions because TTL has expired.
    - Dotted Name: `legacy_auth.pam.password_for_auto_renew`
    - Type: `str`
    - Default Value: `""` (Meaning: no password is set, and thus no automatic attempts will be made at auto-renewing PAM authentication.)
    - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__PASSWORD_FOR_AUTO_RENEW`.  (But note that use of the environment variable could pose a threat to password security.)

-   Setting: Whether to write the (native encoded) new hashed password to the iRODS password file.  This step is only performed while auto-renewing PAM authenticated sessions.
    - Dotted Name: `legacy_auth.pam.store_password_to_environment`
    - Type: `bool`
    - Default Value: `False`
    - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__STORE_PASSWORD_TO_ENVIRONMENT`

-   Setting: Force the use of PAM_AUTH_REQUEST_AN API for entering a new PAM password into the catalog.  This API accommodates longer passwords and avoids the step of parsing a semicolon-delimited
    "context" parameter.
    - Dotted Name: `legacy_auth.pam.force_use_of_dedicated_pam_api`
    - Type: `bool`
    - Default Value: `False`
    - Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__LEGACY_AUTH__PAM__FORCE_USE_OF_DEDICATED_PAM_API`

-   Setting: The maximum number of rows a Query iteration will retrieve.  Can be overridden using Query's `limit` method. This is synonymous with the variable `irods.query.IRODS_QUERY_LIMIT`, and a read (or write) of this configuration setting will read (or affect) that variable.
    -   Dotted Name: `genquery1.irods_query_limit`
    -   Type: `int`
    -   Default Value: 500 (The library's traditional maximum batch row count, also stored as `irods.query._IRODS_QUERY_LIMIT`)
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__GENQUERY1__IRODS_QUERY_LIMIT`

-   Setting: Default choice of XML parser for all new threads.
    -   Dotted Name: `connections.xml_parser_default`
    -   Type: `str`
    -   Default Value: `"STANDARD_XML"`
    -   Possible Values: Any of `["STANDARD_XML", "QUASI_XML", "SECURE_XML"]`
    -   Environment Variable Override: `PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT`

For example, if `~/.python_irodsclient` contains the line :

```
connections.xml_parser_default        "QUASI_XML"
```

then the session below illustrates the effect of defining the
appropriate environment variable. Note the value stored in the variable
must be a valid input for `ast.literal_eval()`; that is, a
primitive Pythonic value - and quoted, for instance, if a string.

```bash
$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH="" \
  PYTHON_IRODSCLIENT_CONFIG__CONNECTIONS__XML_PARSER_DEFAULT="'SECURE_XML'" \
  python -c "import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())"
XML_Parser_Type.SECURE_XML
$ PYTHON_IRODSCLIENT_CONFIGURATION_PATH="" \
  python -c "import irods.message, irods.client_configuration as c; print (irods.message.default_XML_parser())"
XML_Parser_Type.QUASI_XML
```

Computing and Retrieving Checksums
----------------------------------

Each data object may be associated with a checksum by calling `chksum()`
on the object in question. Various behaviors can be elicited by passing
in combinations of keywords (for a description of which, please consult
the [header documentation](https://github.com/irods/irods/blob/4-3-stable/lib/api/include/irods/dataObjChksum.h).)

As with most other iRODS APIs, it is straightforward to specify keywords
by adding them to an options dictionary:

```python
>>> data_object_1.chksum() # - computes the checksum if already in the catalog, otherwise computes and stores it
...                        # (i.e. default behavior with no keywords passed in.)
>>> from irods.manager.data_object_manager import Server_Checksum_Warning
>>> import irods.keywords as kw
>>> opts = { kw.VERIFY_CHKSUM_KW:'' }
>>> try:
...     data_object_2.chksum( **opts ) # - Uses verification option. (Does not create or save a checksum in the catalog).
...     # or:
...     opts[ kw.NO_COMPUTE_KW ] = ''
...     data_object_2.chksum( **opts ) # - Uses both verification and no-compute options. (Like `ichksum -K --no-compute`)
... except Server_Checksum_Warning:
...     print('some checksums are missing or wrong')
```

Additionally, if a freshly created `irods.message.RErrorStack` instance is
given, information can be returned and read by the client:

```python
>>> from irods.message import RErrorStack
>>> r_err_stk = RErrorStack()
>>> warn = None
>>> try:   # Here, data_obj has one replica, not yet checksummed.
...     data_obj.chksum( r_error = r_err_stk , **{kw.VERIFY_CHKSUM_KW:''} )
... except Server_Checksum_Warning as exc:
...     warn = exc
>>> print(r_err_stk)
[RError<message = u'WARNING: No checksum available for replica [0].', status = -862000 CAT_NO_CHECKSUM_FOR_REPLICA>]
```

Working with metadata
---------------------

Showing the Attribute-Value-Units (AVUs) on an object with no metadata attached shows an empty list:

```python
>>> from irods.meta import iRODSMeta
>>> obj = session.data_objects.get("/tempZone/home/rods/test1")
>>> print(obj.metadata.items())
[]
```

Adding multiple AVUs with the same name field:

```python
>>> obj.metadata.add('key1', 'value1', 'units1')
>>> obj.metadata.add('key1', 'value2')
>>> obj.metadata.add('key2', 'value3')
>>> obj.metadata.add('key2', 'value4')
>>> print(obj.metadata.items())
[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13185 key2 value4 None>,
<iRODSMeta 13183 key1 value2 None>, <iRODSMeta 13184 key2 value3 None>]
```

We can also use Python's item indexing syntax to perform the equivalent
of an "imeta set \...", e.g. overwriting all AVUs with a name field
of "key2" in a single update:

```python
>>> new_meta = iRODSMeta('key2','value5','units2')
>>> obj.metadata\[new_meta.name\] = new_meta
>>> print(obj.metadata.items())
[<iRODSMeta 13182 key1 value1 units1>, <iRODSMeta 13183 key1 value2 None>,
<iRODSMeta 13186 key2 value5 units2>]
```

With only one AVU on the object with a name of "key2", *get_one*
is assured of not throwing an exception:

```python
>>> print(obj.metadata.get_one('key2'))
<iRODSMeta 13186 key2 value5 units2>
```

However, the same is not true of "key1":

```python
>>> print(obj.metadata.get_one('key1'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/[...]/python-irodsclient/irods/meta.py", line 41, in get_one
    raise KeyError
KeyError
```

Finally, to remove a specific AVU from an object:

```python
>>> obj.metadata.remove('key1', 'value1', 'units1')
>>> print(obj.metadata.items())
[<iRODSMeta 13186 key2 value5 units2>, <iRODSMeta 13183 key1 value2 None>]
```

Alternately, this form of the `remove()` method can also be useful:

```python
>>> for avu in obj.metadata.items():
...    obj.metadata.remove(avu)
>>> print(obj.metadata.items())
[]
```

If we intended on deleting the data object anyway, we could have just
done this instead:

```
>>> obj.unlink(force=True)
```

But notice that the force option is important, since a data object in
the trash may still have AVUs attached.

At the end of a long session of AVU add/manipulate/delete operations,
one should make sure to delete all unused AVUs. We can in fact use any
`*Meta` data model in the queries below, since unattached AVUs are
not aware of the (type of) catalog object they once annotated:

```python
>>> from irods.models import (DataObjectMeta, ResourceMeta)
>>> len(list( session.query(ResourceMeta) ))
4
>>> from irods.test.helpers import remove_unused_metadata
>>> remove_unused_metadata(session)
>>> len(list( session.query(ResourceMeta) ))
0
```

When altering a fetched iRODSMeta, we must copy it first to avoid
errors, due to the fact the reference is cached by the iRODS object
reference. A shallow copy is sufficient:

```python
>>> meta = album.metadata.items()[0]
>>> meta.units
'quid'
>>> import copy; meta = copy.copy(meta); meta.units = 'pounds sterling'
>>> album.metadata[ meta.name ] = meta
```

Since v1.1.4, `set()` can be used instead:

```python
>>> album.metadata.set( meta )
```

In versions of iRODS 4.2.12 and later, we can also do:

```python
>>> album.metadata.set( meta, \*\*{kw.ADMIN_KW: ''} )
```

or even:

```python
>>> album.metadata(admin = True)\[meta.name\] = meta
```

Since v1.1.5, the "timestamps" keyword is provided to enable the loading
of create and modify timestamps for every AVU returned from the server:

```python
>>> avus = album.metadata(timestamps = True).items()
>>> avus[0].create_time
datetime.datetime(2022, 9, 19, 15, 26, 7)
```

Atomic operations on metadata
-----------------------------

Since iRODS 4.2.8, the atomic metadata API
allows a group of metadata add and remove operations to be performed
transactionally, within a single call to the server. This capability is available
since PRC v0.8.6.

For example, if 'obj' is a handle to an object in the iRODS
catalog (whether a data object, collection, user, or storage resource),
we can send an arbitrary number of AVUOperation instances to be executed
together as one indivisible operation on that object:

```python
>>> from irods.meta import iRODSMeta, AVUOperation
>>> obj.metadata.apply_atomic_operations( AVUOperation(operation='remove', avu=iRODSMeta('a1','v1','these_units')),
...                                       AVUOperation(operation='add', avu=iRODSMeta('a2','v2','those_units')),
...                                       AVUOperation(operation='remove', avu=iRODSMeta('a3','v3')) \# , ...
... )
```

The list of operations are applied in the order given, so that a
"remove" followed by an "add" of the same AVU is, in effect, a
metadata "set" operation. Also note that a "remove" operation will
be ignored if the AVU value given does not exist on the target object at
that point in the sequence of operations.

We can also source from a pre-built list of AVUOperations using
Python's `f(*args_list)` syntax. For example, this
function uses the atomic metadata API to very quickly remove all AVUs
from an object:

```python
>>> def remove_all_avus( Object ):
...     avus_on_Object = Object.metadata.items()
...     Object.metadata.apply_atomic_operations( *[AVUOperation(operation='remove', avu=i) for i in avus_on_Object] )
```

Extracting JSON encoded server information in case of error
-----------------------------------------------------------

Some server APIs, including atomic metadata and replica truncation, can fail for various reasons and generate an
exception.  In these cases the message object returned from the server is made available in the 'server_msg' attribute
of the iRODSException object.

This enables an approach like the following, which logs server information possibly underlying the error that was evoked:

```python
    try:
        Object.metadata.apply_atomic_operations( ops )
        # or:
        DataObject.replica_truncate( size )
    except iRODSException as exc:
        log.error('Server API call failure. Traceback = %r ; iRODS Server info = %r',
            traceback.extract_tb(sys.exc_info()[2]),
            exc.server_msg.get_json_encoded_struct())
```

For `DataObject.replica_truncate(...)`, note that `exc.server_msg.get_json_encoded_struct()` can be used in the exception-handling
code path to retrieve the same information that would have been routinely returned from the truncate call itself, had it actually
completed without error.

Special Characters
------------------

iRODS supports Unicode characters into collection and
data object names. However, certain non-printable ASCII characters, in addition to
the backquote character, have historically presented problems
- especially for clients using the iRODS human readable XML protocol.
Consider this small, contrived application:

```python
    from irods.helpers import make_session

    def create_notes( session, obj_name, content = u'' ):
        get_home_coll = lambda ses: "/{0.zone}/home/{0.username}".format(ses)
        path = get_home_coll(session) + "/" + obj_name
        with session.data_objects.open(path,"a") as f:
            f.seek(0, 2) # SEEK_END
            f.write(content.encode('utf8'))
        return session.data_objects.get(path)

    with make_session() as session:

        # Example 1 : exception thrown when name has non-printable character
        try:
            create_notes( session, "lucky\1.dat", content = u'test' )
        except:
            pass

        # Example 2 (Ref. issue: irods/irods #4132, fixed for 4.2.9 release of iRODS)
        print(
            create_notes(session, "Alice`s diary").name  # The difference in the printed name is subtle, as
                                                         # "`" is transformed to "'" in iRODS pre-4.2.9
        )
```

This creates two data objects, but with less than optimal success. The
first example object is created but receives no content because an
exception is thrown trying to query its name after creation. In the
second example, for iRODS 4.2.8 and before, a deficiency in packStruct
XML protocol causes the backtick to be read back as an apostrophe, which
could create problems manipulating or deleting the object later.

Since v1.1.0, both problems can be mitigated by switching in the
QUASI_XML parser for the default one:

```
    from irods.message import (XML_Parser_Type, ET)
    ET( XML_Parser_Type.QUASI_XML,
        server_version = session.server_version
    )
```

The server_version parameter can be used independently to change the
current thread's choice of entities during QUASI_XML transactions with the server.
(This is only a concern when interacting with servers before iRODS 4.2.9.)

```
    ET(server_version = (4,2,8))
```

Two dedicated environment variables may also be used to customize the
Python client's XML parsing behavior via the setting of global defaults
during start-up.

For example, we can set the default parser to QUASI_XML, optimized for
use with version 4.2.8 of the iRODS server, in the following manner:

```
    Bash-Shell> export PYTHON_IRODSCLIENT_DEFAULT_XML=QUASI_XML PYTHON_IRODSCLIENT_QUASI_XML_SERVER_VERSION=4,2,8
```

Other alternatives for PYTHON_IRODSCLIENT_DEFAULT_XML are
"STANDARD_XML" and "SECURE_XML". These two latter options denote
use of the xml.etree and defusedxml modules, respectively.

Only the choice of "QUASI_XML" is affected by the specification of a
particular server version.

These global defaults, once set, may be overridden on
a per-thread basis using `ET(parser_type, server_version)`.

The current thread's XML parser can always be reverted to the global default by the
explicit use of `ET(None)`.  However, when frequently switching back and forth between
parsers, it may be more convenient to use the `xml_mode` context manager:

```
    # ... Interactions with the server now use the default XML parser.

    from irods.helpers import xml_mode
    with xml_mode('QUASI_XML'):
        # ... Interactions with the server, in the current thread, temporarily use QUASI_XML

    # ... We have now returned to using the default XML parser.
```

Application Cleanup
-------------------

Using the `irods.at_client_exit` module, we may register user-defined functions to be executed at or around the
time when the Python iRODS Client is engaged in object teardown (also called "cleanup") operations.
This is analogous to Python's [atexit module](https://docs.python.org/3/library/atexit.html#module-atexit),
except that here we have the extra resolution to specify that a function or callable object be expressly before,
or expressly after, aforementioned object teardown stage:

```python
    from irods import at_client_exit
    at_client_exit.register_for_execution_after_prc_cleanup(lambda: print("PRC cleanup