vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* You can manually rotate your root keys with the API or from the CLI, or set up a rotation policy to automatically rotate your keys. Rotating keys replaces the root key's original cryptographic material and generates new material. For more information, see [Key rotation for VPC resources](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-key-rotation).
* Customer-managed encryption provides better audit records for root key usage. Your key access is logged in the [Activity Tracker](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-managingbyok-activity-tracker-events) service. With the Activity Tracker, you can track interactions with the VPC, such as when a root key is rotated.







 IBM Cloud VPC encryption technology 

Encryption is handled by IBM Cloud VPC's hypervisor technology for your instances. This feature provides greater level of security over solutions that provide only storage node encryption-at-rest. Data is always encrypted with envelope encryption within the IBM Cloud.

Stock and custom images use QEMU Copy On Write Version 2 (QCOW2) file format. LUKS encryption format secures the QCOW2 format files. IBM Cloud uses the AES-256 cipher suite and XTS cipher mode options with LUKS. This combination provides a much greater level of security than AES-CBC, along with better management of passphrases for key rotation, and provides key replacement options if your keys are compromised.

In total, four keys protect your data:



* An IBM-managed key encrypts your data in the backend storage system. IBM-managed encryption on the storage system is always applied, even when you use customer-managed encryption. This key protects your data while in transit and while at rest.

Encryption of the network link between your workload outside of IBM Cloud and a workload inside IBM Cloud is your responsibility. For more information, see [Encryption in Security and regulation compliance](https://cloud.ibm.com/docs/vpc?topic=vpc-responsibilities-vpcsecurity-compliance).
With TLS secrets, you can specify one certificate CRN. With non-TLS secrets, you can specify multiple fields to pull non-certificate secrets. If you do not specify a secret type when you create a secret, TLS is applied by default. For more information on supported secrets, see [Working with secrets of different types](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-what-is-secretsecret-types).

Are secrets that are stored in a registered Secrets Manager instance automatically updated?
: Yes. If you have a Secrets Manager instance registered to your cluster, the secrets on the cluster are automatically updated with the values from Secrets Manager once a day. These updates are made using the value of the the secret from the corresponding CRN.

Are my secrets automatically updated if I do not create and register a Secrets Manager instance?
: If you do not have a Secrets Manager instance registered to your cluster, your default Ingress secrets continue to update automatically every 90 days and are applied to your cluster. However, any secrets you created that reference the default Ingress secret are not automatically updated.
: Example scenario: You have a default Ingress certificate in the default namespace. You run the ibmcloud ks ingress secret create command and reference the CRN of the default Ingress certificate to mirror the certificate in the istio-system namespace. Without a Secrets Manager instance, the default Ingress certificate in the default namespace automatically updates. However, you are responsible for regularly updating the certificate in the istio-system namespace with the kubectl commands or another rotation method.

I created secrets that reference the default Ingress certificate, but I have not created and registered a Secrets Manager instance. How do I manage my secrets?
: If you don't register a Secrets Manager instance, IBM Cloud Kubernetes Service only automatically updates the default Ingress secret. You are responsible for managing any other secrets by using kubectl commands or another rotation method. If you have any secrets that reference the default Ingress certificate, you should remove them by using ibmcloud ks ingress secret rm.
2. In the row for the secret that you want to edit, click the Actions menu ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Edit details.
3. Use the Automatic rotation option to enable or disable automatic rotation for the secret.





When you update a secret's rotation settings, you trigger an immediate rotation.





 Setting an automatic rotation policy for public certificates 

If you prefer to schedule your public SSL/TLS certificates to be automatically renewed, you can enable automatic rotation for certificates when you order them. You can also enable auto rotation by editing the details of an existing certificate. In the certificate's next rotation cycle, Secrets Manager begins attempting to reorder the certificate 31 days before its expiry date. The service continues to attempt to renew the certificate daily until it is successful.



1. If you're [ordering a public certificate](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-public-certificatesorder-public-certificates), enable the rotation options.



1. To rotate the certificate automatically, switch the rotation toggle to On. Your certificate is automatically reordered 31 days before its expiration date.
2. To request a new private key for the certificate on each rotation, switch the rekey toggle to On.



2. If you're editing an existing public certificate, schedule automatic rotation by updating its details.



1. In the Secrets table, view a list of your existing Public certificates.
2. In the row for the certificate that you want to edit, click the Actions menu ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Edit details.
3.
* private_cert_configuration_action_rotate_crl: Rotate the certificate revocation list (CRL) of an intermediate certificate authority.



ibmcloud secrets-manager configuration-action-create --name NAME --config-action-prototype CONFIG-ACTION-PROTOTYPE [--x-sm-accept-configuration-type X-SM-ACCEPT-CONFIGURATION-TYPE]



 Command options 

--name (string)
: The name of the configuration. Required.

The maximum length is 128 characters. The minimum length is 2 characters. The value must match regular expression /^[A-Za-z0-9](?:_-\.[A-Za-z0-9]+)$/.

--config-action-prototype ([ConfigurationActionPrototype](https://cloud.ibm.com/docs/secrets-manager-cli-plugin?topic=secrets-manager-cli-plugin-secrets-manager-clicli-configuration-action-prototype-example-schema))
: The request body to specify the properties of the action to create a configuration. Required.

Provide a JSON string option or specify a JSON file to read from by providing a filepath option that begins with a @, e.g. --config-action-prototype=@path/to/file.json.

--x-sm-accept-configuration-type (string)
: The configuration type of this configuration - use this header to resolve 300 error responses.

Allowable values are: public_cert_configuration_ca_lets_encrypt, public_cert_configuration_dns_classic_infrastructure, public_cert_configuration_dns_cloud_internet_services, iam_credentials_configuration, private_cert_configuration_root_ca, private_cert_configuration_intermediate_ca, private_cert_configuration_template.





 Example
* private_cert_configuration_action_rotate_crl: Rotate the certificate revocation list (CRL) of an intermediate certificate authority.



ibmcloud secrets-manager configuration-action-create --name NAME --config-action-prototype CONFIG-ACTION-PROTOTYPE [--x-sm-accept-configuration-type X-SM-ACCEPT-CONFIGURATION-TYPE]



 Command options 

--name (string)
: The name of the configuration. Required.

The maximum length is 128 characters. The minimum length is 2 characters. The value must match regular expression /^[A-Za-z0-9](?:_-\.[A-Za-z0-9]+)$/.

--config-action-prototype ([ConfigurationActionPrototype](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-cli-plugin-secrets-manager-clicli-configuration-action-prototype-example-schema))
: The request body to specify the properties of the action to create a configuration. Required.

Provide a JSON string option or specify a JSON file to read from by providing a filepath option that begins with a @, e.g. --config-action-prototype=@path/to/file.json.

--x-sm-accept-configuration-type (string)
: The configuration type of this configuration - use this header to resolve 300 error responses.

Allowable values are: public_cert_configuration_ca_lets_encrypt, public_cert_configuration_dns_classic_infrastructure, public_cert_configuration_dns_cloud_internet_services, iam_credentials_configuration, private_cert_configuration_root_ca, private_cert_configuration_intermediate_ca, private_cert_configuration_template.





 Example
What is a secrets engine? 

Secrets enginesare components in IBM Cloud® Secrets Manager that are used to process operations for secrets of different types.



 Learning about engine configurations 

In Secrets Manager, a secrets engine serves as the back end for a specific type of secret, such as a password or an API key. Depending on its type, a secrets engine can store data, generate secrets on demand, and more.

Some engines require a configuration before you can enable them for your instance. For example, when you try to [order SSL/TLS certificates](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-public-certificates) or create [IAM credentials](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-iam-credentials) for the first time, a message is displayed to show that an engine configuration is required.

Zoom

![The image shows the secrets selection screen in the Secrets Manager UI.](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/secrets-manager/images/engine-required.svg)

Figure 1. Example - Engine configuration required





 What's in a configuration? 

Engine configurations that you add to your instance contain the properties that the service can use to enable operations for a secret type. For example, if you define [certificate authority (CA) and DNS provider configurations](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-add-certificate-authority) for your instance, you can use Secrets Manager to order public certificates (public_cert).

Check out the following image to see how a configuration is structured.

Zoom

![This image shows the components of a secrets engine configuration. The information in the image is detailed in the surrounding content.]
: You can now rotate master keys on demand to meet industry standards and cryptographic best practices in your Hyper Protect Crypto Services with Unified Key Orchestrator instance. You can understand how keys are protected during master key rotation and use the IBM Cloud console to view the progress. For more information about how master key rotation works, see [Master key rotation introduction](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-uko-master-key-rotation-intro).

Added: Master key rotation for EP11 keystores
: You are now able to rotate master keys when your service instance has EP11 keystores enabled. Previously, this function is not available.

Both of the functions above are currently only supported in the Frankfurt region in Germany.





 1 Feb 2023 

Added: Hyper Protect Crypto Services key management functions
: The Hyper Protect Crypto Services key management service API is updated to version 22.11. The following functions are added:



* [List Keys with sorting](https://cloud.ibm.com/apidocs/hs-crypto::text=values%3A%20length%20%E2%89%A4%20256-,sort,-string) to include lastRotateDate sorting.
* [List Keys with advanced filtering](https://cloud.ibm.com/apidocs/hs-crypto::text=Default%3A%20id-,filter,-string) to including lastRotateDate filtering.
* [Create key with policy overrides](https://cloud.ibm.com/apidocs/hs-cryptocreatekeywithpoliciesoverrides) to enable users with Manager role to create keys with policies in a single call, overriding instance level policies.
Master key rotation - Standard Plan 

After you load a master key to your Hyper Protect Crypto Services instance, you can rotate the master key on demand to meet industry standards and cryptographic best practices.

A master key is used to wrap encryption keys that are managed in the service instance. With the master key rotation, you retire the original master key and load a new master key that reencrypts the entire key storage.

When the master key is being rotated, you can still perform some KMS key actions such as listing keys, retrieving key metadata, or deleting keys, but you cannot create or rotate keys. You cannot call either the PKCS #11 API or GREP11 API during the master key rotation.



 How master key rotation works 

Master key rotation works by securely transferring the value between two types of master key registers in crypto units: new master key register and current master key register. Depending on [the approach that you use to initialize your service instance](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-initialize-instance-mode), the rotation process is slightly different.

Key objects in the in-memory keystore are not automatically rotated after the master key rotation. If PKCS #11 keystores are enabled in your service instance, you need to restart all active PKCS #11 applications to clear the in-memory keystore after the master key rotation is complete. For detailed information, see

[PKCS #11 implementation components](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-uko-pkcs11-introuko-pkcs11-components).



 Rotating master keys by using smart cards and Management Utilities 

Master keys that are created with the Management Utilities can be rotated by using the smart cards, where master key parts are stored. Before you rotate the master key, you need to create the key parts that you are going to use.

You can create a new master key value using either 2 or 3 key parts. To be able to rotate the master key, the current master key registers must be in Valid state with the same verification pattern and the new master key registers must be Empty.
* private_cert_configuration_action_rotate_crl: Rotate the certificate revocation list (CRL) of an intermediate certificate authority.



ibmcloud secrets-manager configuration-action-create --name NAME --config-action-prototype CONFIG-ACTION-PROTOTYPE [--x-sm-accept-configuration-type X-SM-ACCEPT-CONFIGURATION-TYPE]



 Command options 

--name (string)
: The name of the configuration. Required.

The maximum length is 128 characters. The minimum length is 2 characters. The value must match regular expression /^[A-Za-z0-9](?:_-\.[A-Za-z0-9]+)$/.

--config-action-prototype ([ConfigurationActionPrototype](https://cloud.ibm.com/docs/cli?topic=cli-secrets-manager-clicli-configuration-action-prototype-example-schema))
: The request body to specify the properties of the action to create a configuration. Required.

Provide a JSON string option or specify a JSON file to read from by providing a filepath option that begins with a @, e.g. --config-action-prototype=@path/to/file.json.

--x-sm-accept-configuration-type (string)
: The configuration type of this configuration - use this header to resolve 300 error responses.

Allowable values are: public_cert_configuration_ca_lets_encrypt, public_cert_configuration_dns_classic_infrastructure, public_cert_configuration_dns_cloud_internet_services, iam_credentials_configuration, private_cert_configuration_root_ca, private_cert_configuration_intermediate_ca, private_cert_configuration_template.





 Example
Update the metadata of a secret, such as its name or description.

To update the actual contents of a secret, rotate the secret by using the Invoke an action on a secret method.

ibmcloud secrets-manager secret-metadata-update --secret-type SECRET-TYPE --id ID --metadata METADATA --resources RESOURCES



 Command options 

--secret-type (string)
: The secret type. Required.

Allowable values are: arbitrary, iam_credentials, imported_cert, public_cert, private_cert, username_password, kv.

--id (string)
: The v4 UUID that uniquely identifies the secret. Required.

The value must match regular expression /[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}/.

--metadata ([CollectionMetadata](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-secrets-manager-cli-v1cli-collection-metadata-example-schema-v1))
: The metadata that describes the resource array. Required.

--resources ([SecretMetadata]](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-secrets-manager-cli-v1cli-secret-metadata-example-schema-v1))
: A collection of resources. Required.





 Examples 

ibmcloud secrets-manager secret-metadata-update --secret-type=arbitrary --id=exampleString --metadata='{"collection_type": "application/vnd.ibm.secrets-manager.secret+json", "collection_total": 1}' --resources='[{"labels": "dev","us-south"], "name": "updated-secret-name", "description": "Updated description for this secret."
sc.hadoopConfiguration.set("parquet.encryption.kms.client.class", "com.ibm.parquet.key.management.InMemoryKMS")
sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")

KeyToolkit.rotateMasterKeys("<path to encrypted files>", sc.hadoopConfiguration)

Note: Key rotation can be performed only for files written with "parquet.encryption.key.material.store.internally" parameter set to "false".
Root key rotation works by securely transitioning key material from an Active to a Deactivated key state. To replace the deactivated or retired key material, new key material moves into the Active state and becomes available for cryptographic operations. For more information about different key states, see [Monitoring the lifecycle of encryption keys](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-key-states).



 Using Hyper Protect Crypto Services to rotate root keys 

Keep in mind the following considerations as you prepare to use Hyper Protect Crypto Services for rotating root keys.

Rotating root keys that are generated in Hyper Protect Crypto Services
: You can use Hyper Protect Crypto Services to rotate a root key that was generated in Hyper Protect Crypto Services by setting a rotation policy for the key, or by rotating the key on demand. The metadata for the root key, such as the key ID, does not change when you rotate the key.

Rotating root keys that you bring to the service
: To rotate a root key that you initially imported to the service, you must generate and provide new key material for the key. You can use Hyper Protect Crypto Services to rotate imported root keys on demand by supplying new key material as part of the rotation request. The metadata for the root key, such as the key ID, does not change when you rotate the key. Because you must provide new key material to rotate an imported key, automatic rotation policies are not available for root keys that have imported key material.

Managing retired root key material
: Hyper Protect Crypto Services creates new key material after you rotate a root key. The service retires the old key material and retains the retired versions until the root key is deleted. When you use the root key for envelope encryption, Hyper Protect Crypto Services uses only the latest key material that is associated with the key. The retired key material can no longer be used to protect keys, but it remains available for unwrap operations. If Hyper Protect Crypto Services detects that you're using retired key material to unwrap DEKs, the service provides a newly wrapped DEK that's based on the latest root key material.

Enabling root key rotation for IBM Cloud data services
Managed key rotation - Unified Key Orchestrator Plan 

You can manually rotate managed keys in your IBM Cloud® Hyper Protect Crypto Services with Unified Key Orchestrator instance on demand. Key rotation takes place when you retire the original key material of the managed key, and you re-key it by generating a new cryptographic key material.

Rotating keys regularly helps you meet industry standards and cryptographic best practices. The following table describes the main benefits of key rotation:



Table 1. Describes the benefits of key rotation

 Benefit Description 

 Cryptoperiod management for keys Key rotation limits how long your information is protected by a single key. By rotating your managed keys at regular intervals, you also shorten the cryptoperiod of the keys. The longer the lifetime of an encryption key, the higher the probability for a security breach. 
 Incident mitigation If your organization detects a security issue, you can immediately rotate the key to mitigate or reduce costs that are associated with key compromise. 



Key rotation is treated in the NIST Special Publication 800-57, Recommendation for Key Management. To learn more, see [NIST SP 800-57 Pt. 1 Rev. 5](https://csrc.nist.gov/publications/detail/sp/800-57-part-1/rev-5/final)

Note that key rotation might charge extra fees depending on the type of your managed key. For the pricing details, refer to the following links:



* AWS Key Management Service keys: [AWS Key Management Service Pricing](https://aws.amazon.com/kms/pricing/)
* Azure Key Vault keys: [Azure Key Vault pricing](https://azure.microsoft.com/en-us/pricing/details/key-vault/)
* Google Cloud KMS keys: [Google Cloud Key Management Service Pricing](https://cloud.google.com/security-key-managementsection-11).
* IBM Cloud KMS keys: No extra fees.
When reading a Parquet file, the identifier of the master encryption key (MEK) and the encrypted key encryption key (KEK) with its identifier, and the encrypted data encryption key (DEK) are extracted from the file metadata.

The key encryption key is decrypted with the master encryption key, either locally if the master keys are managed by the application, or in a KeyProtect service if the master keys are managed by IBM® Key Protect for IBM Cloud®. The key encryption keys are cached by access token, so there is no need to interact with the KeyProtect service for each decrypted column or file if they use the same key encryption key. Then the data encryption key (DEK) is decrypted locally, using the key encryption key (KEK).





 Key Rotation (optional feature) 

Envelope encryption practice has an advanced option to rotate (change) master keys to minimize the risk of sensitive data leakage due to compromised MEKs. DEKs can’t be changed, because this means encrypting the Parquet files entirely again. The new MEKs (and transparently re-generated KEKs) are used to re-encrypt the DEKs.

To enable key rotation in Parquet files, you must set the "parquet.encryption.key.material.store.internally" parameter to "false" in the Hadoop configuration when writing the parquet files.

With this parameter set, Parquet stores the keys (wrapped with MEKs) in separate small files, instead of inside the Parquet files. When key rotation is performed, only the small key material files are replaced. No need to modify the Parquet files (which are often immutable). Most KMS systems support key rotation by replacing the contents of the stored master keys and keeping the MEK ID intact (but updating the key version). After a user or administrator has performed the master key rotation inside the KMS instance using the KMS-specific API, the Parquet key rotation mechanism can be triggered (for example, by the same administrator) by calling:

public static void KeyToolkit.rotateMasterKeys(String folderPath, Configuration hadoopConfig)