vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Most KMS systems support key rotation by replacing the contents of the stored master keys and keeping the MEK ID intact (but updating the key version). After a user or administrator has performed the master key rotation inside the KMS instance using the KMS-specific API, the Parquet key rotation mechanism can be triggered (for example, by the same administrator) by calling:

public static void KeyToolkit.rotateMasterKeys(String folderPath, Configuration hadoopConfig)

When key rotation is performed, every KEK in all key material files in folderPath is unwrapped with the old MEK version to enable decrypting the DEK. A new KEK is generated for each master key ID, and wrapped with the new MEK version. Creating a new KEK, instead of re-using the old one, doesn't only mean that security is improved but also that the performance of subsequent data read operations is increased. This is because a single key rotation process, run by the administrator, operates across multiple files created by different processes, meaning many different KEKs for the same MEK ID. The key rotation process replaces the old KEKs with a single new KEK, which allows the readers to run a single unwrap interaction with KMS for each master key.

For examples of how to use key rotation:



* For key management by application, see [Key rotation in key management by application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-key-management-application-serverlesskey-rotation-key-mgt-application).
* For key management by Key Protect, see [Key rotation key management by Key Protect](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-key-management-key-protect-serverlesskey-rotation-key-mgt-keyprotect).







 Learn more 

Check out the following sample notebook to learn how to use Parquet Encryption:



* [Parquet Encryption by Key Management Application](https://dataplatform.cloud.ibm.com/exchange/public/entry/view/013c690997e27f3a8d91332653054441)
When you use the rotated manage key to decrypt data, the service uses the same version of the key material that was used for encryption before, and then rewraps data by using the latest key material. For more information, see [Rotating your keys](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-rotation). 







 How often should keys be rotated? 

The best practice is to rotate your manage keys regularly. Unified Key Orchestrator allows no more than one key rotation per hour for each key.





 Rewrapping data after rotating a managed key 

After a managed key rotation is complete, new key material becomes available for cryptographic operations. To ensure that your data is protected by the latest version of a managed key, rewrap your data after you rotate a managed key. Depending on your key type, refer to the following table for corresponding instructions.



Table 4. Rewrapping data after key rotation

 Key type Instructions for rewrapping data 

 AWS Key Management Service keys [Rewrapping data with AWS Key Management Service](https://docs.aws.amazon.com/kms/latest/APIReference/API_ReEncrypt.html). 
 Azure Key Vault keys [Azure key vault documentation](https://learn.microsoft.com/en-us/azure/key-vault/). 
 Google Cloud KMS keys [Rewrapping data with Google Cloud KMS](https://cloud.google.com/kms/docs/re-encrypt-data). 
 IBM Cloud KMS keys [Rewrapping data with IBM Cloud Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-rewrap-keys). 
 IBM Key Protect keys [Rewrapping data with IBM Key Protect](https://cloud.ibm.com/docs/key-protect?topic=key-protect-rewrap-keys). 









 What's next
[Rotate a root key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-rotate-keys) that is stored in your Key Protect service.

When you rotate your root key, you replace the key with a new key material.

If Key Protect created the key (the key was not created with the -k parameter) then Key Protect creates a new key. You cannot specify kp key rotate with the -k parameter.

If you created a key and provided the key material ( kp key create key-name -k $KEY_MATERIAL) then you must provide a new key material when you rotate the key (kp key rotate $KEY_ID_OR_ALIAS -k $NEW_KEY_MATERIAL)

You cannot rotate a standard key, that is, a key created using the kp create key-name --standard-key command.

ibmcloud kp key rotate KEY_ID_OR_ALIAS
-i, --instance-id INSTANCE_ID
[--key-ring KEY_RING_ID]
[-k, --key-material KEY_MATERIAL]



 Examples 

These are examples of kp key rotate.



 Example 1 

Rotate a root key.

 create a root key
$ ibmcloud kp key create my-root-key

Creating key: 'my-root-key', in instance: '390086ac-76fa-4094-8cf3-c0829bd69526'...
OK
Key ID Key Name
1a6d5be8-287c-4eb3-9c44-cf0c2b0d67ad my-root-key

 rotate the key
$ ibmcloud kp key rotate 1a6d5be8-287c-4eb3-9c44-cf0c2b0d67ad

Rotating root key...
OK

 show key details
$ ibmcloud kp key show 1a6d5be8-287c-4eb3-9c44-cf0c2b0d67ad
[Actions icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Details. If there was an issue with the request, the Status field provides information about why the rotation did not complete successfully.
8. Redeploy the latest certificate version to your TLS termination point.

To access the current version, you can [download the certificate](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-access-secrets) or retrieve it programmatically by using the [Get a secret](https://cloud.ibm.com/apidocs/secrets-manager/secrets-manager-v2get-secret) API.







 Creating new versions of public certificates with your own DNS provider in the UI 

To rotate a public certificate that was created by using a manual DNS provider in the UI, complete the following steps.



1. In the console, click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/icon_hamburger.svg)> Resource List.
2. From the list of services, select your instance of Secrets Manager.
3. In the row for the certificate that you want to rotate, click the Actions menu ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Rotate.
4. Optional: Add a name and description to easily identify your certificate.
5. Optional: Click Update.
6. Click Challenges to access the TXT record name and value that are associated with each of your domains.
The key algorithm that you select determines the encryption algorithm (RSA or ECDSA) and key size to use to generate keys and sign certificates. For longer living certificates, it is recommended to use longer key lengths to provide more encryption protection. Options include RSA2048, RSA4096, ECDSA256, and ECDSA384.
12. Optional: Enable advanced options for the certificate.



1. To bundle your issued certificate with intermediate certificates, switch the bundle toggle to On. After your certificates are bundled, they can no longer be unbundled.
2. To enable automatic rotation for the certificate, switch the rotation toggle to On. Your certificate is rotated 31 days before it expires.
3. To request a new private key with the certificate on each rotation, switch the rekey toggle to On.



13. Select a DNS provider configuration.

The configuration that you select determines the DNS provider to validate the ownership of your domains. To view the configurations that are defined for your instance, you can go to Secrets engines > Public certificates.
14. Add the domains to include in your request.

You can include up to 100 domains, subdomains, or wildcards. The Common Name, or fully qualified domain name of the certificate, can't exceed 64 characters in length. A wildcard can be selected as the Common Name.



1. Click Select domains.
2. From your list of domains, select the Common Name of the certificate.



15. Click Next.
16. Review the details of your certificate.
17. Click Add.

When you order a certificate, domain validation takes place to verify the ownership of your selected domains. This process can take a few minutes to complete. After you submit your certificate details, Secrets Manager sends your request to the selected certificate authority. After a certificate is issued, you can deploy it to your integrated apps, download it, or rotate it manually. Your private key for SSL/TLS is generated directly in Secrets Manager and stored securely.

Need to check your order status?
30 September 2020 

Added: Master key rotation
: You can now rotate your master key on demand by using the IBM Cloud® Trusted Key Entry CLI plug-in so as to meet industry standards and cryptographic best practices. For more information about how it works, see [Master key rotation introduction](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-master-key-rotation-intro).

For the detailed instructions, see [Rotating master keys](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-rotate-master-key-cli-key-part).

Added: Support for performing cryptographic operations with the standard PKCS #11 API
: Hyper Protect Crypto Services now supports performing cryptographic operations with the standard Public-Key Cryptography Standards (PKCS) #11 API.

With the support of PKCS #11 API, you don not need to change your existing applications that use PKCS #11 standard to make it run in the Hyper Protect Crypto Services cloud HSM environment. The PKCS #11 library accepts the PKCS #11 API requests from your applications and remotely accesses the cloud HSM to execute the corresponding cryptographic functions.

For more information about the PKCS #11 API use cases, see

[Using Hyper Protect Crypto Services as PKCS #11 HSMs](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-use-casespkcs11_hsm).

To learn more about the PKCS #11 API, see

[Introducing PKCS #11](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-pkcs11-intro) and

[PKCS #11 API reference](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-pkcs11-api-ref).





 31 August 2020 

Added: Support for import tokens to securely upload encryption keys
ibmcloud secrets-manager secret-version-unlock --secret-type=arbitrary --id=exampleString --version-id=exampleString --locks=exampleString







 ibmcloud secrets-manager all-locks 

List the lock details that are associated with all secrets in your Secrets Manager instance.

ibmcloud secrets-manager all-locks [--limit LIMIT] [--offset OFFSET] [--search SEARCH] [--groups GROUPS]



 Command options 

--limit (int64)
: The number of secrets with associated locks to retrieve. By default, list operations return the first 25 items. To retrieve a different set of items, use limit with offset to page through your available resources.

Usage: If you have 20 secrets in your instance, and you want to retrieve only the first 5, use ..?limit=5.

The maximum value is 100. The minimum value is 1.

--offset (int64)
: The number of secrets to skip. By specifying offset, you retrieve a subset of items that starts with the offset value. Use offset with limit to page through your available resources.

Usage: If you have 100 secrets in your instance, and you want to retrieve secrets 26 through 50, use ..?offset=25&limit=25.

The minimum value is 0.

--search (string)
: Filter locks that contain the specified string in the field "name".

Usage: If you want to list only the locks that contain the string "text" in the field "name", use ..?search=text.

The maximum length is 64 characters.

--groups ([]string)
: Filter secrets by groups.

You can apply multiple filters by using a comma-separated list of secret group IDs. If you need to filter secrets that are in the default secret group, use the default keyword.

Usage: To retrieve a list of secrets that are associated with an existing secret group or the default group, use ..?groups={secret_group_ID},default.
Secrets Manager integration 

IBM Cloud Secrets Manager helps you to create secrets dynamically and lease them to applications while you control access from a single location. With Secrets Manager, you can create, lease, and centrally manage secrets that are used in App Configuration service or your custom-built applications.

By integrating App Configuration with Secrets Manager, instead of storing secrets, App Configuration stores metadata of Secrets Manager instance. This metadata is called as a secret reference as part of the Properties type in App Configuration.

Take these steps to get Secrets Manager integrated with App Configuration:



1.  Log in to your IBM Cloud account.
2.  Create a Secrets Manager instance and create your secrets.

If you already have a Secrets Manager instance, go to next step else create a Secrets Manager instance. For more information, see [creating a Secrets Manager instance](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-create-instance&interface=ui).
3.  Check the permissions in Secrets Manager instance.

Make sure you have a Secrets Manager instance with Viewer access to the resource group where Secrets Manager instance is created or exists. Also, you need to have Reader access to the Secrets Manager instance.

For more information about providing user authorizations, see [here](https://cloud.ibm.com/docs/account?topic=account-serviceauth&interface=ui).
4.  Create a property of type Secret reference. For more information, see [Properties](https://cloud.ibm.com/docs/app-configuration?topic=app-configuration-ac-properties).
5.  Use the App Configuration SDKs to connect your application to retrieve and use secrets from Secrets Manager. For more information on how to access secrets with App Configuration Node sdk, see [here](https://cloud.ibm.com/docs/app-configuration?topic=app-configuration-ac-integrate-sdksac-integrate-node-evaluate-secret-property).
* UI
* API







 Managing data encryption 

In IBM Cloud®, you can secure your resources with your own root keys. You can rotate your root keys regularly or manually as needed. You can verify key rotation by viewing Activity Tracker events. You can disable or delete root keys that are compromised or that you no longer need. You can restore root keys that were deleted within 30 days of their deletion.



 Manage root keys 

Manage your root keys by taking the following actions:



* View the association of root keys to the resources that they protect by [viewing root key registrations](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-managingbyok-root-key-registration).
* View key rotation information by looking at [root key registration](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-managingbyok-root-key-verify-rotation) in the KMS instance.
* [Rotate your root keys](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-key-rotation) at regular intervals or manually rotate imported root keys. Shortening the crypto period of a key reduces the possibility of a security breach.
* Decide whether importing your own HSM root key or generating a root key by the KMS is preferable. If you want to set up a rotation policy for automatic key rotation, you must use KMS-generated root keys.
* See what happens to your [root key state](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-managingbyok-root-key-states) when you take certain actions, such as disabling the key.
To have more control over your application secrets, you want the ability to store your cluster secrets in an external secrets management service, where you can [encrypt them at rest](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-mng-data), [monitor their activity](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-at-events), and easily manage them.

With Secrets Manager, you can centralize and secure the secrets that are used by the apps that run in your Kubernetes clusters. Rather than injecting your secrets at deployment time, you can configure your apps to securely retrieve secrets from Secrets Manager at run time. When it's time to rotate the secret, you can do so from Secrets Manager. For example, consider the following scenario:

Zoom

![The diagram shows the basic flow between Secrets Manager and your Kubernetes cluster.](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/secrets-manager/images/iks-external-secrets-flow.svg)

Figure 1. External Secrets flow



1. As a developer, you use Secrets Manager to store a secret for an application that you want to deploy in a Kubernetes cluster.
2. Secrets Manager provides an ID for the secret. You include the ID in the ExternalSecrets configuration file for your app and you apply the configuration to the cluster.
3. The External Secrets controller fetches the ExternalSecrets objects in the configuration file that you defined by using the Kubernetes API.
4. At application run time, the controller retrieves the secret data from Secrets Manager, and converts the ExternalSecrets objects to Kubernetes secrets for your cluster.



This scenario features a third-party tool that can impact the compliance readiness of workloads that run in your Kubernetes cluster.
Rotating master keys by using recovery crypto units 

To rotate the master key by using recovery crypto units, follow these steps. In this case, a random master key value is generated in a recovery crypto unit, securely transferred to other crypto units, and rotated automatically with the ibmcloud tke auto-mk-rotate command.

When master key rotation is taking place, you are temporarily not able to access your keystore. To learn how master key rotation works, see the introduction to the master key rotation [for the standard plan](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-master-key-rotation-intro) or [for the Unified Key Orchestrator plan](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-uko-master-key-rotation-intro).

Make sure that you are assigned the Manager service access role or the Crypto unit administrator role to perform TKE CLI operations. For more information about the access management, see Managing user access [for the standard plan](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-manage-access) or [for the Unified Key Orchestrator plan](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-uko-manage-access).

When the master key is being rotated, you can still perform some KMS key actions such as listing keys, retrieving key metadata, or deleting keys, but you cannot create or rotate keys. You cannot call either the PKCS #11 API or GREP11 API during the master key rotation.



 Rotating master keys 

Before you rotate the master key by using recovery crypto units, make sure that you complete the [steps to set up the IBM Cloud CLI with TKE plug-in](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-initialize-hsm-prerequisite).

To rotate master keys by using recovery crypto units, use the following command:

ibmcloud tke auto-mk-rotate
(https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-what-is-secret)[![carousel thumbnail](https://cloud.ibm.com/docs-content/v1/content/secrets-manager//images/secure-manage-secrets-video-thumbnail.png)<br><br>Securely managing your cloud secrets with Secrets Manager](https://www.youtube.com/watch?v=rOp7aGyavnk)

 Latest updates 

[View more](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-release-notes)17 April 2023 | Now available: new API and CLI versions

The new API, SDK, Terraform, and CLI versions offer more options to manage your Secrets Manager instance. The Secrets Manager API v1 has been deprecated in favor of v2.

3 March 2023 | Now available: Terraform support

Secrets Manager now supports Terraform.

11 December 2022 | Support for context-based restrictions (CBR)

Manage user and service access to your Secrets Manager resources by using context-based restrictions.

14 November 2022 | Manually configure your own DNS provider

Your DNS provider isn't integrated with the service? You can now manually configure it by using the UI.

18 October 2022 | Add custom metadata to your secret versions and auto-rotate IAM credentials

Do you need more space to log details about your secret versions? Add relevant information as metadata.
Rotate key doesn't have any additional fields outside of those from the [Common Fields](https://cloud.ibm.com/docs/key-protect?topic=key-protect-at-eventsat-events-common-fields) section





 Patch key 

The following fields include extra information:



* The requestData.initialValue.keyRingId field includes the ID of the key ring that the key previously was a part of.
* The requestData.newValue.keyRingId field includes the ID of the key ring that the key is currently a part of.







 Purge key 

The following fields include extra information:



* The responseData.deletionDate field represents the date the key was deleted.
* The responseData.purgeAllowedFrom field represents the date from which the purge action was allowed.
* The responseData.purgeEligibleOn field represents the date on which the key metadata is eligible to be permanently removed from the system by Key Protect.







 Get key total 

The following field includes extra information:



* The responseData.totalResources field includes the total amount of keys within the Key Protect instance.







 List keys 

The following field includes extra information:



* The responseData.totalResources field includes the total amount of keys returned in the response.







 Get key or key metadata 

The following fields include extra information:



* The requestData.keyType field includes the type of key that was retrieved.
* The responseData.keyState field includes the integer that correlates to the state of the key.
* The responseData.keyVersionId field includes the unique identifier of the key version used to wrap input ciphertext on wrap requests.
* The responseData.keyVersionCreationDate field includes the date that the current version of the key was created.
* The responseData.expirationDate includes the date that the key will expire on.







 List key versions 

The following field includes extra information:



* The responseData.totalResources field includes the total amount of key versions returned in the response.







 Set or Unset key for deletion 

The following fields include extra information:
5. Select or enter the new certificate data.

Keep in mind that manually rotating a certificate replaces the content of the certificate with the new data that you provide only. Private keys and intermediate certificates from previous versions are not retained.
6. To rotate the certificate immediately, click Rotate.
7. Optional: Check the version history to view the latest updates.

In the row of the secret that you rotated, click the Actions menu ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/actions-icon-vertical.svg)> Version history to verify that a new version was created successfully.
8. Redeploy the latest certificate version to your TLS termination point.

To access the current version, you can [download the certificate](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-access-secrets) or retrieve it programmatically by using the [Get a secret](https://cloud.ibm.com/apidocs/secrets-manager/secrets-manager-v2get-secret) API.







 Creating new versions of public certificates 

If your Secrets Manager service instance is enabled for [public certificates](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-public-certificatesorder-public-certificates), you can manually renew a certificate that was previously ordered from a third-party certificate authority.



1. In the console, click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/icon_hamburger.svg)> Resource List.
2. From the list of services, select your instance of Secrets Manager.
3. In the Secrets Manager UI, go to the Secrets list.
4.