vania-janet/multiturn-rag-retrieval-data huggingface dataset:
For performance reasons, create both the Key Protect and Data Engine instances in the same IBM Cloud region.
4. Give the new Data Engine instance [access](https://cloud.ibm.com/docs/account?topic=account-serviceauth) to your Key Protect key.



* Open your IBM Cloud dashboard.
* From the menu bar, click Manage > Access (IAM), and select Authorizations.
* Click Create.
* In the source service menu, select IBM Cloud Data Engine.
* In the source service instance menu, select the service instance to authorize.
* In the target service menu, select IBM Key Protect.
* In the target service instance menu, select the service instance to authorize.
* Enable the Reader role.
* Click Authorize.









 Further considerations 

Customer key encryption in Data Engine applies to the queries that you are processing. With each query, you explicitly specify IBM Cloud® Object Storage locations for input and target data, which is not controlled by Data Engine. So, if you are processing sensitive data, make sure that your query results are written to an IBM Cloud Object Storage location that has appropriate protection. Default target locations, where results are stored by default if no other result locations are specified, are not and cannot be encrypted.

IBM Cloud Object Storage documents how to configure customer key encryption for the Cloud Object Storage buckets in [Managing encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryption) storing the actual data.

If you use Key Protect to encrypt your queries, IBM staff cannot view the encrypted query texts and error messages. Therefore, provide this data explicitly to IBM service in a support case, so you can make sure that no sensitive information is exposed.





 Deleting a key or an instance 

Deleting a key or the containing Key Protect instance from the system will shred its contents and any data still encrypted with that key. When it is removed, it cannot be undone or reversed. A Data Engine instance that is associated with a deleted key cannot be used for any further queries. You need to create a Data Engine instance for future queries.
You can safeguard and encrypt your information from corruption, compromise, or loss in Schematics by:



* [KMS integration for BYOK or KYOK](https://cloud.ibm.com/docs/schematics?topic=schematics-kms-integration)
* [Managing data encryption](https://cloud.ibm.com/docs/schematics?topic=schematics-secure-datapi-encrypt)
* Restrict network access for all the resources to provision by [Allowing specific IP addresses](https://cloud.ibm.com/docs/schematics?topic=schematics-allowed-ipaddresses).









 Next steps 

Check out the [Schematics use cases](https://cloud.ibm.com/docs/schematics?topic=schematics-how-it-works).
Name A human-readable alias for easy identification of your key. Length must be within 2 - 90 characters (inclusive). <br> <br>To protect your privacy, ensure that the key name does not contain personally identifiable information (PII), such as your name or location. Note that key names do not need to be unique. 
 Key material The base64-encoded key material, such as an existing key-wrapping key, that you want to store and manage in the service. For more information, check out [Base64 encoding your key material](https://cloud.ibm.com/docs/key-protect?topic=key-protect-import-standard-keyshow-to-encode-standard-key-material). Ensure that the key material is 16, 24, or 32 bytes long, and corresponds to 128, 192, or 256 bits in length. The key must also be base64-encoded. 
 Key alias Optional. [Key aliases](https://cloud.ibm.com/docs/key-protect?topic=key-protect-create-key-alias) are ways to describe a key that allow them to be identified and grouped beyond the limits of a display name. Keys can have up to five aliases. 
 Key ring Optional. [Key rings](https://cloud.ibm.com/docs/key-protect?topic=key-protect-grouping-keys) are groupings of keys that allow those groupings to be managed independently as needed. Every key must must be a part of a key ring. If no key ring is selected, keys are placed in the default key ring. Note that to place the key you're creating in a key ring, you must have the Manager role over that key ring. For more information about roles, check out [Managing user access](https://cloud.ibm.com/docs/key-protect?topic=key-protect-manage-access). 



When you are finished filling out the key's details, click Import key to confirm.
To apply an RBAC policy without encrypting, use the Clear Selections option, then select the RBAC policy.



1. Optional: Specify a Key ID for each column from the drop-down menu or accept the default.
2. Click REVIEW at the bottom left panel to review your selections.
3. Review your policy and select a Migration Plan:
4. Save: Saves the data schema for future use.
5. Deploy: Establishes a data schema for an environment that was not rocessed through Data Security Broker migration and data type conversion but does not migrate the data.
6. Deploy and Migrate: Establishes a data schema and transforms the existing data in the data store.



NOTE: If you have only configured RBAC Policies, without specifying encryption, then the Deploy and Migrate option is not available. This is because RBAC does not transform the underlying data.
* [Setting up a VPE for Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-virtual-private-endpoints-for-vpcvirtual-private-endpoints-for-vpc-setup)
* [Using your VPE for Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-virtual-private-endpoints-for-vpcuse-vpe-for-hpcs)



[Using service endpoints to privately connect to Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionsecure-connection)



* [Understanding the network access policy](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionunderstand-network-access-policies)
* [Before you begin](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionprivate-endpoint-prereqs)
* [Step 1: Configure the private network of IBM Cloud on your virtual server](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionconfigure-network)
* [Step 2: Provision a service instance and select the network access](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionservice-endpoint-private-endpoints)
* [Step 3: Target the Hyper Protect Crypto Services private endpoint for the TKE CLI plug-in](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectiontarget-tke-private-endpoint)
* [Step 4: Initialize the service instance](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-secure-connectionsecure-connection-key-ceremony)
export KP_PRIVATE_ADDR=https://private.<region>.kms.cloud.ibm.com

Replace <region> with the region abbreviation that represents the geographic area where your Key Protect instance resides. For the complete list of endpoints, see [Regions and endpoints](https://cloud.ibm.com/docs/key-protect?topic=key-protect-regionsconnectivity-options).







 Step 3. Create a Key Protect resource on the private network 

Test your private network connection by using the [Key Protect CLI plug-in](https://cloud.ibm.com/docs/key-protect?topic=key-protect-set-up-cli).



1. Create a [root key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-create-root-keys) by targeting the private endpoint.

ibmcloud kp key create <key_name> -i <instance_ID>

Replace <key_name> with a human-readable alias for easy identification of your key. Replace <instance_ID> with the IBM Cloud instance ID that identifies your Key Protect instance.
2. Optional: Verify that the key was created successfully by listing the keys that are available in your Key Protect instance.

ibmcloud kp keys -i <instance_ID>

Replace <instance_ID> with the IBM Cloud instance ID that identifies your Key Protect instance.







 Next steps 

You're now set to interact with Key Protect through a private endpoint.



* To find out more about managing keys with Key Protect, [check out the Key Protect CLI reference doc](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-reference).
When you use root keys regionally or cross-regionally, all network traffic is directed to a private endpoint. In other words, your keys encrypt and decrypt resources over a private network that's inaccessible to the internet. For more information about private endpoints, see [Secure access to services by using service endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview). You can see a list of private endpoints in the UI by going to the Resource List > Security and clicking a Key Protect or Hyper Protect Crypto Services instance. In the left menu, select Endpoints.





 General procedure for setting up customer-managed encryption 

Setting up customer-managed encryption for your Block Storage for VPC volumes involves several steps.



1. Generate your own root key. You can use your on-premises HSM. Or you can create your root key by using an IBM Cloud HSM or Hyper Protect Crypto Services. IBM Cloud data centers provide a dedicated HSM to create and protect your keys. By using Hyper Protect Crypto Services, you can take control of your cloud data encryption keys and cloud hardware security module.
2. Provision the key management service (KMS) that best meets your needs.
3. Use your KMS to securely import your root key to the cloud service. For added security, create an import token in your KMS to encrypt and import root keys to the service.
4. From IBM Cloud® Identity and Access Management (IAM), authorize service between Cloud Block Storage (source service) and your KMS (target service). For custom images, also authorize between Image Service for VPC (source service) and IBM Cloud Object Storage (target service).
5. The IBM Cloud VPC infrastructure uses your root key to secure the data by wrapping the passphrases for volumes or custom images. For custom images, the root key wraps the passphrase before you import the custom image file to the cloud.
Managing an allowed IP policy 

After you set up your IBM® Key Protect service instance, you can manage allowed IP policies by using the Key Protect service API.

For the best results now available, use [context-based restrictions (CBR)](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-whatis) instead of an allowed IP policy to define and enforce access restrictions for IBM Cloud resources based on the network location of access requests.



 Managing allowed IP settings 

Allowed IP for Key Protect instances is a type of network policy that you can use to restrict access to your Key Protect instance through public and private endpoints. When you enable this policy at the instance level, Key Protect only permits access to the resources in your Key Protect instance from IP addresses that you have listed on your allowed IP policy.

Setting and retrieving an allowed IP policy is only supported through the application programming interface (API). Allowed IP policy support will be added to the user interface (UI), command line interface (CLI), and software development kit (SDK) in the future. To find out more about accessing the Key Protect APIs, check out [Setting up the API](https://cloud.ibm.com/docs/key-protect?topic=key-protect-set-up-api).

Key Protect currently supports the allowed IP policy feature on an account basis. If you need to enable an allowed IP policy for your Key Protect instance, [get support](https://cloud.ibm.com/docs/get-support?topic=get-support-using-avatar) or [create a case](https://cloud.ibm.com/unifiedsupport/cases/form) to request access to this feature.

Before you enable and allowed IP policy for your Key Protect instance, keep in mind the following considerations:



* If you are using a private endpoint, your private environment should be set up prior to enabling an allowed IP policy for your Key Protect instance.
Set the name to a unique value, such as <your-initials>-secure-file-upload.
2. Set Resiliency to Regional.
3. Set Location to the same location where you created the Key Protect service instance.
4. Set Storage class to Standard



3. Under Service integrations (optional) / Encryption, enable Key management



1. Select the Key Protect service instance created earlier by clicking on Use existing instance
2. Select secure-file-storage-root-enckey as the key and click Associate key.



4. Under Service integrations (optional) / Monitoring & Activity tracking, enable Activity tracking to have events recording in Activity Tracker.



1. After clicking the checkmark the service information for the Activity Tracker instance in the region should be shown.
2. Now, enable Track Data events and select read & write as Data Events.



5. Click Create bucket.









 A database map relationships between users and their files 

The IBM Cloudant database will contain metadata for all files uploaded from the application.



1. Create an instance of [IBM Cloudant](https://cloud.ibm.com/catalog/services/cloudant) service.



1. Select Cloudant as the offering.
2. Select a Multitenant environment and a region same as the previous services.
3. Set the name to secure-file-storage-cloudant.
4. Use the same resource group as for the previous services.
5. Set Authentication method to IAM.
6. Click Create.



2. Back to the Resource List, locate the newly created service and click on it. Note: You will need to wait until the status changes to Active.



1. Under Service credentials, create New credential.
2. Set the name to secure-file-storage-cloudant-acckey.
3. Set Role to Manager.
4. Keep the default values for the the remaining fields.
5. Click Add.



3. Expand the newly created credentials and make note of the values.
For more information, see [Enabling the HIPAA Supported setting](https://cloud.ibm.com/docs/account/eu_hipaa_supported.htmlenabling-the-hipaa-supported-setting) and the Service Description terms for more details. Provisioning a cluster to manage HIPAA data can take longer than the estimated 5-day period.
3. Select a key management service instance.

All IBM Cloudant environments are encrypted. If you would like to use bring-your-own-key (BYOK) encryption with Key Protect, select the Key Protect instance that holds the encryption key from the drop-down menu. Otherwise, choose the Automatic disk encryption key (default) option, which means the environment is encrypted with an IBM Cloudant-managed key. In order to BYOK with Key Protect, you must ensure that IBM Cloudant is authorized to access the selected key management service instance. You can manage service-to-service authorizations at any time. Visit Manage > Access (IAM) and choose Authorizations.

When you configure service-to-service authorizations, remember the authorization works at the account level, not the resource group level.
4. Select a disk encryption key.

Choose the disk encryption key from the drop-down menu that resides in the Key Protect instance that is chosen in the key management service instance parameter. If you use the default IBM Cloudant-managed key option, then this parameter is set to Automatic disk encryption key (default).



Zoom

![Configure the host by specifying the location for deployment, HIPAA compliance, Key Management Service instance, and disk encryption key.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/tutorials/images/img0005f.png)

Figure 2. Host configuration
7. Click Create to start the provisioning process.

Billing is calculated and prorated every day. Make sure that you want to provision and pay for an environment before you click Create.

Zoom
If this option is used, go to the link listed in the CLI output to generate a one-time passcode.
2. Select the account and resource group where you would like to create a Key Protect instance.

In this tutorial, you interact with the Washington DC region. If you're logged into a different region, be sure to set Washington DC as your target region by running the following command.

ibmcloud target -r us-east
3. Provision an instance of Key Protect within that account and resource group.

First, specify the resource group for the instance by issuing:

ibmcloud target -g <your-resource-group>

For example, ibmcloud target -g Default

Then you can create the instance by issuing:

ibmcloud resource service-instance-create "import-keys-demo" kms tiered-pricing us-east

This tutorial won't incur any charges to your IBM Cloud account.
4. Optional: Verify that the Key Protect instance was created successfully by listing your available Key Protect instances.

ibmcloud resource service-instances

Success! You're now set with the Key Protect instance where you can store and manage your encryption keys. Continue to the next step.







 Step 2. Set up the Key Protect API 

Now that you've provisioned an instance of Key Protect, you're ready to start using the API.

Key Protect provides a graphical user interface and a REST API to create, track, and manage encryption keys. The Key Protect API requires a valid IBM Cloud IAM token and an instance ID to authenticate with the service.

In this step, you use the IBM Cloud CLI to gather the authentication credentials you need to start interacting with Key Protect APIs. To retrieve and prepare your credentials for later steps, you also set the credentials as environment variables in your terminal.



1. In your terminal window, set the Key Protect API endpoint as an environment variable.

export KP_API_URL=https://<region>.kms.cloud.ibm.com
2. Generate an IBM Cloud access token using the Key Protect CLI Plugin, and set it as an environment variable.
With the Keep Your Own Key (KYOK) feature, these keys are protected by master key within the FIPS 140-2 Level 4 HSM boundary and IBM cannot access your keys with technical assurance.



1. [Log in to the Hyper Protect Crypto Services instance](https://cloud.ibm.com/login).
2. Click Managed keys from the navigation and click Create key.
3. Under Vault, select a vault for the key for access control, and click Next.

If you want to assign the key to a new vault, click Create vault. For more instructions, see [Creating vaults](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-create-vaults).
4. Under General, select IBM Key Protect and click Next.
5. Under Key properties, specify the following details of the key. Click Next to continue when you are done.



Table 2. Key properties

 Property Description 

 Key name A unique, human-readable name for easy identification of your key. For Key Protect keys, it must be 2–50 characters in length. The characters can be letters (case-sensitive), digits (0–9), or spaces. For example, kp satellite tiger key. 
 Description (Optional) An extended description for your key, with up to 200 characters in length. For example, KP on satellite keys for project Tiger DB encryption 
 Algorithm The encryption algorithm to encrypt data for the key. For example, AES. 
 Length The number of bits that represents the encryption strength of the key. For example, 256. 
 State Pre-active keys are not to be activatedin target keystores until you manually activate them. Active keys are automatically activated in the target keystores. Select Active for installation to Key Protect on Satellite.For more information about key states, see [Monitoring the lifecycle of encryption keys in Unified Key Orchestrator](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-uko-key-states). 
 Activation date Plan a date to activate the Pre-active key.
To see how to allow or block source IP addresses, try the [Using Calico network policies to block traffic tutorial](https://cloud.ibm.com/docs/openshift?topic=openshift-policy_tutorialpolicy_tutorial).





 Example Calico policies to restrict public or private network traffic 

We provide a set of example [Calico public network policies](https://github.com/IBM-Cloud/kube-samples/tree/master/calico-policies/public-network-isolation) that further restrict public/private network traffic on cluster workers. These policies allow the traffic that is necessary for the cluster to function, and block certain other traffic.

These policies are not meant to block everything, nor do they necessarily meet any compliance requirements on their own. They are intended as a starting point and must be edited to meet your unique use cases. For more information, see the [README](https://github.com/IBM-Cloud/kube-samples/tree/master/calico-policies).

Whenever new locations for Red Hat OpenShift on IBM Cloud and other IBM Cloud are enabled, the subnets for these locations are added to the Calico policies. Be sure to [watch the GitHub repository](https://github.com/IBM-Cloud/kube-samples/tree/master/calico-policies/public-network-isolation) for any updates to these policies.



 Applying public network policies 

Before you begin, [install and configure the Calico CLI, and set the context for your cluster to run Calico commands](https://cloud.ibm.com/docs/openshift?topic=openshift-network_policiescli_install).



1. Clone the IBM-Cloud/kube-samples repository.

git clone https://github.com/IBM-Cloud/kube-samples.git
2. Change directories to the public policy directory for the region that your cluster is in. Example command for a cluster in US South:
If you need full control over your encryption keys, you can use [IBM Cloud Hyper Protect Crypto Services](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-overview) to create, add, and manage encryption keys and associate the keys with your Hyper Protect DBaaS for MongoDB service instance to encrypt your databases.

The following diagram shows how Hyper Protect DBaaS for MongoDB integrates with Hyper Protect Crypto Services.

Zoom

![Integration with Hyper Protect Crypto Services](https://cloud.ibm.com/docs-content/v1/content/cb7bbaa0f4c728092d73d289faafb8ca3fbf51b7/hyper-protect-dbaas-for-mongodb/images/DBaaS-KYOK.svg)

Integration with Hyper Protect Crypto Services

Your backups are also protected by the data encryption key that is wrapped by your root key from Hyper Protect Crypto Services.



 Step 1. Create or add a root key in Hyper Protect Crypto Services 

To create or add a

root keyin Hyper Protect Crypto Services, complete the following steps:



1. [Provision](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-provision) and [initialize your Hyper Protect Crypto Services service instance](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-initialize-hsm) if you haven't.
2. Go to your instance of Hyper Protect Crypto Services and [create a root key](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-create-root-keys) or [import an existing root key](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-import-root-keys).







 Step 2. Grant service authorization 

Authorize Hyper Protect Crypto Services for use with Hyper Protect DBaaS for MongoDB service instances:



1.
In the Advanced Configuration section, select Add Key Protect Keys.
4. From the list of Key Protect service instances, select the instance that contains the root key that you want to use for key wrapping.
5. For Key Name, select the alias of the root key.
6. Click Create to confirm the bucket creation.



From the IBM Cloud Object Storage GUI, you can browse the buckets that are protected by a Key Protect root key.





 What's next 



* For more information about associating your storage buckets with Key Protect keys, see [Manage encryption](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-encryptionencryption).