vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Securing your data in watsonx.data 

Know, how your data is encrypted in IBM® watsonx.data to ensure data security.

The watsonx.data service has security that is built into all levels of its architecture.



1. Web Console UI, API, Presto, and Hive metastore data in motion is encrypted by using SSL/TLS 1.3.
2. Web Console UI, API, Presto, and Hive metastore authentication and authorization is via IBM Cloud IAM.
3. Presto Kubernetes worker node NVMe SSDs used for local ephemeral RaptorX caching are encrypted by using AES-256.
4. Hive metastore backend database is encrypted by using Key Protect.
5. Backend object storage repositories for internal metadata and 10 GB limited trial bucket are encrypted by using AES-256. Customer Bring-your-own-key (BYOK) via Key Protect and Keep-your-own-key(KYOK)via Hyper Protect Crypto(HPCS) are supported at provision time





 Integrating your data and keys 

The backend object storage bucket for internal metadata and 10 GB limited trial bucket for watsonx.data can be encrypted with your encryption keys. If you need to control the encryption keys, use IBM Key Protect or Hyper Protect Crypto Services to create, add, and manage encryption keys. Then, you can associate those keys with your watsonx.data deployment to encrypt your buckets.

IBM Key Protect helps you provision encrypted keys for apps across IBM Cloud services. You manage the lifecycle of your keys and benefit as your keys are secured by FIPS140-2 Level 3 certified cloud-based hardware security modules (HSMs) that protects from information theft.

Hyper Protect Crypto Services is a single-tenant, dedicated HSM that is controlled by you. The service is built on FIPS 140-2 Level 4-certified hardware, the highest offered by any cloud provider in the industry. To get started, you need to provision a Key Protect instance or a Hyper Protect Crypto Services instance on your IBM Cloud account.
[Checkmark icon](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/icons/checkmark-icon.svg) 









 What's in a secret? 

Secrets that you store with the service consist of metadata attributes and a secret value. While the metadata attributes help you to identify a secret, the secret value is the data that protected services need to authenticate and authorize you or your application.

Check out the following image to see how a secret is structured.

Zoom

![This image shows the components of a secret. The information in the image is detailed in the surrounding content.](https://cloud.ibm.com/docs-content/v1/content/991f60ddbe3ab0a9aa69481a07a0a73b359fa329/secrets-manager//images/example-secret.svg)

Figure 1. JSON representation of Secrets Manager secret



1. The name, id, and description, and other common fields hold identifying information about a secret. These fields store the general attributes of your secret that you can use to understand its purpose and history.
2. For most secret types, the secret_data object contains the actual value of your secret.

When you use the Secrets Manager API to retrieve the value of a secret, the fields that you see in the secret_data object differ depending on the type of secret that you are inspecting. For example, the following truncated example shows how secret data is represented for an arbitrary secret.

{
"name": "my-arbitrary-secret",
"secret_type": "arbitrary",
...
"secret_data": {
"payload": "The quick brown fox jumped over the lazy dog."
}
}

If you're working with IAM credentials, the secret data is listed alongside the other common fields that describe your secret. For example, the following truncated example shows how secret data is represented for an IAM credential.

{
Securing your data in Planning Analytics as a Service 

To ensure that you can securely manage your data when you use IBM Planning Analytics® as a Service, it is important to know what data is stored and encrypted and how you can delete any stored data.



  How your data is stored and encrypted in Planning Analytics as a Service 

Planning Analytics as a Service stores personal data in one of three data stores:



*  Amazon Relational Database Service (RDS): Might contain user's email address, first name, last name, and phone number
*  Amazon MemoryDB for Redis: Might contain user's email address
*  MongoDB Atlas: Might contain user's email address



Users can update their personal data directly within Planning Analytics as a Service.

Data is encrypted in transit by mutual TLS connections. Data is encrypted at rest by the Amazon RDS, Amazon MemoryDB for Redis, and MongoDB Atlas services.



  Deleting Planning Analytics as a Service instances 

Deleting the Planning Analytics as a Service instance deletes all data.

The Planning Analytics as a Service data retention policy describes how long your data is stored after you delete the service. The data retention policy is included in the Planning Analytics as a Service service description, which you can find in the [IBM Cloud Terms and Notices](https://cloud.ibm.com/docs/overview?topic=overview-terms).
Known issues (Limitations) 

Use this topic to find the details of the limitations that are applicable to the Data Security Broker software.



 Application-level Encryption: 

Data Security Broker encryption gives database applications a no-code way to implement application-level encryption. Application-level encryption can be implemented by including a database proxy that intercepts and encrypts sensitive data in accordance with the pre-established data protection policy.

Data Security Broker encryption has some restrictions, like any other approaches to the application-level encryption implementation. The various aspects of the application level encryption limitations are explained below:

Performance hit Data encryption consumes more time and resources than data decryption. This overhead is normally negligible. If the proxy (Shield) has an appropriately sized CPU and Memory, there should not be any noticeable performance penalties. Most users experience a 10--20% reduction in the overall application performance. This can be mitigated by horizontal scaling. In the event of failure, Shield can be horizontally scaled by adding more of them behind a load balancer.

Key management You must have a place to store your secret key or key file. You can use IBM Key Protect. IBM Key Protect provides full encryption visibility and control, allowing you to see and manage data encryption and the entire key lifecycle from a single location. Alternatively, you can secure the information with a password, but this reduces security, depending on the level of your password's strength.

Accessibility If your key file is lost, your data is also lost. So, make a backup of your data and your key.

Database operations carried out on encrypted versions of the data The information is kept in encrypted form in the database and the database engine and, consequently, the database administrator is never permitted to see the information in form of the plaintext.

Many of today's server and network technologies allow for easier configuration and implementation to minimize the impact on utilization. Implementing encryption of data in transit from endpoint to endpoint both remotely and internally is mandatory in today's cyber risk environment.

The following are considered equality check operators and are supported:



* =
* <>
* IS NULL
* IS NOT NULL
* IN
* NOT
* JOIN (all types)
* GROUP BY
[Protecting your sensitive data in Secrets Manager](https://cloud.ibm.com/docs/secrets-manager?topic=secrets-manager-mng-datadata-encryption) 
 [Certificate Manager](https://cloud.ibm.com/docs/certificate-manager) You can use Certificate Manager to order and manage SSL/TLS certificates for your applications and services. [Protecting your sensitive data in Certificate Manager](https://cloud.ibm.com/docs/certificate-manager?topic=certificate-manager-mng-datadata-encryption) 
 [Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance) With Security and Compliance Center, you can govern cloud resource configurations and centrally manage your compliance with organization and regulatory guidelines. When you work with the Security and Compliance Center, data is generated by the service as you interact with it. [Protecting your sensitive data in Security and Compliance Center](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-mng-datadata-encryption) 







 Developer tools service integrations 

You can integrate Hyper Protect Crypto Services with the following developer tools services.



Table 7. Supported developer tools services.

 Service Description Integration instruction 

 [IBM Cloud® Continuous Delivery](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-getting-started) The Continuous Delivery service provides a suite of tools that support DevOps best practices. You can use the service to manage toolchains, operate delivery pipelines, gain insights into code quality and vulnerabilities, integrate third-party tools, and more. <br><br> * [Creating a Continuous Delivery service instance](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-create_cd_service)<br> * [Protecting your personal data when you use the Professional plan](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-cd_data_securitycd_professional_plan)<br><br><br> 







 Understanding your integration
The following components encrypt personal data by using only the provider-managed encryption key.



Table 4. Values that are encrypted by using the provider-managed key

 Component Value 

 Git Repos and Issue Tracking <br><br> * Issues, pull requests, and source code<br> * Personal information, such as name, email, profile picture, address, and other information from the profile page<br><br><br> 



For more information about creating a Continuous Delivery service instance that encrypts data with a customer key, see [Creating a Continuous Delivery service instance](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-create_cd_service).





 Protecting your data when you use third-party tool integrations 

When you configure a tool integration for a toolchain, you explicitly enable data sharing between the Continuous Delivery service and the tool integration. The kind of data that is shared, and whether data is sent to the tool, received from the tool, or both, varies depending on the type of tool integration.

Before you configure a tool integration, make sure you understand what data is shared. If you work with regulated data or data that you consider sensitive, make sure that the third-party tool usage and any data that might be shared with it do not compromise the confidentiality, integrity, or availability of your resources, or otherwise violate regulatory controls.

To learn more about tool integrations, see [Configuring tool integrations](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-integrations).





 Protecting your data when you use Git Repos and Issue Tracking 

Git Repos and Issue Tracking is an IBM-hosted component of the Continuous Delivery service. All of the data that you provide to Git Repos and Issue Tracking, including but not limited to source files, issues, pull requests, and project configuration properties, is managed securely within Continuous Delivery. However, Git Repos and Issue Tracking supports various mechanisms for exporting, sending, or otherwise sharing data to users and third parties.

The ability of Git Repos and Issue Tracking to share information is typical of many social coding platforms.
Wrapping data encryption keys with root keys 

You can manage and protect your encryption keys with a

root keyby using the IBM Cloud® Hyper Protect Crypto Services key management service API, if you are a privileged user.

When you wrap a

data encryption key (DEK)with a root key, Hyper Protect Crypto Services combines the strength of multiple algorithms to protect the privacy and the integrity of your encrypted data.

To learn how key wrapping helps you control the security of at-rest data in the cloud, see [Envelope encryption](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-envelope-encryption).



 Wrapping keys by using the API 

You can protect a specified data encryption key (DEK) with a root key that you manage in Hyper Protect Crypto Services.

When you supply a root key for wrapping, ensure that the root key is 128, 192, or 256 bits so that the wrap call can succeed. If you create a root key in the service, Hyper Protect Crypto Services generates a 256-bit key from the HSMs, supported by the AES-CBC algorithm.

[After you designate a root key in the service](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-create-root-keys), you can wrap a DEK with advanced encryption by making a POST call to the following endpoint.

https://api.<region>.hs-crypto.cloud.ibm.com:<port>/api/v2/keys/<key_ID>/actions/wrap



1. [Retrieve your service and authentication credentials to work with keys in the service.](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-set-up-kms-api)
2. Copy the key material of the DEK that you want to manage and protect.
* [Data encryption at rest in IBM Cloud](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restshared-encryption-at-rest-ibm-cloud)



* [Setting up Hyper Protect Crypto Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restsetup-hpcs)



* [Data encryption at rest in Satellite location](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restshared-encryption-at-rest-satellite-location)
* [Integrating Hyper Protect Crypto Services with other IBM Cloud services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restrelated-resources)
* [Related controls in IBM Cloud Framework for Financial Services](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restrelated-controls)
* [Next steps](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-at-restnext-steps)



[Data encryption in transit](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-in-transitshared-encryption-in-transit)



* [TLS requirements](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-in-transitshared-encryption-in-transit-tls-requirements)



* [TLS protocol requirements](https://cloud.ibm.com/docs/framework-financial-services?topic=framework-financial-services-shared-encryption-in-transitshared-encryption-in-transit-tls-protocol-requirements)
Data Encryption using IBM Cloud Databases for PostgreSQL 



 Overview 

Data Security Broker functions as an application-level encryption (ALE) equivalent in this mode encrypting data on a field-level basis. This is performed using Data Security Broker Manager to enumerate the data schema and enable an encryption key mapping.





 Procedure 

Complete the following steps to encrypt the data with Data Security Broker Manager on an IBM Cloud PostgreSQL Database:



1. Login to Data Security Broker Manager.
2. Click on an application and select the drop down which is present in the Migration Details field in the right side and click Encrypt.
3. Select the Database and the table where you have the data created and select the Column which needs to be encrypted. Choose the Data Protection policy, Encryption mode, and masking mode for the encryption process and click Review.

Zoom

![Encryption](https://cloud.ibm.com/docs-content/v1/content/57ab3ec034294d8a6de510b97bec3035d4654761/security-broker/images/encryption_schema.svg)

Standard Encryption
4. Choose Deploy Policy & Migrate Data under the Deployment Plan option. There are three options that you can choose to implement your data encryption policy. For more information on Deployment plans, see Deployment Plans in IBM Cloud Data Security Broker. Select the Security Broker Shield service IP address in the Migration Shield field and click Save to start the encryption process.
5. The status of the application shows Migrating when the encryption process starts.
6. Once the encryption is complete, the status is changed to Protected. You can view more information by clicking Migration Details in the Applications sidebar.



If there is new data which gets inserted in the database, by default, the data is encrypted by using the default data encryption policy that is being selected by the user.





 Reference 





 Format Preserving Encryption (FPE) Supported Data Types 

The following tables lists the FPE supported data types for the data encryption in Data Security Broker Manager:





 PostgreSQL 



Table 1.
Using Hyper Protect Crypto Services PKCS 11 for Oracle Transparent Database Encryption 

Transparent Data Encryption (TDE) is a well-established technology to encrypt sensitive data in databases. TDE is supported by various popular database systems, both in the cloud and on premises, like Oracle® database. With TDE, a database system encrypts data on database storage media, such as table spaces and files, and on backup media. The database system automatically and transparently encrypts and decrypts data when it is used by authorized users and applications. Database users do not need to be aware of TDE and database applications do not need to be adapted specifically for TDE.

Typically, TDE uses a two-tiered key hierarchy, which is composed of a TDE master encryption key and a TDE data encryption key. The TDE data encryption key is used to encrypt and decrypt data, while the TDE master encryption key is used to encrypt and decrypt the TDE data encryption key.

Therefore, one important question when you plan for TDE is: Where do you keep the TDE master encryption key, and how do you secure it?



 Objectives 

This tutorial shows how you can keep complete and exclusive control of your TDE master encryption keys by storing them in IBM Cloud® Hyper Protect Crypto Services. For this purpose, you need to use the PKCS #11 integration feature of Hyper Protect Crypto Services.

With this tutorial, you are going to implement the setup that is depicted in the following illustration.

Zoom

![Transparent Database Encryption by using the standard PKCS #11 API](https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto/images/pkcs-database.svg)

Figure 1. Transparent Database Encryption by using the standard PKCS #11 API

In this setup, the Oracle Database is to call operations to manage the TDE master encryption keys on the Hyper Protect Crypto Services PKCS #11 library.
This key is encrypted by a passphrase and wrapped (encrypted) by the root key, creating a wrapped DEK or WDEK. The WDEK is stored as metadata with the volume or image, and is not available on VPC interfaces.

Custom images are encrypted by your own LUKS passphrase that you create by using QEMU. After the image is encrypted, you wrap the passphrase with your root key that is stored in the KMS.

For more information about envelope encryption, see the following documentation:



* Key Protect - [Protecting data with envelope encryption](https://cloud.ibm.com/docs/key-protect?topic=key-protect-envelope-encryption)
* Hyper Protect Crypto Services - [Protecting your data with envelope encryption](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-envelope-encryption)







 Root key regional and cross-regional considerations 

When you use customer-managed encryption for volumes, file shares, and custom images, you can keep your root keys in the same region as your resources or in a different region (cross-regional).

Cross-regional keys offer more key availability with the tradeoff of slightly higher latency. You can create and manage root keys in one region and apply them to resources in another region. Cross-regional keys are available in all regions.

For optimal performance and security, use root keys in the same region as your encrypted resources. Regional service uses private endpoints to multiple availability zones. If a specific zone is unavailable, you can continue to access your root keys and encrypted resources from another zone.

When you use root keys regionally or cross-regionally, all network traffic is directed to a private endpoint. In other words, your keys encrypt and decrypt resources over a private network that's inaccessible to the internet. For more information about private endpoints, see [Secure access to services by using service endpoints](https://cloud.ibm.com/docs/account?topic=account-service-endpoints-overview).
gRPC is a modern open source high-performance remote procedure call (RPC) framework that can connect services in and across data centers for load balancing, tracing, health checking, and authentication. Applications access Hyper Protect Crypto Services by calling the EP11 API remotely through gRPC. For more information about gRPC, see [the gRPC documentation](https://grpc.io/docs/guides/index.html).

With the GREP11 API, you can perform the following operations:



* Key generation.
* Encrypt and decrypt.
* Sign and verify.
* Wrap and unwrap keys.
* Derive keys.
* Build message digest.
* Retrieve mechanism information.



For some operations, there are a series of sub-operations. For example, the multi-part data encryption operation is composed of the EncryptInit(), EncryptUpdate(), and EncryptFinal() sub-operations.



* EncryptInit() is used to initialize an operation.
* Encrypt() is used to encrypt single-part data without the need to perform EncryptUpdate() and EncryptFinal() sub-operations. This operation needs to be performed after the EncryptInit() call.
* EncryptUpdate() and EncryptFinal() are used in combination to perform multi-part data encryption. These sub-operations need to be performed after the EncryptInit() call.
* EncryptSingle() is an IBM EP11 extension to the standard PKCS #11 specification, and is used to perform a single call to encrypt single-part data without the need to run the EncryptInit() and Encrypt() sub-operations.



The following diagram shows the three calling sequence flows of GREP11 functions to perform encryption. The flows can also apply to other operations such as decrypt, digest, sign, and verify.