vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Data encryption keys (DEKs) are designed to encrypt your data and can be generated and managed by your service or an IBM Cloud service.

Envelope encryption offers several benefits for protecting your data:



* Protection under a combination of multiple algorithms Envelope encryption uses the best benefits from symmetric and public key algorithms to keep your keys secure.



1. Symmetric key algorithms work faster, are more scalable, and more secure than public key algorithms. Public key algorithms use complicated mathematics that increase computational overhead, especially when dealing with large volumes of data. Public key algorithms are also more susceptible to brute force attacks due to having a private key algorithm component that is easily recognizable by hackers. Symmetric key algorithms requires less computed power and are resistant to brute force attacks due to having a less recognizable structure.
2. Public key algorithms allow for easier access control when granting access to keys at an individual level compared to symmetric key algorithms. Symmetric key algorithms have a key exchange problem, which is that access to a secret key can only be exchanged through a secure transfer. By using public key algorithms, encrypted DEKs (wDEKs) can be shared and unencrypted only by those with access to the encrypting root key, mitigating the key exchange problem of symmetric algorithms.



* Easier key management You can encrypt multiple DEKs under a singular root key, which minimizes the amount of keys that you might need to manage in a key management service. You can also choose to save time on key maintenance by only rotating your root keys, instead of rotating and re-encrypting all of your DEKs. Note that in cases such as personnel turnover, process malfunctions, or the detection of a security issue, it is recommended to rotate all DEKs and root keys associated with the incident.
* Data Key Protection Since your DEKs are wrapped by a root key, you do not have to worry about how to store the encrypted data key. Due to this, you can store the wDEK with alongside the associated encrypted data.



Key Protect uses the Advanced Encryption Standard algorithm in Galois/Counter Mode (AES GCM) to wrap and unwrap DEKs. CRKs that are not imported are created with 256-bit key material.
Unwrapping data encryption keys with root keys 

You can unwrap a

data encryption key (DEK)to access the contents by using the IBM Cloud® Hyper Protect Crypto Services key management service API, if you are a privileged user. Unwrapping a DEK decrypts and checks the integrity of the contents, returning the original key material to your IBM Cloud data service.

To learn how key wrapping helps you control the security of at-rest data in the cloud, see [Envelope encryption](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-envelope-encryption).



 Unwrapping keys by using the API 

[After you make a wrap call to the service](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-wrap-keys), you can unwrap a specified data encryption key (DEK) to access the contents by making a POST call to the following endpoint.

https://api.<region>.hs-crypto.cloud.ibm.com:<port>/api/v2/keys/<key_ID>/actions/unwrap

Root keys that contain the same key material can unwrap the same data encryption key (WDEK).



1. [Retrieve your service and authentication credentials to work with keys in the service](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-set-up-kms-api).
2. Copy the ID of the

root keythat you used to perform the initial wrap request.

You can retrieve the ID for a key by making a GET /v2/keys request, or by viewing your keys in the IBM Cloud console.
3. Copy the ciphertext value that was returned during the initial wrap request.
4. Run the following cURL command to decrypt and authenticate the key material.
sc.hadoopConfiguration.set("parquet.encryption.key.list","key1: AAECAwQFBgcICQoLDA0ODw==, key2: AAECAAECAAECAAECAAECAA==")
val encryptedParquetPath = "squares.parquet.encrypted"
val parquetFile = spark.read.parquet(encryptedParquetPath)
parquetFile.show()







 Internals of encryption key handling 

When writing a Parquet file, a random data encryption key (DEK) is generated for each encrypted column and for the footer. These keys are used to encrypt the data and the metadata modules in the Parquet file.

The data encryption key is then encrypted with a key encryption key (KEK), also generated inside Spark/Parquet for each master key. The key encryption key is encrypted with a master encryption key (MEK), either locally if the master keys are managed by the application, or in a KeyProtect service if the master keys are managed by IBM® Key Protect for IBM Cloud®.

Encrypted data encryption keys and key encryption keys are stored in the Parquet file metadata, along with the master key identity. Each key encryption key has a unique identity (generated locally as a secure random 16-byte value), also stored in the file metadata. Key encryption keys are cached by access token, so there is no need to interact with the KeyProtect service for each encrypted column or file if they use the same master encryption key.

When reading a Parquet file, the identifier of the master encryption key (MEK) and the encrypted key encryption key (KEK) with its identifier, and the encrypted data encryption key (DEK) are extracted from the file metadata.

The key encryption key is decrypted with the master encryption key, either locally if the master keys are managed by the application, or in a KeyProtect service if the master keys are managed by IBM® Key Protect for IBM Cloud®.
They are symmetric key-wrapping keys that are used as roots of trust for wrapping (encrypting) and unwrapping (decrypting) other keys that are stored in a data service. With Hyper Protect Crypto Services, you can create, store, and manage the lifecycle of root keys to achieve full control of other keys stored in the cloud.

Data encryption keys
: Data encryption keys (DEKs) are cryptographic keys that you use for data encryption. They are provided by user-owned applications and are used to encrypt data stored in applications. Root keys that are managed in Hyper Protect Crypto Services serve as wrapping keys to protect DEKs.

After you create a key in Hyper Protect Crypto Services, the system returns a key ID that is used to uniquely identify the key resource. You can use this ID value to make API calls to the service.





 How it works 

Envelope encryption combines the strength of multiple encryption algorithms to protect your sensitive data in the cloud. It works by wrapping one or more data encryption keys (DEKs) with advanced encryption by using a root key that you can fully manage. This key wrapping process creates wrapped DEKs that protect your stored data from unauthorized access or exposure. Unwrapping a DEK reverses the envelope encryption process by using the same root key, resulting in decrypted and authenticated data.

Root keys that are managed in a Hyper Protect Crypto Services service instance are also encrypted by the master key that ensures you full control of the entire key hierarchy.

The following diagram shows a contextual view of envelope encryption.

Zoom

![The diagram shows a contextual view of envelope encryption.](https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto//images/envelope-encryption.svg)

Figure 1. Contextual view of envelope encryption

Envelope encryption is treated briefly in the NIST Special Publication 800-57, Recommendation for Key Management. To learn more, see [NIST SP 800-57 Pt.
Securing your data in VMware Solutions 

Know what data is stored and encrypted and how to delete any stored data to ensure that you can securely manage your personal data when you use IBM Cloud® for VMware® Solutions.



 Data storage and encryption in VMware Solutions 

When a user onboards to VMware Solutions and orders instances, we store and manage user data of configuration and metadata that is associated with the user and ordered instances. That user data includes the following items.



* For both VMware Shared and vCenter Server, the user data includes the following items:



* IBMid (email)
* Instance configuration information
* Instance access information such as login credentials to VMware Cloud Director, VMware vCenter Server®, and VMware NSX® Manager.



* For vCenter Server, the user data includes the following items:



* IBM Cloud classic infrastructure credentials (username and API key)





This configuration data and metadata is stored and managed by IBM. It is encrypted at REST and in transit. Additionally, sensitive data such as API key and access information are encrypted with customer–specific encryption keys.

With vCenter Server, you can bring your own data to IBM Cloud bare metal servers and IBM Cloud File and Block storage that is managed by your VMware instance. All of this data is managed by you and not managed by IBM, and you have the option of encrypting it using various solutions.

These solutions include the following options:



* KMIP™ for VMware service along with IBM Cloud Key Protect or IBM Cloud Hyper Protect Crypto Services to enable vSAN™ or VMware vSphere® encryption for your workloads
* Other self–managed VMware–compatible encryption technologies



If you use VMware Shared, your workload data exists in an IBM–managed cloud infrastructure account. You are provided with the default vSphere encryption option for your VMs, which uses IBM–managed keys that are backed by the IBM Cloud KMIP for VMware and Hyper Protect Crypto Services. You can optionally implement your own encryption solutions within your VMware workloads.





 Data storage and encryption in the VMware Shared Veeam Availability Suite service
Understanding encryption for Portworx 



 Encryption overview 

The following image illustrates the encryption workflow in Portworx when you set up per-volume encryption.

Zoom

![Encrypting Portworx volumes](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_px_volume_encryption.png)

Figure 1. Encrypting Portworx volumes



1. The user creates a PVC with a Portworx storage class and requests the storage to be encrypted.
2. Portworx invokes the IBM Key Protect or Hyper Protect Crypto Services API WrapCreateDEK to create a passphrase by using the customer root key (CRK) that is stored in the Portworx secret.
3. The IBM Key Protect or Hyper Protect Crypto Services service instance generates a 256-bit passphrase and wraps the passphrase in the DEK. The DEK is returned to the Portworx cluster.
4. The Portworx cluster uses the passphrase to encrypt the volume.
5. The Portworx cluster stores the DEK in plain text in the Portworx etcd database, associates the volume ID with the DEK, and removes the passphrase from its memory.







 Decryption overview 

The following image illustrates the decryption workflow in Portworx when you set up per-volume encryption.

Zoom

![Decrypting Portworx volumes](https://cloud.ibm.com/docs-content/v1/content/bce0f0917a9eea684d1b4b704ac6343a1f65f446/openshift/images/cs_px_volume_decryption.png)

Figure 2. Decrypting Portworx volumes



1.
[Encryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959744) C_EncryptInit Yes Initializes an encryption operation. 
 [Encryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959744) C_Encrypt Yes Encrypts single-part data. 
 [Encryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959744) C_EncryptUpdate Yes Continues a multiple-part encryption operation. 
 [Encryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959744) C_EncryptFinal Yes Finishes a multiple-part encryption operation. 
 [Decryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959745) C_DecryptInit Yes Initializes a decryption operation. 
 [Decryption](http://docs.oasis-open.org/pkcs11/pkcs11-base/v2.40/os/pkcs11-base-v2.40-os.html_Toc416959745) C_Decrypt Yes Decrypts single-part encrypted data.
* UI
* API







 Unwrapping keys 

IBM® Key Protect combines the strength of multiple algorithms to protect the privacy and the integrity of your encrypted data. You can unwrap a

data encryption key(DEK), to access its contents by using the IBM® Key Protect for IBM Cloud® API and Console. Unwrapping a DEK decrypts and checks the integrity of its contents, returning the original key material to your IBM Cloud data service.

To learn how key wrapping helps you control the security of at rest data in the cloud, see [Protecting data with envelope encryption](https://cloud.ibm.com/docs/key-protect?topic=key-protect-envelope-encryption).



 Unwrapping a key using the console 

If you already have an instance of Key Protect and you wish to encrypt your DEK by using a graphical interface, you can use the IBM Cloud console.

After you import or create your [own keys](https://cloud.ibm.com/docs/key-protect?topic=key-protect-create-root-keys), complete the following steps to wrap your data using the key:



1. Log in to the [IBM Cloud console](https://cloud.ibm.com/login/).
2. From the Menu, choose the Resource List to view a list of your resources.
3. From your IBM Cloud resource list, select your provisioned instance of Key Protect.
4. Choose the Root key that you used to originally wrap your data from the list of your keys.
5. Click the ⋯ icon to open a list of options for the DEK that you want to unwrap.
6. Click the Envelope encryption option to open the side panel. Choose the "Unwrap key" tab if it is not already highlighted.
7. Supply any required text for the above in the appropriate fields, including the ciphertext and any additional data you stored locally for the purpose of unwrapping the DEK.
8. Click Unwrap Key button.







 Unwrapping keys by using the API
Your wrapped data encryption key, containing the base64 encoded key material, is returned in the response entity-body. The response body also contains the ID of the key version that was used to wrap the supplied plaintext. The following JSON object shows an example returned value.

{
"ciphertext": "eyJjaXBoZXJ0ZXh0IjoiYmFzZTY0LWtleS1nb2VzLWhlcmUiLCJpdiI6IjRCSDlKREVmYU1RM3NHTGkiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJ1dWlkLWdvZXMtaGVyZSJ9",
"keyVersion": {
"id": "02fd6835-6001-4482-a892-13bd2085f75d"
}
}

If you omit the plaintext attribute when you make the wrap request, the service returns both the generated data encryption key (DEK) and the wrapped DEK in base64 encoded format.

{
"plaintext": "Rm91ciBzY29yZSBhbmQgc2V2ZW4geWVhcnMgYWdv",
"ciphertext": "eyJjaXBoZXJ0ZXh0IjoiYmFzZTY0LWtleS1nb2VzLWhlcmUiLCJpdiI6IjRCSDlKREVmYU1RM3NHTGkiLCJ2ZXJzaW9uIjoiNC4wLjAiLCJoYW5kbGUiOiJ1dWlkLWdvZXMtaGVyZSJ9",
"keyVersion": {
"id": "12e8c9c2-a162-472d-b7d6-8b9a86b815a6"
}
}

The plaintext value represents the unwrapped DEK, and the ciphertext value represents the wrapped DEK and are both base64 encoded. The keyVersion.id value represents the version of the root key that was used for wrapping.

If you want Key Protect to generate a new data encryption key (DEK) on your behalf, you can also pass in an empty body on a wrap request. Your generated DEK, containing the base64 encoded key material, is returned in the response entity-body, along with the wrapped DEK.
![IBM Cloud Schematics architecture and data encryption process](https://cloud.ibm.com/docs-content/v1/content/ba89e173287b28d1453d2a327d8a1f74ae2f1662/schematics/images/schematics_architecture.png)

IBM Cloud Schematics architecture and data encryption process



1. A user sends a request to create a Schematics workspace to the Schematics API server. An IAM request is made to check if the user is authorized to perform Schematics operations for the workspace.
2. The API server retrieves the Terraform template and input variables from your GitHub or GitLab source repository, or a tape archive file (.tar) that you uploaded from your local machine. User data in transit is protected with TLS.
3. All user-initiated actions, creating a workspace, generating a Terraform execution plan, or applying a plan are sent to RabbitMQ and added to the internal queue. The Schematics engine retrieves requests from RabbitMQ and executes the actions. User data in transit is protected with TLS.
4. The Schematics engine runs the tasks to provision, modify, or delete IBM Cloud resources.
5. To protect user data at rest, IBM Cloud Schematics encrypts data with AES GCM 256 encryption. Envelope encryption with root keys managed with Key Protect and Hyper Protect Crypto Services is used to generate and encrypt unique data encryption keys (DEK) for the data objects.
6. Workspace transactional data is encrypted using the DEKs, including logs and the Terraform tf.state file at rest. The encrypted data stored in an IBM Cloud Object Storage bucket .
7. Workspace operational data, workspace and job names, pointers to user data in IBM Cloud Object Storage and search keys, are stored in IBM Cloudant. All information stored in Cloudant is encrypted with AES 256.
You can view more information by clicking Migration Details in the Applications sidebar.



If there is new data which gets inserted in the database, by default, the data is encrypted by using the default data encryption policy that is being selected by the user.





 Reference 





 Format Preserving Encryption (FPE) Supported Data Types 

The following tables lists the FPE supported data types for the data encryption in Data Security Broker Manager:





 PostgreSQL 



Table 1. FPE Supported Data Types caption-side=

 Original Data Type FPE Data Type 

 smallint fpe-int 
 int fpe-int 
 integer fpe-int 
 bigint fpe-int 
 bytea fpe-int 
 numeric fpe-decimal 
 decimal fpe-decimal 
 numeric (s,p) fpe-decimal 
 decimal (s,p) fpe-decimal 
 money fpe-decimal 
 var - fpe-decimal - fpe-alphanum - fpe-latin1 
 char - fpe-win1252 - fpe-cc 
 text - fpe-email1 - fpe-email2 
 date fpe-datetime 
 time fpe-datetime 
 timestamp fpe-datetime 
 uuid fpe-hexadecimal 







 Counter-Mode (CTR) Supported Data Types 

Data Security Broker Shield only supports one word for a data type name. BYTEA is a PostgreSQL data type that has the capability to store hexadecimal data which is used to store encrypted data. BYTEA is an equivalent of VARBINARY in MySQL or RAW datatype in Oracle database.

PostgreSQL The following table lists PostgreSQL supported data types for M_CTR mode in Data Security Broker Manager.



Table 2. CTR Supported Data Types caption-side=

 Original Data Type Encrypted Data Type 

 SMALLINT BYTEA 
 INT, INTEGER BYTEA 
 BIGINT BYTEA 
 REAL, FLOAT4 BYTEA 
 FLOAT8 - Used in Data Security Broker Shield for "double precision" BYTEA 
 DECIMAL, NUMERIC BYTEA
Achieving full disk encryption with LUKS in RHEL 

You can encrypt partitions on your RHEL server with Linux® Unified Key Setup-on-disk-format (LUKS), which is important when it comes to mobile computers and removable media. LUKS allows multiple user keys to decrypt a main key that is used for the bulk encryption of the partition. The following instructions are applicable to RHEL versions RHEL6 or newer.

These steps assume that the server can access a new, unencrypted IBM Cloud® Block Storage for Classic volume that was not formatted, or mounted. For more information about connecting Block Storage for Classic to a Linux® host, see [Connecting to storage on Linux®](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-mountingLinux).

{site.data.keyword.blockstorageshort}} that is provisioned in [most data centers](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-selectDC) is automatically provisioned with provider-managed encryption-at-rest. For more information, see [Securing Your Data - Provider-managed Encryption-At-Rest](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-mng-data).



 What LUKS does 



* Encrypts entire block devices and is therefore suited for protecting the contents of mobile devices such as removable storage media or Notebook disk drives.
* The underlying contents of the encrypted block device are arbitrary, making it useful for encrypting swap devices. The encrypting can also be useful with certain databases that use specially formatted block devices for data storage.
* Uses the existing device mapper kernel subsystem.
* Provides passphrase strengthening, which protects against dictionary attaches.
* Allows users to add backup keys or passphrases because LUKS devices contain multiple key slots.







 What LUKS doesn't do 



* Allow applications that require many (more than eight) users to have distinct access keys to same devices.
* Work with applications that require file-level encryption.
The encrypting can also be useful with certain databases that use specially formatted block devices for data storage.
* Uses the existing device mapper kernel subsystem.
* Provides passphrase strengthening, which protects against dictionary attaches.
* Allows users to add backup keys or passphrases because LUKS devices contain multiple key slots.







 What LUKS doesn't do 



* Allow applications that require many (more than eight) users to have distinct access keys to same devices.
* Work with applications that require file-level encryption. For more information, see [RHEL Security Guide](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/security_guide/index).







 Setting up a LUKS-encrypted volume with Endurance Block Storage for Classic 

The process of data encryption creates a load on the host that might potentially impact performance.



1. Type the following command at a shell prompt as root to install the required package:

 yum install cryptsetup-luks
2. Get the disk ID:

 fdisk –l | grep /dev/mapper
3. Locate your volume in the listing.
4. Encrypt the block device;



1. This command initializes the volume, and you can set a passphrase.

 cryptsetup -y -v luksFormat /dev/mapper/3600a0980383034685624466470446564
2. Respond with YES (all uppercase letters.)
3. The device now appears as an encrypted volume:

 blkid | grep LUKS
/dev/mapper/3600a0980383034685624466470446564: UUID="46301dd4-035a-4649-9d56-ec970ceebe01" TYPE="crypto_LUKS"



5. Open the volume, and create a mapping.
Instead, use [Kubernetes secrets](https://kubernetes.io/docs/concepts/configuration/secret/), which are base64 encoded by default. To enable encryption of your Kubernetes secrets, [enable a key management service (KMS) provider](https://cloud.ibm.com/docs/containers?topic=containers-encryptionkeyprotect) for your cluster.
5. Persistent storage encryption: You can choose to store data by [setting up file, block, object, or software-defined Portworx persistent storage](https://cloud.ibm.com/docs/containers?topic=containers-storage-plan). If you store your data on file or block storage, your data is automatically encrypted at rest. If you use object storage, your data is also encrypted during transit. With Portworx, you can choose to [set up volume encryption](https://cloud.ibm.com/docs/containers?topic=containers-storage_portworx_encryption) to protect your data during transit and at rest. The IBM Cloud infrastructure storage instances save the data on encrypted disks, so your data at rest is encrypted.







 Understanding Key Management Service (KMS) providers 

You can protect Kubernetes secrets and any credentials stored in a secret by enabling a [key management service (KMS) provider](https://kubernetes.io/docs/tasks/administer-cluster/kms-provider/) which provides an encryption solution using a root key you control.



 Available KMS providers 

By default, IBM Cloud Kubernetes Service supports the following KMS providers.



* IBM® Key Protect for IBM Cloud® for [public cloud](https://cloud.ibm.com/docs/key-protect?topic=key-protect-getting-started-tutorial) or [on-prem](https://www.ibm.com/docs/en/cloud-private/3.2.0?topic=apis-key-management-service) environments.