vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Different columns can be encrypted with different keys.
6. By default, the main Parquet metadata module (the file footer) is encrypted to hide the file schema and list of sensitive columns. However, you can choose not to encrypt the file footers in order to enable legacy readers (such as other Spark distributions that don't yet support Parquet encryption) to read the unencrypted columns in the encrypted files.
7. Encryption keys can be managed in one of two ways:



* Directly by your application. See [Key management by application](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-key-management-application-serverless).
* By IBM® Key Protect for IBM Cloud®, a centralized key management system (KMS) for generating, managing, and destroying encryption keys used by IBM Analytics Engine. See [Key management by Key Protect](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-key-management-key-protect-serverless){: {: external}}.



IBM® Key Protect for IBM Cloud® helps you manage your encrypted keys by aligning with IBM Cloud Identity and Access Management (IAM) roles.

Note: Only master encryption keys (MEKs) need to be managed (either by your application, or by Key Protect). Data encryption keys (DEKs) are automatically handled by Spark/Parquet. For details on key handling inside Parquet encryption, see [Internals of encryption key handling](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-parquet-encryption-serverlessinternals-of-encryption-key-handling).

For each sensitive column, you must specify which master key to use for encryption. Also, a master key must be specified for the footer of each encrypted file (data frame) - because the footers keep metadata like the schema and list of sensitive columns, that can be sensitive too. By default, the footer key will be used for footer encryption.
After the connection is successful you can begin entering the credentials for your on-premises data source.





 Data source connection and data isolation 

[Connecting to external data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) reduces the data isolation of your service instance because data in transit between the source and the service cannot be isolated. All other isolation (at-rest, administration, query) remains in full. All in-flight communication between and within services and data sources is encrypted using TLS v1.2. The private keys for the TLS certificates are encrypted at rest using AES-256-GCM, the service certificates expire every three years, and the certificate revocation lists are updated monthly. All credentials are sent over an encrypted connection using TLS v1.2 and encrypted at rest using AES-256. Connections to those data sources use whatever secure protocols are supported by those data sources.
eyJjaXBoZXJ0ZXh0IjoiKzhjbHVqcUNP ...<redacted>... NmY3MTJjNGViIn0=

 capture the ciphertext
$ CIPHERTEXT=eyJjaXBoZXJ0ZXh0IjoiKzhjbHVqcUNP ...<redacted>... NmY3MTJjNGViIn0=

 unwrap the ciphertext to reveal the plaintext (DEK)
$ ibmcloud kp key unwrap 807eb0a6-cc10-4bfe-8331-41a6f712c4ea $CIPHERTEXT

Unwrapping key...
OK
Plaintext Rewrapped Plaintext
mXW4GmBhkRG1+Dzkx1D6dMX99a4KyYwtwbghaaLEQZ0=





 Example 2 

Generate a base64-encoded data encryption key (DEK) and wrap it with the root key.

 create a root key
$ KEY_ID=$(ibmcloud kp key create my-root-key --output json | jq -r '.["id"]')

$ echo $KEY_ID

9052d914-bebe-4177-90e9-6073a167b507

 create a random, base64-encoded, 32-byte data encryption key (DEK)
$ PLAINTEXT=$(openssl rand -base64 32)

$ echo $PLAINTEXT

tEqoiCu9CeUsbqRe6yBUkrJg80QKKG/z46OkX0YQMV8=

 wrap the plaintext key with the root key, creating a DEK (this is the ciphertext)
$ CIPHERTEXT=$(ibmcloud kp key wrap $KEY_ID -p $PLAINTEXT --output json | jq -r '.["Ciphertext"]')

$ echo $CIPHERTEXT
* [Review the checklist](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-planningplanning-for-data-encryption) for planning data encryption and complete the [prerequisites](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-encryption-planningbyok-encryption-prereqs).
* [Create virtual server instances with customer-managed encryption volumes](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-instances-byok).
* [Create a Block Storage for VPC volume with customer-managed encryption](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-vpc-encryption).
* [Create file shares with customer-managed encryption](https://cloud.ibm.com/docs/vpc?topic=vpc-file-storage-vpc-encryption).
[IBM Db2 default encryption by using the standard PKCS #11 API](https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto//images/pkcs-db2.svg)

Figure 5. IBM Db2 default encryption by using the standard PKCS #11 API



With the PKCS #11 library integration, Hyper Protect Crypto Services supports the industry-standard PKCS #11 API. The Hyper Protect Crypto Services PKCS #11 library connects your database to Hyper Protect Crypto Services to perform cryptographic operations. The database system can invoke operations to manage the TDE master encryption keys or the master keys in the Hyper Protect Crypto Services PKCS #11 library. The Hyper Protect Crypto Services PKCS #11 library then interacts with your Hyper Protect Crypto Services instance to provide the highest level of security for storing and managing your TDE master encryption keys or your master keys in the cloud. It, in turn, provides the highest level of security to your data encryption keys and your data.



* For a tutorial on how to use TDE with Hyper Protect Crypto Services, see

[Tutorial: Using Oracle Transparent Database Encryption with Hyper Protect Crypto Services PKCS #11](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-tutorial-tde-pkcs11).
* For a tutorial on how to use Db2 default encryption with Hyper Protect Crypto Services, see

[Using IBM Db2 default encryption with Hyper Protect Crypto Services PKCS #11](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-tutorial-db2-pkcs11).







 Protecting data in transit with TLS/SSL offloading 

Transport Layer Security (TLS) and Secure Sockets Layer (SSL) are cryptographic protocols that are designed to provide communication security over a computer network. The TLS/SSL protocol aims primarily to provide privacy and data integrity between two or more communicating computer applications.
You can create data volumes with customer-managed encryption by calling the [Virtual Private Cloud (VPC) API](https://cloud.ibm.com/apidocs/vpc).

Make a POST /volumes request to create a volume encrypted using your own encryption keys. Use the encryption_key parameter to specify your customer root key (CRK), shown in the example as crn:[...key:...].

You can also specify the CRN of a root key from a different account in the POST /volumes call. For more information, see [About cross account key access and use](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-byok-cross-acct-key&interface=uibyok-vol-cross-acct-about).

The following example creates a general-purpose data volume with customer-managed encryption.

curl -X POST "$vpc_api_endpoint/v1/volumes?version=2022-06-22&generation=2" -H "Authorization: $iam_token" -d '{
"name": "my-volume-1",
"iops": 100,
"capacity": 20,
"zone": {
"name": "us-south-3"
},
"profile": {
"name": "general-purpose"
},
"encryption_key":{
"crn":"crn:[...key:...]"
},
"resource_group": {
"id": "a342dbfb-3ea7-48d1-96e8-2825ec5feab4"
}
}

A successful response looks like the following example.

{
"id": "8948ad59-bc0f-7510-812f-5dc64f59fab8",
"crn": "crn:[...]",
"name": "my-volume-1",
The latest key version (rewrappedKeyVersion) that is associated with the new ciphertext is also returned. Store and use the new ciphertext value for future envelope encryption operations so that your data is protected by the latest root key.





 Decoding your key material 

When you unwrap a data encryption key, the key material is returned in base64 encoding. You will need to decode the key before encrypting it.



 Using OpenSSL to decode key material 



1. Download and install [OpenSSL](https://github.com/openssl/opensslfor-production-use).
2. Decode your base64 encoded key material string by running the following command:

openssl base64 -d -in <infile> -out <outfile>



Replace the variables in the example request according to the following table.



Table 2. Describes the variables that are needed to decode your key material.

 Variable Description 

 infile The name of the file where your base64 encoded key material string resides. 
 outfile The name of the file where your decoded key material will be be outputted once the command has ran. 



If you want to output the decoded material in the command line directly rather than a file, run the command openssl enc -base64 -d <<< '<key_material_string>', where key_material_string is the returned plaintext from your unwrap request.
: The enterprise ID of the Box account.

Public Key ID
: The public key ID that Box generates.

Private Key
: A part of the key pair that is generated to interact with the Box website.

Passphrase
: The passphrase that is required to decrypt the private key if the private key is an encrypted file.
5. Click Next.
6. Name the collection.
7. If the language of the documents in Box is not English, select the appropriate language.

For a list of supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).
8. Optional: Change the synchronization schedule.

For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscrawlschedule).
9. Choose the folders that you want to crawl.
10. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

When you choose to list extensions for file types to exclude, you must add at least one file extension.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).
11. If you want the web crawl to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
12. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page.
Securing your data 

To ensure that you can securely manage your data when you use IBM Cloud Activity Tracker, it is important to know exactly what data is stored and encrypted, and how you can delete any stored personal data.



 What data is stored in IBM Cloud Activity Tracker 

IBM Cloud Activity Tracker collects management and data events from IBM Cloud services and resources:



* Management Events are generated when an API call changes the state of a Cloud resource. A resource might be an entire service instance or a resource managed by the service.
* Data Events are generated when an API call reads or modifies a resource's data.



Data from [IBM Cloud services and resources](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-cloud_services) is collected automatically. However, some services might require an upgrade of the service plan, a configuration setting, or both, for you to be able to collect and analyze them.





 How your data is stored and encrypted in Activity Tracker 

Data is hosted on the IBM Cloud.

Events can be classified as global or location-based. [Learn more](https://cloud.ibm.com/docs/services/activity-tracker?topic=activity-tracker-event_types).



* Location-based events maintain data locality to the services that run in that Cloud location.
* Global events are collected and made available through the region in your account that you configure to collect global auditing events for Activity Tracker. Global events are collected and made available through the IBM Cloud Activity Tracker instance that is configured in Frankfurt for Activity Tracker hosted event search offerings.



IBM Cloud Activity Tracker collects and aggregates logs.



* All the data that is hosted in an auditing instance is encrypted at rest using AES 256.
* When an IBM Cloud service sends data to an auditing instance, data is encrypted in transit over HTTPS.
* When a user requests an export, the data is encrypted during transit, and is also encrypted at rest in IBM Cloud Object Storage (COS).



The service plan that you choose for an IBM Cloud Activity Tracker instance defines the number of days that data is stored and retained.
* Public key: The public component of a key pair that is used by anyone to encrypt messages intended for a particular recipient that has access to the private key of the key pair. The public key is also used to verify signatures created by the private key.
* Private key: The private component of a key pair that is used to decrypt messages. The private key is also used to create signatures.
* Secret key: A secret key is a generated stream of bits that is used to encrypt and decrypt messages symmetrically.





Zoom

![PKCS #11 object classes](https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto//images/object-class.svg)

Figure 2. PKCS #11 object classes

Currently, the data object class is not supported and only X.509 Public Key Certificate objects are supported in this implementation.





 Cloud Identity and Access Management 

IBM Cloud Identity and Access Management (IAM) provides user authentication and access control for the implementation of the PKCS #11 API. By using API keys, the PKCS #11 library obtains bearer tokens that are used to perform Cryptoki API calls.

This implementation of PKCS #11 equates an API key with a user's PIN. For more information about setting up the service ID and the corresponding API key, see [Create service IDs for the SO user, normal user, and anonymous user](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-best-practice-pkcs11-accessstep2-create-service-id-api-key).





 Crypto service 

As part of the PKCS #11 library initialization process, a gRPC connection is made from the PKCS #11 library to the IBM Cloud.
AC-19 (5) - Full Device / Container-based Encryption 



  Control requirements 

AC-19 (5) - 0
:   The organization employs [IBM Assignment: file level, full disk/device, or both] to protect the confidentiality and integrity of information on [Assignment: organization-defined mobile devices].





  NIST supplemental guidance 

Container-based encryption provides a more fine-grained approach to the encryption of data/information on mobile devices, including for example, encrypting selected data structures such as files, records, or fields.
[New system property](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-sn-18.png)

Figure 19. ServiceNow - New system property
5. Add the App Configuration instance ID. Use the password2 type to store the key. This type will encrypt the data and can only be decrypted with the ServiceNow instance.

Zoom

![System property](https://cloud.ibm.com/docs-content/v1/content/f3dada89f147af3b88388a2218ddbf64b3c41e5f/app-configuration/images/ac-sn-15.png)

Figure 20. ServiceNow - System property
6. Click Submit to save the key in ServiceNow instance. Once saved, you can see that the data is encrypted.
Encrypting SQL queries with Key Protect 

By default, IBM Cloud® Data Engine uses server-managed encryption at rest for all job information that is recorded about your stored queries. If you are processing sensitive data in your queries that is governed by special regulations, you can additionally use customer-managed keys to encrypt the SQL query texts and error messages that are stored in the job information.

IBM® Key Protect for IBM Cloud® is a centralized key management system (KMS) for generating, managing, and destroying encryption keys that are used by IBM Cloud® services. You can associate a key that is managed in IBM Key Protect with an SQL query instance to encrypt your queries. Customer key encryption can be configured only when you are creating the SQL query instance, and that configuration cannot be changed later. However, you can always create a new SQL query instance with a different configuration and use that for future queries. Encryption is only available for instances based on the Data Engine Standard plan.



 About customer-managed keys 

Data Engine uses

envelope encryptionto implement customer-managed keys. Envelope encryption describes encrypting one encryption key with another encryption key. The key used to encrypt the actual data is known as adata encryption key (DEK). The DEK itself is never stored but is wrapped by a second key that is known as the key encryption key (KEK) to create a wrapped DEK. To decrypt data, the wrapped DEK is unwrapped to get the DEK. This process is possible only by accessing the KEK, which in this case is your root key that is stored in Key Protect.

You own the KEK, which you create as a root key in the Key Protect service. The Data Engine service never sees the root (KEK) key. Its storage, management, and use to wrap and unwrap the DEK is performed entirely within the key management service. If you disable or delete the key, the data can no longer be decrypted.

Key Protect keys are secured by FIPS 140-2 Level 3 certified cloud-based

hardware security modules (HSMs).
callback(err, response);
});







 ReencryptSingle 

With the ReencryptSingle function, you can decrypt data with the original key and then encrypt the raw data with a different key in a single call within the cloud HSM. The key types that are used for this operation can be the same or different. This function is an IBM EP11 extension to the standard PKCS #11 specification. This single call is a viable option where a large amount of data needs to be reencrypted with different keys, and bypasses the need to perform a combination of DecryptSingle and EncryptSingle functions for each data item that needs to be reencrypted. It does not return any state to host and returns only the reencrypted data.

Enterprise PKCS #11 over gRPC

Enterprise PKCS #11





 Description Binds to EP11 m_ReencryptSingle. 
 Parameters message ReencryptSingleRequest { bytes DecKey = 1; bytes EncKey = 2; Mechanism DecMech = 3; Mechanism EncMech = 4; bytes Ciphered = 5; } message ReencryptSingleResponse { bytes Reciphered = 6; } 
 Return values Wraps EP11 error into message Grep11Error. 



Code snippets



* Golang code snippet

var msg = []byte("Data to encrypt")
EncryptKey1Request := &pb.EncryptSingleRequest{
Key: GenerateKey1Response.KeyBytes,
Mech: &pb.Mechanism{Mechanism: ep11.CKM_AES_CBC_PAD, Parameter: util.SetMechParm(iv)},
Plain: msg,
}
EncryptKey1Response, err := cryptoClient.EncryptSingle(context.Background(), EncryptKey1Request)
if err != nil {
At-rest encryption All data that is stored in an IBM Cloudant instance is encrypted at rest by using LUKS1 with 256-bit Advanced Encryption Standard (AES-256). By default, IBM Cloudant manages the encryption keys for all environments. If you require bring-your-own-key (BYOK) encryption for encryption-at-rest, BYOK is enabled by using your encryption key that is stored in an IBM Cloud Key Protect instance. IBM Cloudant supports the BYOK feature for new IBM Cloudant Dedicated Hardware plan instances that are deployed in all regions. For more information, see the [Creating an IBM Cloudant Dedicated Hardware plan instance](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-and-leveraging-an-ibm-cloudant-dedicated-hardware-plan-instance-on-ibm-cloud) tutorial for details on how to choose BYOK at provisioning time. 
 In-flight encryption All access to IBM Cloudant is encrypted by using HTTPS. 
 Client-side encryption Customers can use client-side encryption to ensure that the data protection is controlled by the data owner and the data is never visible to the service provider. 
 TLS IBM Cloudant requires the use of TLS 1.2+. IBM Cloudant strongly recommends that you do not pin certificates in your application. Certificates renew regularly, at least annually, and intermediate and root certificates could change when they do. IBM Cloudant does not send out notifications before certificate renewals. We recommend that you keep your certificate truststore up to date with the latest root certificates. IBM Cloudant acquires its certificates from DigiCert. You can find their root certificates on the [DigiCert Trusted Root Authority Certificates](https://www.digicert.com/kb/digicert-root-certificates.htm) page. IBM Cloudant sends a notification if we move to a different certificate authority. 
 Public Endpoints All IBM Cloudant instances are provided with external endpoints that are publicly accessible. 
 Private Endpoints All instances that you deploy on Dedicated Hardware plan environments also have private (internal) endpoints.
(https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto//images/envelope-encryption.svg)

Figure 1. Contextual view of envelope encryption

Envelope encryption is treated briefly in the NIST Special Publication 800-57, Recommendation for Key Management. To learn more, see [NIST SP 800-57 Pt. 1 Rev. 4](https://www.nist.gov/publications/recommendation-key-management-part-1-general-0)





 Wrapping keys 

Root keys help you group, manage, and protect data encryption keys (DEKs) stored in the cloud. You can wrap one or more DEKs with advanced encryption by designating a root key in Hyper Protect Crypto Services that you can fully manage.

After you designate a root key in Hyper Protect Crypto Services, you can send a key wrap request to the service by using the [Hyper Protect Crypto Services key management service API](https://cloud.ibm.com/apidocs/hs-cryptoactiononkey). The key wrap operation provides both confidentiality and integrity protection for a DEK.

The following diagram shows the key wrapping process in action.

Zoom

![Wrapping data](https://cloud.ibm.com/docs-content/v1/content/217f108cc44e2ce15fb692d4b57b4c628464e908/hs-crypto//images/wrapping-keys.svg)

Figure 2. Wrapping data





 Unwrapping keys 

Unwrapping a data encryption key (DEK) decrypts and authenticates the contents within the key, returning the original key material to your data service.

If your business application needs to access the contents of your wrapped DEKs, you can use the [Hyper Protect Crypto Services key management service API](https://cloud.ibm.com/apidocs/hs-cryptoactiononkey) to send an unwrap request to the service.
Many of today's server and network technologies allow for easier configuration and implementation to minimize the impact on utilization. Implementing encryption of data in transit from endpoint to endpoint both remotely and internally is mandatory in today's cyber risk environment.

The following are considered equality check operators and are supported:



* =
* <>
* IS NULL
* IS NOT NULL
* IN
* NOT
* JOIN (all types)
* GROUP BY
* DISTINCT



Indexes can be created on encrypted columns, but the ciphertext will be used to create the index and not the underlying cleartext values.





 Unsupported Data Types 

Only timestamptz and timetz data types are not supported for the PostgreSQL Database in Data Security Broker.