vania-janet/multiturn-rag-retrieval-data huggingface dataset:
To find the ID of the snapshot from the CLI, see [View snapshots from the CLI](https://cloud.ibm.com/docs/vpc?topic=vpc-snapshots-vpc-view&interface=clisnapshots-vpc-view-cli).

See the following example.

ibmcloud is instance-volume-attachment-add data-vol-1 a67f49de-fccc-4e5c-824e-dcbd06d009af --profile general-purpose --source-snapshot 52de6e85-7068-4247-90fd-d2fa91fd9864

For more information about available command options, see [ibmcloud is instance-volume-attachment-add](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-referenceinstance-volume-attachment-add).





 Creating a stand-alone volume from a snapshot from the CLI 

You can create a stand-alone Block Storage for VPC data volume from a snapshot from the CLI. The volume is not attached to a virtual server instance. When you run the ibmcloud is volume command, the response shows the attachment type as unattached. You can attach the volume to an instance later.

Use this option when you're not sure which virtual server instance you want to attach the volume to. Or, if you want to restore data from an unattached volume that was detached from an instance.

Run the ibmcloud is volume-create command and specify the snapshot parameter and name or ID of the snapshot. For more information, see [Create a stand-alone Block Storage for VPC volume from a snapshot](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-block-storage&interface=clicreate-vol-from-snapshot-cli).

For more information about available command options, see [ibmcloud is volume-create](https://cloud.ibm.com/docs/vpc?topic=vpc-vpc-referencevolume-create).
EncryptFinalResponse, err := cryptoClient.EncryptFinal(context.Background(), EncryptFinalRequest)
* JavaScript code snippet

client.EncryptFinal({
State: state
}, (err, data={}) => {
cb(err, Buffer.concat([ciphertext, data.Ciphered]));
});







 EncryptSingle 

The EncryptSingle function processes data in one pass with one call. It does not return any state to host and returns only the encrypted data. This function is an IBM EP11 extension to the standard PKCS #11 specification and is a combination of the EncryptInit and Encrypt functions. It enables you to complete an encryption operation with a single call instead of a series of calls.

Enterprise PKCS #11 over gRPC

Enterprise PKCS #11





 Description Binds to EP11 m_EncryptSingle 
 Parameters message EncryptSingleRequest { bytes Key = 1; Mechanism Mech = 2; bytes Plain = 3; } message EncryptSingleResponse { bytes Ciphered = 4; } 
 Return values Wraps EP11 error into message Grep11Error. 



Code snippets



* Golang code snippet

// Generate 16 bytes of random data for the initialization vector
GenerateRandomRequest := &pb.GenerateRandomRequest{
Len: (uint64)(ep11.AES_BLOCK_SIZE),
}
GenerateRandomResponse, err := cryptoClient.GenerateRandom(context.Background(), GenerateRandomRequest)
if err != nil {
return nil, fmt.Errorf("GenerateRandom error: %s", err)
}

iv := GenerateRandomResponse.Rnd[:ep11.AES_BLOCK_SIZE]
fmt.Println("Generated IV")

plainText := "Encrypt this message"
'uri':'cos://<region_name>/<bucket_name>/xx123.Rhel_7_encrypted.raw',
'bootMode':'my boot mode',
'cloud-init': True,
'byol': True,
'encrypted': True,
'ibmApiKey':'<api_key>',
'crkCrn': 'crn:v1:bluemix:public:hs-crypto:us-south:a/0d06ba51fa0e431290956d1761da1b7b:5ef6cebe-26d7-4ef3-abdc-fb50f345780f:key:a9640391-aec5-4c86-8942-6e6c59bb40b5',
'wrappedDek':'my wrapped DEK',
}
ret = group_svc.createFromIcos(config)
print(ret)
Show more

For more information about locating values that are needed to import the encrypted image from IBM Cloud Object Storage, see the following table.



Table 1. Values needed for importing encrypted image

 Field Value 

 ibmApiKey Specify the API key that you noted when you created it. If the API key is lost, you must create a new API key. For more information, see [Managing your API keys](https://cloud.ibm.com/docs/account?topic=account-userapikeyuserapikey). 
 crkCrn Specify the [Cloud Resource Name (CRN)](https://cloud.ibm.com/docs/account?topic=account-crn) for the root key that you used to wrap your data encryption key. To locate and copy your root key CRN, go to the [Key Protect service instance !
GenerateRandomResponse, err := cryptoClient.GenerateRandom(context.Background(), GenerateRandomRequest)
* JavaScript code snippet

client.GenerateRandom({
Len: ep11.AES_BLOCK_SIZE
}, (err, response) => {
callback(err, response);
});









 Encrypting and decrypting data 

By specifying the cryptographic mechanism, you can perform symmetric or asymmetric encryption and decryption functions. You might need to call a series of subfunctions to encrypt or decrypt data. For example, the multi-part data encryption operation is composed of the EncryptInit, EncryptUpdate, and EncryptFinal suboperations.



 EncryptInit 

The EncryptInit function initializes an encryption operation. You need to call this function first to perform an encryption.

Enterprise PKCS #11 over gRPC

Enterprise PKCS #11

PKCS #11





 Description Binds to EP11 m_EncryptInit, which is an implementation of PKCS #11 C_EncryptInit. 
 Parameters message EncryptInitRequest { Mechanism Mech = 2; bytes Key = 3; } message EncryptInitResponse { bytes State = 1; } 
 Return values Wraps EP11 error into message Grep11Error. 



Code snippets



* Golang code snippet

// Generate 16 bytes of random data for the initialization vector
GenerateRandomRequest := &pb.GenerateRandomRequest{
Len: (uint64)(ep11.AES_BLOCK_SIZE),
}
GenerateRandomResponse, err := cryptoClient.GenerateRandom(context.Background(), GenerateRandomRequest)
if err != nil {
return nil, fmt.Errorf("GenerateRandom error: %s", err)
}
To protect your data in a Portworx volume, you can create an instance of a KMS provider such as IBM Key Protect or Hyper Protect Crypto Services.

If you don't want to use IBM Key Protect or Hyper Protect Crypto Services root keys to encrypt your volumes, you can select Kubernetes Secret as your Portworx secret store type during the Portworx installation. This setting gives you the option to store your own custom encryption key in a Kubernetes secret after you install Portworx. For more information, see the [Portworx documentation](https://docs.portworx.com/key-management/kubernetes-secrets/configuring-kubernetes-secrets-with-portworx).



 Getting your KMS instance and credentials 



 Setting up volume encryption with Hyper Protect Crypto Services 



1. [Create a service instance](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-provision&interface=ui).
2. [Create a root key](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-create-root-keys&interface=ui).
3. [Create a service ID](https://cloud.ibm.com/docs/account?topic=account-serviceids&interface=cliserviceids).
4. [Retrieve the Key Management public endpoint URL](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-regionsservice-endpoints). Make sure that you note your endpoint in the correct format; for example, https://api.us-south.hs-crypto.cloud.ibm.com:<port>. For more information, see the [Hyper Protect Crypto Services API documentation](https://cloud.ibm.com/apidocs/hs-cryptogetinstance).







 Setting up volume encryption with IBM Key Protect 



1.
Adding Keystore in Data Security Broker Manager 

Before you can enroll your applications, add databases, and enable encryption, you must enroll your Keystore, so that the Data Security Broker Manager can access and create data encryption keys (DEKs) that is used to protect your data.

Follow the steps below to enroll a Keystore that you can use with Data Security Broker Shields and databases.



1.  Login to Data Security Broker Manager.
2.  Select Keystores from the left navigation and click Add Keystore +.

Zoom

![Keystores](https://cloud.ibm.com/docs-content/v1/content/57ab3ec034294d8a6de510b97bec3035d4654761/security-broker/images/keystore.svg)

Figure 1. Keystores
3.  Specify a Keystore name and provide a valid Description.
4.  Select the Keystore Type from the Keystore Type dropdown menu. Enter values for the Instance ID, App Namespace, IBM Key Protect Alias, and IAM API Key. Select the region in the IBM Region drop down and choose the Data Encryption Key (DEK) Storage type from the DEK Storage Type drop down. For information on how to create encryption keys, see [Creating and importing encryption keys](https://cloud.ibm.com/docs/key-protect?topic=key-protect-tutorial-import-keys).

Keystore parameters are specific to each Keystore type or vendor.

For more information on configuring the IBM Key Protect and configuring the IBM Cloud Hyper Protect Crypto Services (HPCS), refer to the [Configure IBM Key Protect and HPCS](https://cloud.ibm.com/docs/security-broker?topic=security-broker-sb_configure_Keyprotect) section.
5.  Click Add Keystore to create a Keystore.
6.  Once you have completed adding a keystore, the next step is to enroll an application. Refer to the [Enrolling an Application in Data Security Broker Manager](https://cloud.ibm.com/docs/security-broker?topic=security-broker-sb_enroll_app) section to enroll an application in the Data Security Broker Manager.
Background knowledge: From the HPCR image version ibm-hyper-protect-container-runtime-1-0-s390x-9, for new Hyper Protect Virtual Servers for VPC instances, the data volume is partitioned into two parts. The first partition (100 Mib) is reserved for internal metadata only (not to be accessed by a workload); the second partition remains as the data volume for workload. Only new volumes are partitioned.

Zoom

![Integration with key management service](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/hpvs-kms-integration.svg)

Integration with key management service

Currently, only Hyper Protect Crypto Services is supported as the key management service.

The following table is a summary of the seeds. The third seed is the one provided by your key management service.



Table 1. Seeds that are used to generate the LUKS encryption passphrase

 Seed Provider From Required or optional 

 seed1 Deployer persona env - volumes section of the contract Required 
 seed2 Workload persona workload - volumes section of the contract Required 
 seed3 Hyper Protect Crypto Services Hyper Protect Crypto Services generates the third seed and wraps it with the CRK only if kms details are provided in the contract. This is done by Hyper Protect Virtual Servers calling a wrap API. The wrapped seed is stored in the metadata partition of the data volume. Optional 



Zoom

![LUKS passphrase generation](https://cloud.ibm.com/docs-content/v1/content/f02b9c9adcbb7328d95a45e105cec371d50f15eb/vpc/images/hpvs-luks.svg)

LUKS passphrase generation

The key daemon is started if the volume is secured with protection from the KMS instance. It's responsible for reacting to status changes of the CRK (which will be introduced later in this documentation).



 About customer-managed keys
key_protect_key The root key CRN of a Key Management Services like Key Protect or Hyper Protect Crypto Service (HPCS) that you want to use for disk encryption. 
 key_protect_instance The instance CRN of a Key Management Services like Key Protect or Hyper Protect Crypto Service (HPCS) that you want to use for disk encryption. 
 remote_leader_id A CRN of the leader database to make the replica(read-only) deployment. The leader database is created by a database deployment with the same service ID. A read-only replica is set up to replicate all of your data from the leader deployment to the replica deployment by using asynchronous replication. 





* (Optional) For file share configuration, you specify information for the file share and the file share replica:



* share_size - size of the file share, min 10 GB max 3200 GB
* share_profile - profile for the file share, either tier-3iops, tier-5iops, or tier-10iops
* enableFileShare - whether to enable the file share, either true or false
* replica_zone - the zone for the replica, should be in the same region as the infrastructure, for example, if the infrastructure is in us-east-1, the replica should be in us-east-2 or us-east-3
* replica_profile - should match share_profile, either tier-3iops, tier-5iops, or tier-10iops









 Customizing a three-tier highly available infrastructure VPC in a single availability zone with the catalog tile 



1. From the IBM Cloud catalog, select HA Infrastructure Deployment for 3-tier SAZ.
2. Configure your workspace:



* Enter a name for the workspace.
* Select a Resource group.
* Select a Location for your workspace. The workspace location does not have to match the resource location.



3. Set the required deployment variables. Review the input parameters and provide values that match your solution. Some parameters have default values. The values that you must specify include:



 Component type Parameter values needed 

 User-specific information <br><br> * API key<br> * SSH key name.
The key management service instance includes the customer root key that you want to use for customer-managed encryption. 
 Encryption service instance If you provision multiple key management service instances in your account, select the one that includes the customer root key that you want to use for customer-managed encryption. 
 Key name Select the data encryption key within the Key Protect instance that you want to use for encrypting the volume. 
 Key ID Displays the key ID that is associated with the data encryption key that you selected. 



If you created your Key Protect or Hyper Protect Crypto Services instance by using a private endpoint, the root keys that were created by using that instance are not shown in the UI. You must use the CLI or API to access and use these root keys.





 Editing boot volumes to use customer-managed encryption in the UI 

When you create an instance from the UI, you can specify customer-managed encryption by editing the boot volume properties. For more information, see [Provisioning virtual server instances with customer-managed encryption volumes in the UI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-instances-byokprovision-byok-ui).





 Creating customer-managed encryption data volumes from the CLI 



 Before you begin 

Before you can use the CLI, you must install the IBM Cloud CLI and the VPC CLI plug-in. For more information, see the [CLI prerequisites](https://cloud.ibm.com/docs/vpc?topic=vpc-set-up-environmentcli-prerequisites-setup).



1. Log in to the IBM Cloud.

ibmcloud login --sso -a cloud.ibm.com

This command returns a URL and prompts for a passcode. Go to that URL in your browser and log in. If successful, you get a one-time passcode. Copy this passcode and paste it as a response on the prompt. After successful authentication, you are prompted to choose your account. If you have access to multiple accounts, select the account that you want to log in as. Respond to any remaining prompts to finish logging in.
2.
GenerateRandomRequest := &pb.GenerateRandomRequest{
Len: (uint64)(ep11.AES_BLOCK_SIZE),
}
GenerateRandomResponse, err := cryptoClient.GenerateRandom(context.Background(), GenerateRandomRequest)
if err != nil {
return nil, fmt.Errorf("GenerateRandom error: %s", err)
}
iv := GenerateRandomResponse.Rnd[:ep11.AES_BLOCK_SIZE]
fmt.Println("Generated IV")

EncryptInitRequest := &pb.EncryptInitRequest{
Mech: &pb.Mechanism{Mechanism: ep11.CKM_AES_CBC_PAD, Parameter: util.SetMechParm(iv)},
Key: GenerateKeyResponse.KeyBytes,
}

EncryptInitResponse, err := cryptoClient.EncryptInit(context.Background(), EncryptInitRequest)
Show more
* JavaScript code snippet

client.EncryptInit({
Mech: {
Mechanism: ep11.CKM_AES_CBC_PAD,
ParameterB: iv
},
Key: key
}, (err, data={}) => {
cb(err, data.State);
});







 Encrypt 

The Encrypt function encrypts single-part data. You don't need to perform the EncryptUpdate and EncryptFinal suboperations for a single-part encryption. Before you call this function, make sure to run EncryptInit first.

Enterprise PKCS #11 over gRPC

Enterprise PKCS #11

PKCS #11





 Description Binds to EP11 m_Encrypt, which is an implementation of PKCS #11 C_Encrypt.
[Wrap a data encryption key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-wrap-keys) by using a root key that is stored in the Key Protect instance that you specify.

ibmcloud kp wrap KEY_ID -i $INSTANCE_ID
[-a, --aad ADDITIONAL_DATA]
[-p, --plaintext DATA_KEY]



 Required parameters 

KEY_ID
: The ID of the root key that you want to use for wrapping.





 Optional parameters 

-a, --aad
: The additional authentication data (AAD) that is used to further secure a key. If provided on wrap must be provided on unwrap.







 kp unwrap 

[Unwrap a data encryption key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-unwrap-keys) by using a root key that is stored in your Key Protect instance.

ibmcloud kp unwrap KEY_ID CIPHERTEXT_FROM_WRAP -i $INSTANCE_ID
[-a, --aad ADDITIONAL_DATA, ..]



 Required parameters 

KEY_ID
: The ID of the root key that you used for the initial wrap request.





 Optional parameters 

-a, --aad
: The additional authentication data (AAD) that was used to further secure a key. You can provide up to 255 strings, each delimited by a comma. If you supplied AAD on wrap, you must specify the same AAD on unwrap.

Important: The Key Protect service does not save additional authentication data. If you supply AAD, save the data to a secure location to ensure that you can access and provide the same AAD during subsequent unwrap requests.







 kp rotate 

[Rotate a root key](https://cloud.ibm.com/docs/key-protect?topic=key-protect-wrap-keys) that is stored in your Key Protect service.

ibmcloud kp rotate KEY_ID -i $INSTANCE_ID
[-k, --key-material KEY_MATERIAL]
Key management by application 

This topic describes how to manage column encryption keys by application. It explains how to provide master keys and how to write and read encrypted data using these master keys.



 Providing master keys 

To provide master keys:



1. Set the class implementing EncryptionPropertiesFactory:

parameter name: "parquet.crypto.factory.class"
parameter value: "com.ibm.parquet.key.management.IBMKeyToolsFactory"
2. Pass the explicit master keys, in the following format:

parameter name: "parquet.encryption.key.list"
parameter value: "<master key ID>:<master key (base64)> , <master key ID>:<master key (base64)>.."

For example:

sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
sc.hadoopConfiguration.set("parquet.encryption.key.list" , "k1:iKwfmI5rDf7HwVBcqeNE6w== , k2:LjxH/aXxMduX6IQcwQgOlw== , k3:rnZHCxhUHr79Y6zvQnxSEQ==")

The length of master keys before base64 encoding can be 16, 24 or 32 bytes (128, 192 or 256 bits).







 Writing encrypted data 

To write encrypted data:



1. Specify which columns to encrypted, and which master keys to use:

parameter name: "parquet.encryption.column.keys"
parameter value: "<master key ID>:<column>,<column>;<master key ID>:<column>,..."
2. Specify the footer key:

parameter name: "parquet.encryption.footer.key"
parameter value: "<master key ID>"

For example:

dataFrame.write
Before using index encryption, you should check the documentation on [PME](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-parquet-encryption) and make sure you are familiar with the concepts.

When using index encryption, whenever a key is configured in any Xskipper API, it's always the label NEVER the key itself.

To use index encryption:



1. Follow all the steps to make sure PME is enabled. See [PME](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-parquet-encryption).
2. Perform all regular PME configurations, including Key Management configurations.
3. Create encrypted metadata for a data set:



1. Follow the regular flow for creating metadata.
2. Configure a footer key. If you wish to set a plain text footer + object name column, set io.xskipper.parquet.encryption.plaintext.footer to true (See samples below).
3. In IndexBuilder, for each index you want to encrypt, add the label of the key to use for that index.



To use metadata during query time or to refresh existing metadata, no setup is necessary other than the regular PME setup required to make sure the keys are accessible (literally the same configuration needed to read an encrypted data set).







 Samples 

The following samples show metadata creation using a key named k1 as a footer + object name key, and a key named k2 as a key to encrypt a MinMax for temp, while also creating a ValueList for city, which is left in plain text.



* For Scala:

// index the dataset
val xskipper = new Xskipper(spark, dataset_path)
// Configuring the JVM wide parameters
val jvmComf = Map(
"io.xskipper.parquet.mdlocation" -> md_base_location,
parameter name: "parquet.encryption.column.keys"
parameter value: "<master key ID>:<column>,<column>;<master key ID>:<column>,..."
2. Specify the footer key:

parameter name: "parquet.encryption.footer.key"
parameter value: "<master key ID>"

For example:

dataFrame.write
.option("parquet.encryption.footer.key" , "k1")
.option("parquet.encryption.column.keys" , "k2:SSN,Address;k3:CreditCard")
.parquet("<path to encrypted files>")

Note: If either the "parquet.encryption.column.keys" parameter or the "parquet.encryption.footer.key" parameter is not set, an exception will be thrown.







 Reading encrypted data 

The required metadata is stored in the encrypted Parquet files.

To read the encrypted data:



1. Set the class implementing EncryptionPropertiesFactory:

sc.hadoopConfiguration.set("parquet.crypto.factory.class","com.ibm.parquet.key.management.IBMKeyToolsFactory")
2. Provide the encryption keys:

sc.hadoopConfiguration.set("parquet.encryption.key.list" , "k1:iKwfmI5rDf7HwVBcqeNE6w== , k2:LjxH/aXxMduX6IQcwQgOlw== , k3:rnZHCxhUHr79Y6zvQnxSEQ==")
3. Call the regular parquet read commands, such as:

val dataFrame = spark.read.parquet("<path to encrypted files>")







 Key rotation 

If key rotation is required, the following Hadoop configuration properties must be set:



* The parameters "parquet.encryption.key.list", "parquet.encryption.new.key.list"
These workloads are typically characterized by having a high percentage of data active at a time. Example applications include transactional and other performance-sensitive databases. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Endurance - <br>10 IOPS per GB tier It is designed for the most demanding workloads such as those created by NoSQL databases, and data processing for Analytics. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 Classic Performance - <br>A high-powered environment with custom IOPS It is designed to manage rapid data changes with well-defined performance requirements. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption at rest. 
 VPC Storage - <br>3 IOPS per GB tier It is designed for general-purpose workloads such as workloads that host small databases for web applications or store virtual machine disk images for a hypervisor. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption + Customer-managed encryption 
 VPC Storage - <br>5 IOPS per GB tier It is designed for high I/O intensity workloads that are characterized by a large percentage of active data, such as transactional and other performance-sensitive databases. 99.999999999% <br>(11 9's) 99.999% <br>(5 9's) Provider-managed AES-256 encryption + Customer-managed encryption 
 VPC Storage - <br>10 IOPS per GB tier It is designed for demanding storage workloads such as data-intensive workloads created by NoSQL databases, data processing for video, machine learning, and analytics.