vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* To crawl any domain that ends with ibm.com:

allow domain .ibm.com
* To crawl only port 443 on IBM domains that begin with server:

allow domain server.ibm.com:443



Rules to crawl URL prefixes
: Specify the HTTP and HTTPS prefixes that you want to either allow or forbid the crawler to crawl.

The wildcard character (*) can occur one or more times in the URL.

The order of the rules is significant. The crawler applies the first rule that matches a candidate URL.

You can define the following types of rules, for example:



* To crawl pages in the public directory on this domain:

allow prefix http://.ibm.com/public/
* To exclude all other directories on this domain:

forbid prefix http://.ibm.com/



Advanced crawler properties
: Only use when instructed to do so by IBM Support.
12. Optional: If you want to ignore any SSL certificates on the target website, set the Ignore certificate switch to On.

This option applies to HTTPS URLs only.
13. If you want the crawler to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
14. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.
* [Deleting environment variables with the CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-envvarenvvar-delete-cli)





[Working with secrets](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret)



* [What are secrets and why would I use them?](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-whatwhy)



* [What kind of secrets can I create in Code Engine?](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-categories)



* [Creating secrets](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-create)



* [Creating a secret from the console](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-create-ui)
* [Creating secrets with the CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-create-cli)



* [Updating secrets](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-update)



* [Updating secrets from the console](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-update-ui)
* [Updating secrets with the CLI](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-update-cli)



* [Referencing secrets](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-ref)



* [Referencing secrets from the console](https://cloud.ibm.com/docs/codeengine?topic=codeengine-secretsecret-ref-ui)
broker-3-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
broker-5-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
broker-0-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
broker-1-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
broker-4-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
broker-2-abcdabcdabcdabcd.kafka.svc07.us-south.eventstreams.cloud.ibm.com:9093
Consumer Group: knative-kafka-source-a4072fe1-1dfa-4470-9d07-bf7a0ff8e340
Topics:
kafka-topic1
Secret key reference (user): kafka-subscription-secret.username
Secret key reference (password): kafka-subscription-secret.password
Ready: true

Conditions:
Type OK Age Reason
ConnectionEstablished true 24s
InitialOffsetsCommitted true 24s
Ready true 24s
Scheduled true 24s
SinkProvided true 24s

Events:
Type Reason Age Source Messages
Normal FinalizerUpdate 26s kafkasource-controller Updated "mykafkasubscription" finalizers

When Ready is true, then the Kafka subscription is ready to trigger events per the specified schedule.







 ibmcloud ce subscription kafka list 

List all Kafka event subscriptions in a project.
List your Key Protect service instances and identify the ID and region of the instance where you want to store the secret: ibmcloud resource service-instances --long --service-name kms. The instance ID is in output column "GUID", the region in column "Location".
4. Select the appropriate target region with ibmcloud kp region-set <region>.
5. Optionally review the existing keys in the instance with following command: ibmcloud kp list -c -i <kp instance id>.
6. Key Protect accepts only base64 encoded secret data. Issue the following command to add a custom secret as properly encoded content to Key Protect: ibmcloud kp create "<key name>" -i <kp instance id> -s -k echo -ne "<your custom secret>" | base64 . The key name that you provide is for your reference only and can be an arbitrary string.
7. List the keys again (as in step 5) to retrieve the key ID and CRN for the new key that you created. Use the key CRN in an [access secret clause](https://cloud.ibm.com/docs/sql-query?topic=sql-query-sql-referenceaccessSecrets) of an SQL query.



You can verify the correct setup and function of your IBM Cloud CLI by downloading and running [this script](https://raw.githubusercontent.com/ibm-cloud-docs/sql-query/master/kp_check.sh) in your command line. It performs some tests for creating and verifying a test key in one of your Key Protect instances.

You cannot use the CLI or web console to review actual key content after it is uploaded. However, you can use the [REST API](https://cloud.ibm.com/apidocs/key-protectretrieve-a-key-by-id) directly to review existing custom key content by issuing the following command:

curl -X GET "https://us-south.kms.cloud.ibm.com/api/v2/keys/<key id>"
[key-encrypt](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-import-token-key-encrypt) Encrypt the key that you import into the service 
 [nonce-encrypt](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-import-token-nonce-encrypt) Encrypt the nonce that is generated by kp import-token create 
 [show](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-import-token-show) Retrieve an import token 







 kp instance command 

The kp instance command manages policies for a Key Protect instance.



Table 2. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [policies](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-instance-policies) List policies associated with an instance 
 policy-update [allowed-network](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-instance-policy-update-allowed) Update the instance policy for "allowed network" 
 policy-update [dual-auth-delete](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-instance-policy-update-dual) Update the instance policy for "dual auth delete"
[nonce-encrypt](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-import-token-nonce-encrypt) Encrypt the nonce that is generated by kp import-token create 
 [show](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-import-token-show) Retrieve an import token 







 kp instance command 

The kp instance command manages policies for a Key Protect instance.



Table 2. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [policies](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-instance-policies) List policies associated with an instance 
 policy-update [allowed-network](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-instance-policy-update-allowed) Update the instance policy for "allowed network" 
 policy-update [dual-auth-delete](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-instance-policy-update-dual) Update the instance policy for "dual auth delete" 
 policy-update [rotation](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-key-policy-update-rotation) Update the instance policy for "rotation" 







 kp key command 

The kp key command manages individual keys.



Table 3. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [alias-create](https://cloud.ibm.com/docs/key-protect?topic=key-protect-key-protect-cli-referencekp-key-alias-create) Create an alias for a unique and convenient reference to a key
[nonce-encrypt](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-import-token-nonce-encrypt) Encrypt the nonce that is generated by kp import-token create 
 [show](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-import-token-show) Retrieve an import token 







 kp instance command 

The kp instance command manages policies for a Key Protect instance.



Table 2. Sub-commands for managing keys

 Sub-command Status v0.7.0 Description 

 [policies](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-instance-policies) List policies associated with an instance 
 policy-update [allowed-network](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-instance-policy-update-allowed) Update the instance policy for "allowed network" 
 policy-update [dual-auth-delete](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-instance-policy-update-dual) Update the instance policy for "dual auth delete" 
 policy-update [rotation](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-key-policy-update-rotation) Update the instance policy for "rotation" 







 kp key command 

The kp key command manages individual keys.



Table 3. Sub-commands for managing keys

 Sub-command Status v0.7.0 Description 

 [alias-create](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-key-alias-create) Create an alias for a unique and convenient reference to a key
Retrieving all instances of all services in resource group Default and all locations
under account MyCompany as myuserid@mycompany.com...
OK
Name Location State Type
Key Protect-17 us-south active service_instance
Key Protect-60 us-south active service_instance
3. Retrieve the instance ID for the Key Protect service instance where your customer root keys are stored. Key Protect-17 is the name of the Key Protect service instance.

ibmcloud resource service-instance "Key Protect-17" --id

For this example, you'll see the following output:

Retrieving service instance Key Protect-17 in resource group Default under account
MyCompany as myuserid@mycompany.com...
crn:v1:bluemix:public:kms:us-south:a/abxx1c2xxxx34x567xxdex891xxx23fx:xxx4g5xx-6789-x1h2-
ixxx-3jkl4xxxx567::7mnxxxo8-91xx-23px-q4rs-xxtuv5w6xxx7

The instance ID is the string that follows the final :: in the CRN. In this example, it's 7mnxxxo8-91xx-23px-q4rs-xxtuv5w6xxx7.
4. List the available keys and their associated CRNs for the Key Protect service instance by specifying the instance ID:

ibmcloud kp keys -c --instance-id 7mnxxxo8-91xx-23px-q4rs-xxtuv5w6xxx7

For this example, you'd see the following output:

Retrieving keys...

SUCCESS

Key ID                                 Key Name               CRN
ef1gxxxh-ijxx-234x-56k7-xxxxlmnoxxp8   test-key               crn:v1:bluemix:public:
$ ibmcloud resource service-instance-create $KMS_NAME kms tiered-pricing us-south

 capture the Key Protect (KP) instance id (GUID)
$ KP_INSTANCE_ID=b0d84b32-09d0-4314-8049-da78e3b9ab6f

 create a policy for COS to read KMS; source is COS, target is KMS (Key Protect)
$ ibmcloud iam authorization-policy-create cloud-object-storage kms Reader --source-service-instance-id $COS_INSTANCE_ID --target-service-instance-id $KP_INSTANCE_ID

 capture the COS-to-KMS authorization policy id
$ COS_KMS_AUTHORIZATION=79375168-2d74-49df-98aa-3e727410fa71

 create a KP root key that COS uses to encrypt data
$ ibmcloud kp key create my-cos-key --output json

{
"id": "52a9d772-8982-4620-bfb4-b070dd812a0c",
"name": "my-cos-key",
"type": "application/vnd.ibm.kms.key+json",
"extractable": false,
"state": 1,
"crn": "crn:v1:bluemix:public:kms:us-south:a/ea998d3389c3473aa0987652b46fb146:b0d84b32-09d0-4314-8049-da78e3b9ab6f:key:52a9d772-8982-4620-bfb4-b070dd812a0c"
}

 capture the KP root key id
$ MY_COS_KEY_ID=52a9d772-8982-4620-bfb4-b070dd812a0c
Create a trusted profile policy for my-profile from a JSON file:

iam trusted-profile-policy-create my-profile --file policy.json

Give my-profile Viewer role for the members of resource group sample-resource-group:

iam trusted-profile-policy-create my-profile --roles Viewer --resource-group-id sample-resource-group

Give my-profile Viewer role for all resources in account:

iam trusted-profile-policy-create my-profile --roles Viewer







 ibmcloud iam trusted-profile-policy 

Display details of an access policy for a specified trusted profile

ibmcloud iam trusted-profile-policy (NAME|ID) POLICY_ID [--output FORMAT] [-f, --force] [-q, --quiet] [--api-version v1 | v2]



 Command options 

NAME|ID (required)
: Name or ID of the profile.

POLICY_ID (required)
: The ID of the policy to retrieve.

-f, --force
: Force failure if multiple profiles are found.

--output FORMAT
: Specify output format. Only 'JSON' is supported.

-q, --quiet
: Suppress verbose output.

--api-version
: Version of the access policy API.





 Examples 

Get policy bdf62c30-35dd-4852-bcb8-2f0dd3929701 of trusted profile my-profile:

ibmcloud iam trusted-profile-policy my-profile bdf62c30-35dd-4852-bcb8-2f0dd3929701







 ibmcloud iam trusted-profile-policies 

List all access policies for a specified trusted profile

ibmcloud iam trusted-profile-policies (NAME|ID) [--output FORMAT] [-f, --force] [-q, --quiet] [--api-version v1 | v2]



 Command options 

NAME|ID (required)
: Name or ID of the profile.

-f, --force
: Force failure if multiple profiles are found.
Update policy 85f3a4d6-c2e1-417e-b2d5-7199d610c160 to give trusted profile my-profile Administrator role for all account management services:

ibmcloud iam trusted-profile-policy-update my-profile 85f3a4d6-c2e1-417e-b2d5-7199d610c160 --roles Administrator --account-management

Update policy bdf62c30-35dd-4852-bcb8-2f0dd3929701 from my-profile with contents in JSON file:

ibmcloud iam trusted-profile-policy-update my-profile bdf62c30-35dd-4852-bcb8-2f0dd3929701 --file @policy.json







 ibmcloud iam trusted-profile-policy-delete 

Delete an access policy for a trusted profile

ibmcloud iam trusted-profile-policy-delete (NAME|ID) POLICY_ID [-f, --force] [-q, --quiet] [--api-version v1 | v2]



 Command options 

NAME|ID (required)
: Name or ID of the profile which contains the policy to delete.

POLICY_ID (required)
: The ID of the policy to delete.

-f, --force
: Delete access policy without confirmation.

-q, --quiet
: Suppress verbose output.

--api-version
: Version of the access policy API.





 Examples 

Delete policy ID bdf62c30-35dd-4852-bcb8-2f0dd3929701 from my-profile without confirmation:

ibmcloud iam trusted-profile-policy-delete my-profile bdf62c30-35dd-4852-bcb8-2f0dd3929701 -f







 ibmcloud iam trusted-profile-link-create 

Create a link to a compute resource for a trusted profile
ibmcloud plugin update key-protect -r "IBM Cloud"
2. To perform key management operations with the Key Protect CLI plug-in, set the KP_PRIVATE_ADDR environment variable to target the key management service private endpoint:

export KP_PRIVATE_ADDR=https://api.private.<region>.hs-crypto.cloud.ibm.com:<port>









 Using the VPE with the API 



* Using the VPE for the [key management service API](https://cloud.ibm.com/apidocs/hs-crypto)

To perform key management operations with the key management service API, use the key management service private endpoints in the API request URL to access the service. For example,

curl GET 
https://api.private.<region>.hs-crypto.cloud.ibm.com:<port>/api/v2/keys 
-H 'authorization: Bearer <IAM_token>' 
-H 'bluemix-instance: <instance_ID>' 
-H 'accept: application/vnd.ibm.kms.key+json'
* Using the VPE for the

[PKCS #11 API](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-pkcs11-api-ref)

To perform cryptographic operations with the PKCS #11 API, set the address field to the EP11 private endpoint in the configuration file. For more information, see

[Set up the PKCS #11 configuration file](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-set-up-pkcs-apistep3-setup-configuration-file).
* Using the VPE for the [GREP11 API](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-ref)

To perform cryptographic operations with the GREP11 API, specify the server address with the EP11 private endpoint in the code.
policy-update [allowed-network](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-instance-policy-update-allowed) Update the instance policy for "allowed network" 
 policy-update [dual-auth-delete](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-instance-policy-update-dual) Update the instance policy for "dual auth delete" 
 policy-update [rotation](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-key-policy-update-rotation) Update the instance policy for "rotation" 







 kp key command 

The kp key command manages individual keys.



Table 3. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [alias-create](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-key-alias-create) Create an alias for a unique and convenient reference to a key 
 [cancel-delete](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-key-cancel-delete) Cancel a previously scheduled request to delete a key 
 [create](https://cloud.ibm.com/docs/key-protect-cli-plugin?topic=key-protect-cli-plugin-key-protect-cli-referencekp-key-create) Create a key or import your own key
To import an existing key, provide a 32-byte (256-bit) key.

To generate and wrap a new DEK, omit the --plaintext parameter.
* --key-ring

A unique, human readable name for the key-ring. Required if the user doesn't have permissions on the default key ring.









 kp keys 

List the keys that are available in your Key Protect instance.

Keys are listed in key id order; see [example 5](https://cloud.ibm.com/docs/cli?topic=cli-key-protect-cli-referencekp-keys-example-5).

ibmcloud kp keys
-i, --instance-id INSTANCE_ID
[-b --sort-by SORT_BY]
[-c, --crn]
[-f, --key-filter FILTERS]
[--key-ring KEY_RING_ID]
[--key-states STATES]
[-l --key-search KEY_SEARCH]
[-n, --number-of-keys NUMBER_OF_KEYS]
[-o, --output OUTPUT]
[-s, --starting-offset STARTING_OFFSET]
[-t, --key-type TYPE]



 Examples 

These are examples of kp keys.



 Example 1 

List keys. Note that this will only list the first 200 keys in your instance by default. To show a greater or fewer number of keys, provide a value for the --number-of-keys parameter.

 list keys
$ ibmcloud kp keys

Retrieving keys...
OK
Key ID Key Name
5f2cc155-fe16-492c-845c-4d1f0688c7ba my-root-key
c36e9f3a-feaf-4033-8603-687784dc7e51 my-root-key





 Example 2 

List keys and show the cloud resource name (CRN) using the --crn parameter.
"href": "$vpc_api_endpoint/v1/subnets/2302-ea5fe79f-52c3-4f05-86ae-9540a10489f5/reserved_ips/0716-6fd4925d-7774-4e87-829e-7e5765d454ad",
"id": "0716-6fd4925d-7774-4e87-829e-7e5765d454ad",
"name": "my-reserved-ip",
"resource_type": "subnet_reserved_ip"
},
"resource_type": "share_mount_target",
"security_groups": [
{
"crn": "crn:...]",
"href": "https://us-south.iaas.cloud.ibm.com/v1/security_groups/r134-1dfeccef-3ad6-4760-8653-a202bc795db4",
"id": "r134-1dfeccef-3ad6-4760-8653-a202bc795db4",
"name": "my-security-group",
"resource_type": "security_group"
}
],
"subnet": {
"crn": "crn:[...]",
"href": "https://us-south.iaas.cloud.ibm.com/v1/subnets/2302-ea5fe79f-52c3-4f05-86ae-9540a10489f5",
"id": "2302-ea5fe79f-52c3-4f05-86ae-9540a10489f5",
"name": "my-subnet",
"resource_type": "subnet"
},
[show](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-import-token-show) Retrieve an import token 







 kp instance command 

The kp instance command manages policies for a Key Protect instance.



Table 2. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [policies](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-instance-policies) List policies associated with an instance 
 policy-update [allowed-network](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-instance-policy-update-allowed) Update the instance policy for "allowed network" 
 policy-update [dual-auth-delete](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-instance-policy-update-dual) Update the instance policy for "dual auth delete" 
 policy-update [rotation](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-key-policy-update-rotation) Update the instance policy for "rotation" 







 kp key command 

The kp key command manages individual keys.



Table 3. Sub-commands for managing keys

 Sub-command Status v0.8 Description 

 [alias-create](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-key-alias-create) Create an alias for a unique and convenient reference to a key 
 [cancel-delete](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-key-cancel-delete) Cancel a previously scheduled request to delete a key 
 [create](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-key-create) Create a key or import your own key 
 [delete](https://cloud.ibm.com/docs/key-protect-cli-pluginkp-key-delete) Delete a key
To learn how the Manager, Writer, and Reader access roles map to specific Key Protect service actions, see [Roles and permissions](https://cloud.ibm.com/docs/key-protect?topic=key-protect-manage-accessmanage-access-roles).
5. Create a [service ID API key](https://cloud.ibm.com/docs/account?topic=account-serviceidapikeys).

ibmcloud iam service-api-key-create API_KEY_NAME SERVICE_ID_NAME
[-d, --description DESCRIPTION]
[--file FILE_NAME]

Replace <service_ID_name> with the unique alias that you assigned to your service ID in the previous step. Save your API key by downloading it to a secure location.
6. Call the [IAM Identity Services API](https://cloud.ibm.com/apidocs/iam-identity-token-api) to retrieve your access token.

$ curl -X POST "https://iam.cloud.ibm.com/identity/token" -H "content-type: application/x-www-form-urlencoded" -H "accept: application/json" -d 'grant_type=urn%3Aibm%3Aparams%3Aoauth%3Agrant-type%3Aapikey&apikey=<API_KEY>' > token.json

In the request, replace <API_KEY> with the API key that you created in the previous step. The following truncated example shows the contents of the token.json file:

{
"access_token": "b3VyIGZhdGhlc...",
"expiration": 1512161390,
"expires_in": 3600,
"refresh_token": "dGhpcyBjb250a...",
"token_type": "Bearer"
}

Use the full access_token value, prefixed by the Bearer token type, to programmatically manage keys for your service using the Key Protect API.