vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Setting up the API 

IBM® Key Protect for IBM Cloud® provides a REST API that can be used with any programming language to store, retrieve, and generate encryption keys.



  Retrieving your IBM Cloud credentials 

To work with the API, you need to generate your service and authentication credentials.

To gather your credentials:



1.  [Generate an IBM Cloud IAM access token](https://cloud.ibm.com/docs/key-protect?topic=key-protect-retrieve-access-token).
2.  [Retrieve the instance ID that uniquely identifies your Key Protect instance](https://cloud.ibm.com/docs/key-protect?topic=key-protect-retrieve-instance-ID).







  Forming your API request 

When you make an API call to the service, structure your API request according to how you initially provisioned your instance of Key Protect.

To build your request, pair a [service endpoint](https://cloud.ibm.com/docs/key-protect?topic=key-protect-regionsservice-endpoints) with the appropriate authentication credentials. For example, if you created a Key Protect instance for the us-south region, use the following endpoint and API headers to browse keys in your service:

$ curl -X GET     "https://us-south.kms.cloud.ibm.com/api/v2/keys"     -H "accept: application/vnd.ibm.collection+json"     -H "authorization: Bearer <IAM_token>"     -H "bluemix-instance: <instance_ID>"

Replace <access_token> and <instance_ID> with your retrieved service and authentication credentials.

Want to track your API requests in case something goes wrong? When you include the -v flag as part of curl request, you get a correlation-id value in the response headers. You can use this value to correlate and track the request for debugging purposes.





  What's next 

You're all set to start managing your encryption keys in Key Protect. To find out more about programmatically managing your keys, [check out the Key Protect API reference doc](https://cloud.ibm.com/apidocs/key-protect).
When you end an active session, it logs out the user and requires credentials to be entered again. Complete the following steps to end a login session:



1. In the IBM Cloud console, click Manage > Access (IAM), and select Users.
2. Select a name from the list.
3. Click View sessions in the User details section.
4. Identify the row with the session details that you want to end.
5. Select the Actions icon ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/4d8c6eec891cff72b666dc24bd3211810c77fb42/icons/action-menu-icon.svg) > End session.
6. Confirm that you want to end the session.
You can bypass creating users through IBM Cloud entirely, and create users directly in Databases for EnterpriseDB with psql. This allows you to make use of PostgreSQL's native [role and user management](https://www.postgresql.org/docs/current/database-roles.html). Users and roles created in psql must have all of their privileges set manually, including privileges to the objects that they create.

Users that are created directly in Databases for EnterpriseDB do not appear in Service Credentials, but you can [add them](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-connection-stringsadding-users-to-_service-credentials_) if you choose.

Note that these users are not integrated with IAM controls, even if added to Service Credentials.





 The emp_admin user 

The emp_admin is an internal IBM Cloud® Databases for EnterpriseDB user that is used by the EDB Migration Portal to communicate directly with an EnterpriseDB database. You can connect to the the EDB Migration Portal using your [IAM](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-iam) login information. You should not drop this user.





 The aq_administrator_role user 

The aq_administrator_role user is a system-defined privilege that can allow a user to interact with queues. This user is provided by default from EnterpriseDB and is not managed by IBM Cloud® Databases for EnterpriseDB. You should not drop this user. For more information, see [CREATE QUEUE](https://www.enterprisedb.com/docs/epas/latest/epas_compat_sql/30_create_queue/) in the EnterpriseDB documentation.
Generating an IAM token for a compute resource 

You can request an Identity and Access Management (IAM) token for Virtual server for VPC that represents the trusted profile for your compute resources. The IAM access token is suitable for use as a bearer token in calls that require user or service credentials.

For Kubernetes, you don't need to take steps to generate an IAM token from a compute resource (CR) token; this is done automatically by the service for new clusters that run Kubernetes version 1.21 or later. Trusted profiles are not supported for earlier versions of Kubernetes. See [Authorizing pods in your cluster to IBM Cloud services with IAM trusted profiles](https://cloud.ibm.com/docs/containers?topic=containers-pod-iam-identity) for more information.

To generate IAM tokens for your compute resources, you must be at least an administrator on All Identity and Access enabled services within the account.



 Generate your tokens 



1. Set up a trusted profile where you establish trust with compute resources. Select Virtual server for VPC as a compute service type. To complete this step, follow the instructions of [Setting up trusted profiles](https://cloud.ibm.com/docs/account?topic=account-create-trusted-profilecreate-profile-compute).
2. Retrieve a compute resource token (CR-token), which is specific to the resource. For example, a virtual server instance.



* For Virtual server for VPC, see [Acquire a JSON web token](https://cloud.ibm.com/docs/vpc?topic=vpc-imd-configure-serviceimd-json-token).



3. To generate an IAM access token that represents the trusted profile, provide the CR-token for IAM. One CR-token might resolve to multiple trusted profiles, therefore you must pass either profile_id or profile_name as a parameter. If both parameters are specified, they must resolve to the same compute resource, otherwise the API call fails.

$ curl -X POST -H "Content-Type: application/x-www-form-urlencoded"
const params = {
id: svcId,
};

iamIdentityService.unlockServiceId(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err);
done(err);
});

For more information, see the [IAM Identity Services API](https://cloud.ibm.com/apidocs/iam-identity-token-apiunlock-service-id).







 Examples of how to use a service ID 

The following are examples of how a Service ID is used with the Object Storage and IBM Cloud Data Engine services.



* Object Storage - [Use the IBM Cloud CLI](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main).
* IBM Cloud Data Engine - [How to use the IBM Cloud Data Engine REST API](https://www.youtube.com/watch?v=jDZKF0CnUvU).
If you have a role, then you can do all the tasks that are associated with that role.





 Roles 

The following section applies only to legacy credentials. For more information about using roles with IAM credentials, see [IBM Cloudant roles](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudantibm-cloudant-roles-ai).

IBM Cloudant has a number of roles available. The roles can be assigned to user accounts or [API keys](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountcreating-api-keys).

The three core roles are defined in the following table:



Table 2. Core roles

 Role Description 

 _admin Change security settings, including adding roles. 
 _reader Read documents from the database. 
 _writer Create, update, and delete documents (except design documents) in the database. 



The _reader and _writer roles are exclusive. If a user has the _writer role, they can't read documents that they create unless they also have the _reader role.

You might want to assign more than one role. For example, a user might need to read from or write to documents within a database, but wouldn't need full administrative control over the database. To fulfill this requirement, the user's account is granted the _reader and _writer roles, but not the _admin role.

A number of "focused" roles are also available. These provide permissions for specific API endpoints. The focused role permissions are similar to the core role permissions, but apply only to the specific API endpoint.

The focused roles are defined in the following table:



Table 3. Focused roles

 Role Description API Endpoints 

 _design Allows create, read, modify, or delete access to design documents.
This means that all of the configurations surrounding collectors, goal-based profiles, collector-based scans, goal-based results, and credentials will be removed at the end of support on 1 July 2023.

Additionally, the following changes will be made:



* Any Identity and Access Management (IAM) access policies or context-based restrictions that are scoped to “All Account Management Services” will no longer apply to Security and Compliance Center. Users who have access to “All Identity and Access Enabled Services” will now have access to Security and Compliance Center.
* If users have access only to products added to a private catalog and not the full IBM Cloud catalog, then Security and Compliance Center must be added to their private catalog to enable them to create an instance.
* If users have a quota on the number of resources for their account that has already been met, instances of the service will not be able to be created.



During this update, you are responsible for the migration of your scan configurations to the new architecture. You can start this migration at any time.

As you begin your migration, please keep the following dates in mind:



* 8 June 2023: The location setting in Security and Compliance Center will be frozen until the upgrade is complete. The location that is specified in the service is the region in which your new instance will be created. Additional instances can be created starting in July.
* 1 July 2023: The collector-based architecture will be removed from Security and Compliance Center.
* 8 July 2023: A regional instance of Security and Compliance Center will be available in your account based on your set location and the global instance will be removed. Additionally, the changes to IAM and context-based restrictions take effect.



You might see two instances of Security and Compliance Center through the Billing page during the migration period although you can’t see two instances in your account. This is due to the way that the migration is being conducted. You are only billed through one. When the migration is complete, the global instance will be removed from your account.





 2 May 2023 

Schedule the frequency of your scans
: You can now schedule the frequency at which your scans are run in the updated architecture. Options include everyday, every 7 days, and every 30 days. You can also now pause your scans.
For example, assign user access to the service, create, delete or scale instances, and bind instances to applications. Service access roles enable users to access VMware as a Service. For more information about the exact actions that are mapped to each role, see [IAM roles and actions](https://cloud.ibm.com/docs/account?topic=account-iam-service-roles-actions).

Platform roles

Service roles



Table 1. IAM platform roles

 Platform role Description of actions 

 Viewer View VMware instances, clusters, hosts, storage, services, virtual data centers, and dashboards. 
 Operator View or update VMware instances, add and remove hosts, storage, and clusters, and install VMware services. 
 Editor In addition to all Operator permissions, create and delete VMware instances. 
 Administrator In addition to all Operator permissions, reset the admin credentials for VMware instances. 





 Assigning access to VMware as a Service in the console 

You can assign access in the console in one of the following ways:



* Access policies per user. You can manage access policies per user from the Manage > Access (IAM) > Users page in the console. For more information about the steps to assign IAM access, see [Managing access to resources](https://cloud.ibm.com/docs/account?topic=account-assign-access-resources&interface=ui).
* Access groups. Access groups are used to streamline access management by assigning access to a group once. Then, you can add or remove users as needed from the group to control their access. You manage access groups and their access from the Manage > Access (IAM) > Access groups page in the console. For more information, see [Assigning access to a group in the console](https://cloud.ibm.com/docs/account?topic=account-groups&interface=uiaccess_ag).
Editor View and update the enterprise name and domain, create accounts and account groups, view usage reports, and import accounts. crn:v1:bluemix:public:iam::::role:Editor 
 Administrator View and update the enterprise name and domain, create accounts and account groups, move accounts between account groups, import existing accounts, and view usage reports crn:v1:bluemix:public:iam::::role:Administrator 
 Usage report viewer View the enterprise, accounts, and account groups and view usage reports for all accounts in the enterprise. crn:v1:bluemix:public:enterprise::::serviceRole:UsageReportsViewer 







 Global catalog 

You can give users access to view private products in the catalog or change the visibility of private products for other users in the account.



Table 8. Roles and example actions for the Global Catalog service

 Roles Actions role_ID value 

 Viewer View private services crn:v1:bluemix:public:iam::::role:Viewer 
 Operator Not applicable crn:v1:bluemix:public:iam::::role:Operator 
 Editor Change object metadata but can't change visibility for private services crn:v1:bluemix:public:iam::::role:Editor 
 Administrator Change object metadata or visibility for private services, and restrict visibility of a public service crn:v1:bluemix:public:iam::::role:Administrator 







 IAM Access Groups 

You can give users access to view, create, edit, and delete access groups in the account by using the IAM Access Groups service.



Table 9. Roles and example actions for the IAM access groups service

 Roles Actions role_ID value 

 Viewer View access groups and members crn:v1:bluemix:public:iam::::role:Viewer 
 Operator Not applicable 
 Editor View, create, edit, and delete groups<br><br>Add or remove users from groups crn:v1:bluemix:public:iam::::role:Editor
ibmcloud login

Parameter value:



* DOMAIN_NAME: The API endpoint for your region. For example, cloud.ibm.com



3. Get the list of the resource groups for your account and select one of the returned resource groups as the target resource group in which to create the IBM Analytics Engine serverless instance:

ibmcloud resource groups
ibmcloud target -g <RESOURCE_GROUP_NAME>

Parameter value:



* RESOURCE_GROUP_NAME: Provide the same name as you specified while provisioning watsonx.data for efficient organizing.



4. Create a service instance:

ibmcloud resource service-instance-create <SERVICE_INSTANCE_NAME> ibmanalyticsengine <PLAN_NAME> <REGION> -p @<PATH_to JSON file with cluster parameters>

Parameter value:



* SERVICE_INSTANCE_NAME: Specify a name for the instance.
* PLAN_NAME: Specify the plan name as plan_name8afde05e-5fd8-4359-a597-946d8432dd45.
* REGION: Specify the region where you like to provision the instance.



Note that currently, standard-serverless-spark is the only supported serverless plan and us-south and eu-de the only supported regions. Choose one that is closer to the region where you have provisioned watsonx.data.



* PATH_to JSON file: Include the path to the JSON file that contains the provisioning parameters.



For example, for the Dallas region:

ibmcloud resource service-instance-create MyServiceInstance ibmanalyticsengine standard-serverless-spark us-south -p @provision.json

You can give the service instance any name you choose. Note that currently, standard-serverless-spark is the only supported serverless plan and us-south and eu-de the only supported regions.

The provision.json file contains the provisioning parameters for the instance that you want to create.

The endpoint to your IBM Cloud® Object Storage instance in the payload JSON file must be the direct endpoint.
2. Select the resource group. Get the list of the resource groups for your account and select one in which to create the IBM Analytics Engine serverless instance:

Action
: Enter:

ibmcloud target -g RESOURCE_GROUP_NAME

Parameter values:



* RESOURCE_GROUP_NAME: The name of resource group in which the serverless instance is to reside



Example
: Enter:

ibmcloud resource groups
ibmcloud target -g default
3. Install the IBM Cloud Object Storage service and then the Analytics Engine V3 CLI:

Action
: Enter:

ibmcloud plugin install cloud-object-storage

Action
: Enter:

ibmcloud plugin install analytics-engine-v3
4. Create a Cloud Object Storage instance:

Action
: Enter:

ibmcloud resource service-instance-create INSTANCE_NAME cloud-object-storage PLAN global

Parameter values:



* INSTANCE_NAME: Any name of your choice
* PLAN: The Cloud Object Storage plan to use when creating the instance



Example
: Enter:

ibmcloud resource service-instance-create test-cos-object cloud-object-storage standard global

Response
: The example returns:

Service instance test-cos-object was created.
Name: test-cos-object
ID: crn:v1:bluemix:public:cloud-object-storage:global:a/867d444f64594fd68c7ebf4baf8f6c90:ebad3176-8a1a-41f2-a803-217621bf6309::
GUID: ebad3176-8a1a-41f2-a803-217621bf6309
Location: global
State: active
Type: service_instance
Sub Type:
Allow Cleanup: false
Locked: false
Created at: 2021-12-27T07:57:56Z
Updated at: 2021-12-27T07:57:58Z
Last Operation:
Status create succeeded
Message Completed create instance operation
"instance_home": {
"region": "us-south",
"endpoint": "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"hmac_access_key": "your-access-key",
"hmac_secret_key": "your-secret-key"
}
}
}
Show more
5. [Track instance readiness](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-provisioning-serverlesslh-instance-readiness).



For more information on the Resource controller REST API for creating an instance, see [Create (provision) a new resource instance](https://cloud.ibm.com/apidocs/resource-controller/resource-controllercreate-resource-instance).





 Tracking instance readiness 

To run applications on a newly created serverless instance, the instance must be in active state.

To track instance readiness:



1. Enter the following command:

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/{instance_id} -H "Authorization: Bearer $token"

Sample response:

{
"id": "dc0e-eab2-4t9e-9441-56620949",
"state": "created",
"state_change_time": "2021-04-21T04:24:01Z",
"default_runtime": {
"spark_version": "3.3",
"instance_home": {
"provider": "ibm-cos",
"type": "objectstore",
"region": "us-south",
"endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"name": "your-service-instance-name",
"resource_plan_id": "8afde05e-5fd8-4359-a597-946d8432dd45",
"resource_group": "resource-group-id",
"target": "us-south",
"parameters": {
"default_runtime": {
"spark_version": "3.1"
},
"instance_home": {
"region": "us-south",
"endpoint": "s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"hmac_access_key": "your-access-key",
"hmac_secret_key": "your-secret-key"
}
}
}
Show more
5. [Track instance readiness](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-provisioning-serverlessinstance-readiness).



For more information on the Resource Controller REST API for creating an instance, see [Create (provision) a new resource instance](https://cloud.ibm.com/apidocs/resource-controller/resource-controllercreate-resource-instance).





 Tracking instance readiness 

To run applications on a newly created serverless instance, the instance must be in active state.

To track instance readiness:



1. Enter the following command:

curl -X GET https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/{instance_id} -H "Authorization: Bearer $token"

Sample response:

{
"id": "dc0e-eab2-4t9e-9441-56620949",
"state": "created",
"state_change_time": "2021-04-21T04:24:01Z",
"default_runtime": {
"spark_version": "3.1",
"resource_aliases_url": "/v2/resource_instances/da82xxxx-xxxx-xxxx-xxxx-d0faf90exxxx/resource_aliases",
"resource_bindings_url": "/v2/resource_instances/da82xxxx-xxxx-xxxx-xxxx-d0faf90exxxx/resource_bindings",
"resource_keys_url": "/v2/resource_instances/da82xxxx-xxxx-xxxx-xxxx-d0faf90exxxx/resource_keys",
"plan_history": {
"resource_plan_id": "8afxxxx-xxxx-xxxx-xxxx-946d843xxxx",
"start_date": "2021-08-05T11:05:51.545526066Z",
"requestor_id": "IBMid-661002042N"
}],
"migrated": false,
"controlled_by": "",
"locked": false
}]}
Show more

See [Retrieving IAM access tokens](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-iam-token-serverless) for how to obtain the authorization bearer access token.

Search for your instance name in the "resources" array in the response by checking the "name" parameter. The response includes the GUID and the provisioning state of your instance. For more details about the response of the API, see [Get a list of all resource instances](https://cloud.ibm.com/apidocs/resource-controller/resource-controllerlist-resource-instances).

Note that the returned state create succeeded indicates that the provision request was successfully accepted. However, to run applications, the instance needs to move to active state. Track the instance readiness before performing any operations on the instance.
Provisioning an Analytics Engine instance 

For watsonx.data, it is recommend to use IBM Analytics Engine Spark to achieve below use-cases:



1. Ingesting large volumes of data into watsonx.data tables. You can also cleanse and transform data before ingestion.
2. Table maintenance operation to enhance watsonx.data performance of the table
3. Complex analytics workload which are difficult to represent as queries.



You can create an IBM Analytics Engine instance:



* [Using the IBM Cloud® console](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-provisioning-serverlesslh-console-provisioning)
* [Using the IBM Cloud® command-line interface](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-provisioning-serverlesslh-cli-provisioning)
* [Using the Resource controller REST API](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-lh-provisioning-serverlesslh-rest-api-provisioning)



You must have access to either the IBM Cloud® us-south (Dallas) or the eu-de (Frankfurt) region. When you add a region for provisioning an Analytics Engine instance, choose one that is closer to the region where you have provisioned watsonx.data to avoid data latency issues.



 Creating a service instance from the IBM Cloud® console 

You can create an instance by using the IBM Cloud® console.

To create an IBM Analytics Engine instance:



1. Log in to the [IBM Cloud® console](https://cloud.ibm.com/catalog).
2. Click Services and select the category Analytics.
3. Search for Analytics Engine and then click the tile to open the service instance creation page.
4. Choose the location that is closer to the region where you have provisioned watsonx.data for deploying the service instance. Currently, us-south and eu-de are the only supported regions.
5. Select a plan.