vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Diagram of the components used by the assistant service](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/components.png)

The following sections provide more detail about each resource that is used by the system. The objective is to give you information that can help you to do initial resource planning and help you to manage changes in data needs over time.



 Microservices 

Watson Assistant consists of the following stateless microservices:



* Analytics: Logs conversations that take place between the assistant and your customers. These user conversation logs are available from the Analytics page in the product and from the /logs API endpoint. Introduced with the 1.5.0 release. The Analytics feature relies on a microservice that is supported only on Red Hat OpenShift 4.5 and later.
* CLU: A Conversational Language Understanding interface, also known as Natural Language Understanding (NLU), that is an entry point to the language understanding pipeline, which is where text is analyzed to find intent and entity mentions. The Store microservice uses the LiteLinks protocol to call this microservice to initiate machine learning training. The NLU microservice controls the lifecycle of the machine learning models. It maps the workspace ID that is used by the Store microservice to the machine language understanding pipeline's internal ID. The NLU microservice uploads the training data for machine learning models to the MinIO object storage. When a language model needs to be trained, such as after intents are edited, the NLU microservice calls the Master microservice. When the updated model is available, NLU calls the TAS microservice to analyze the user input that is provided by the customer. The workspace ID has a UUID format. The internal IDs of models in the language understanding pipeline typically follows the regular expression pattern, vn-[0-9a-f]-[0-9a-f].
COS_INSTANCE_CRN = "<service-instance-id>" eg "crn:v1:bluemix:public:cloud-object-storage:global:a/3bf0d9003xxxxxxxxxx1c3e97696b71c:d6f04d83-6c4f-4a62-a165-696756d63903::"

 Create client
cos = ibm_boto3.client("s3",
ibm_api_key_id=COS_API_KEY_ID,
ibm_service_instance_id=COS_INSTANCE_CRN,
config=Config(signature_version="oauth"),
endpoint_url=COS_ENDPOINT
)
Show more

Key Values



* <endpoint> - public endpoint for your cloud Object Storage with schema prefixed ('https://') (available from the [IBM Cloud Dashboard](https://cloud.ibm.com/resources)). For more information about endpoints, see [Endpoints and storage locations](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-endpointsendpoints).
* <api-key> - api key generated when creating the service credentials (write access is required for creation and deletion examples)
* <service-instance-id> - resource ID for your cloud Object Storage (available through [IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-idt-cli) or [IBM Cloud Dashboard](https://cloud.ibm.com/resources))
* <location> - default location for your cloud Object Storage (must match the region that is used for <endpoint>)



SDK References



* [ServiceResource](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlservice-resource)







 Creating a new bucket
WithCredentials(ibmiam.NewStaticCredentials(aws.NewConfig(), authEndpoint, apiKey, serviceInstanceID)).
WithS3ForcePathStyle(true)

// Create client connection
sess := session.Must(session.NewSession()) // Creating a new session
client := s3.New(sess, conf) // Creating a new client

// Create new bucket
_, err := client.CreateBucket(&s3.CreateBucketInput{
Bucket: aws.String(newBucket), // New Bucket Name
})
if err != nil {
exitErrorf("Unable to create bucket %q, %v", newBucket, err)
}

// Wait until bucket is created before finishing
fmt.Printf("Waiting for bucket %q to be created...n", newBucket)

err = client.WaitUntilBucketExists(&s3.HeadBucketInput{
Bucket: aws.String(newBucket),
})
if err != nil {
exitErrorf("Error occurred while waiting for bucket to be created, %v", newBucket)
}

fmt.Printf("Bucket %q successfully createdn", newBucket)

// Retrieve the list of available buckets
bklist, err := client.ListBuckets(nil)
if err != nil {
exitErrorf("Unable to list buckets, %v", err)
}

fmt.Println("Buckets:")

for _, b := range bklist.Buckets {
fmt.Printf("* %s created on %sn",
aws.StringValue(b.Name), aws.TimeValue(b.CreationDate))
}

// Uploading an object
input3 := s3.CreateMultipartUploadInput{
If not set, the context instance specified by ibmcloud cis instance-set INSTANCE is used.--output: Specify output format, only JSON is supported.<-- </section "id="section-get-analytics-range-app-options" "> --><-- <section "id="section-get-analytics-range-app-examples" "> --> Examples Get analytics data for range applications in domain 31984fea73a15b45779fa0df4ef62f9b. ibmcloud cis range-analytics 31984fea73a15b45779fa0df4ef62f9b --metrics "count,bytesIngress" --dimensions "event,appID" --since "2020-05-22T02:20:00Z"
--until "2020-05-23T02:20:00Z" -i "cis-demo"
<-- </section "id="section-get-analytics-range-app-examples" "> --><-- </section "id="section-get-analytics-range-app" "> --><-- </section "id="section-range-app" "> --><-- <section "id="section-ratelimit" "> --> Rate limiting Manipulate rate limits by using the following ratelimit commands.<-- <section "id="section-create-ratelimit" "> --> ibmcloud cis ratelimit-rule-create Create a new rate limiting rule for a given DNS domain (Enterprise plan only). ibmcloud cis ratelimit-rule-create DNS_DOMAIN_ID (--json @JSON_FILE | JSON_STRING) -i, --instance INSTANCE] --output FORMAT]
ibmcloud cis ratelimit-rule-create DNS_DOMAIN_ID --url URL --description DESCRIPTION] --threshold NUM] --period SECONDS] ...]
* 2 floating IP’s address that you use to access your VPC virtual server instance over the public network



During the second phase, the Ansible Playbook is called and the SAP High availability architecture is installed for both dedicated VSIs SAP App VSI machine and dedicated SAP HANA VSI box. The SAP architecture that is deployed is the SAP S/4HANA release on pacemaker cluster HA dedicated SAP HANA 2.0 VSI’s release as a distributed deployment model. For more information about this architecture, see [Automating SAP HANA stand-alone virtual server instance on IBM Cloud VPC by using Terraform and Ansible](https://cloud.ibm.com/docs/sap?topic=sap-automate-terraform-sap-hana-vsi).





 Highly available system for SAP HANA database 

Zoom

![Figure 1. SAP HA for SAP applications cluster nodes PAS (Active) and AAS (Active) Schematics deploymentS](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/refarch-sap-hana-cluster-pas-aas.svg)

Figure 1. SAP HA for SAP applications cluster nodes PAS (Active) and AAS (Active) Schematics deployment

Zoom

![Figure 2. SAP HA for HANA DB instances cluster nodes primary (Active) and Secondary (Passive)](https://cloud.ibm.com/docs-content/v1/content/f89cc64b006ca59b35404693959c83d5747fee3e/sap/images/refarch-sap-ha-hana-cluster-prim-sec.svg)

Figure 2. SAP HA for HANA DB instances cluster nodes primary (Active) and Secondary (Passive)

Figure 2. SAP HA for HANA DB instances cluster nodes primary (Active) and Secondary (Passive)

At the most basic level, a standard HA HANA cluster in an active-passive configuration has two nodes: one is the primary node and the other is the standby node.
Adding a bucket-catalog pair 

A bucket is an existing, externally managed object storage. It is one of the data sources for IBM® watsonx.data. A catalog defines the schemas and metadata for a data source.

When you add your own object storage bucket or database, or query the data in these data sources through the query engines of watsonx.data, egress charges for pulling data out of these sources might apply depending on your service provider. If you are using managed services, consult your service provider's documentation or support for details about these charges.

To reduce the latency issues, it is recommended to colocate your additional object storage buckets or databases in the region where watsonx.data instance is provisioned.

To add a bucket-catalog pair, complete the following steps.



1.  Log in to the watsonx.data console.
2.  From the navigation menu, select Infrastructure Manager.
3.  To define and connect a bucket, click Add component and select Add bucket.
4.  In the Add bucket window, provide the following details to connect to existing externally managed object storage.



Table 1. Register bucket

 Field         Description                                                                                  

 Bucket type   Select the bucket type from the list.                                                        
 Bucket name   Enter your existing object storage bucket name.                                              
 Display name  Enter the name to be displayed.                                                              
 Endpoint      Enter the endpoint URL.                                                                      
 Access key    Enter your access key.                                                                       
 Secret key    Enter your secret key.                                                                       
 Activation    Activate the bucket immediately or activate it later.                                        
 Catalog type  Select the catalog type from the list.                                                       
 Catalog name  Enter the name of your catalog. This catalog is automatically associated with your bucket.   



5.  Click Add.
For more information about profiles, see [Instance profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-profiles).

The API (or CLI) is useful for determining the interface type that is used for attaching the instance storage disks. The instance object that is returned from a create operation (or get) reports the instance disks in the disks array as shown in the following example partial output.

"disks": [
{
"created_at": "2022-04-19T17:13:28Z",
"href": "https://sample-region.iaasdev.cloud.ibm.com/v1/instances/vsi_aba03916-8804-46ae-9275-b7dc196e94b3/disks/c4e4d137-a44f-47b8-beb0-a22dc8f8c6ea",
"id": "c4e4d137-a44f-47b8-beb0-a22dc8f8c6ea",
"interface_type": "virtio_blk",
"name": "babbling-drove-grove-catnap",
"resource_type": "instance_disk",
"size": 600
}]

This example shows an interface type of virtio_blk.





 Using custom images with instance storage disk 

If you are using custom images, make sure that you load the correct device drivers in the image to use the disks.

If your instance storage device has an interface type of virtio_blk, then the libvirt virtio driver must be installed. The libvirt virtio driver is automatically installed with all IBM provided operating systems. All general-purpose profiles with instance storage offer only virtio_blk connections.





 Using instance storage from your instance 



 Listing the block devices on your instance 

On Linux®, you can use the lsblk command to list the block devices after you logged in to your instance.
By adopting the practices of the IBM Garage Method, the provider can implement an agile and iterative delivery process that supports modern DevOps practices like CI/CD.

Much of the CI/CD process itself is automated with IBM's Continuous Delivery service in the Cloud. The provider can define workflow toolchains to prepare container images, check for vulnerabilities, and deploy them to the Kubernetes cluster.







 Results 



* Lifting the existing monolithic VMs into cloud-hosted containers was a first step that allowed the provider to save on capital costs and begin learning modern DevOps practices.
* Reconfiguring key monolithic apps to a set of fine-grained microservices greatly reduced delivery time for patches, bug fixes, and new features.
* In parallel, the provider implemented simple time-boxed iterations to get a handle on the existing technical debt.









 Research nonprofit securely hosts sensitive data while it grows research with partners 

A Development Exec for a disease research nonprofit has academic and industry researchers who can't easily share research data. Instead, their work's isolated in pockets across the globe due to regional compliance regulations and centralized databases.

IBM Cloud Kubernetes Service delivers secure compute that can host sensitive and data processing on an open platform. That global platform is hosted in near-by regions. So it's tied to local regulations that inspire patients’ and researchers’ confidence that their data is both protected locally and makes a difference in better health outcomes.



 Context 

Securely hosting and sharing disease data for Research Nonprofit



* Disparate groups of researchers from various institutions don’t have a unified way to share data, slowing down collaboration.
* The security concern adds to the collaboration burden that causes even less shared research.
* Developers and Researchers are spread across the globe and across organizational boundaries, which make PaaS and SaaS the best option for each user group.
* Regional differences in health regulations require some data and data processing to remain within that region.







 Solution 

Securely hosting and sharing disease data for Research Nonprofit.

The research nonprofit wants to aggregate cancer research data across the globe. So they create a division that is dedicated to solutions for their researchers.



* INGEST - Apps to ingest research data.
<Priority>1</Priority>
<Filter/>
</Rule>
</ReplicationConfiguration>
Show more





 Delete the replication configuration for a bucket 

curl -X "DELETE" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" -H 'Authorization: bearer $TOKEN'

A successful request returns a 204 response.







 SDK examples 

The following examples make use of the IBM COS SDKs for Python and Node.js, although the implementation of object versioning should be fully compatible with any S3-compatible library or tool that allows for the setting of custom endpoints. Using third-party tools requires HMAC credentials in order to calculate AWS V4 signatures. For more information on HMAC credentials, [see the documentation](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-uhc-hmac-credentials-main).



 Python 

Enabling versioning using the IBM COS SDK for Python can be done using the [low-level client](https://ibm.github.io/ibm-cos-sdk-python/reference/services/s3.htmlclient) syntax.

Using a client:

!/usr/bin/env python3

import ibm_boto3
from ibm_botocore.config import Config
from ibm_botocore.exceptions import ClientError

 Define constants
API_KEY = os.environ.get('IBMCLOUD_API_KEY')
SERVICE_INSTANCE = os.environ.get('SERVICE_INSTANCE_ID')
ENDPOINT = os.environ.get('ENDPOINT')

BUCKET = "my-replication-bucket" The bucket that will enable replication.

 Create resource client with configuration info pulled from environment variables.
cosClient = ibm_boto3.client("s3",
ibm_api_key_id=API_KEY,
ibm_service_instance_id=SERVICE_INSTANCE,
* An SAP HANA system is identified by a single system ID (SID) and contains one or more tenant databases and one system database. Databases are identified by a SID and a database name. From the administration perspective, there is a distinction between tasks performed at system level and those performed at database level. Database clients, such as the SAP HANA cockpit, connect to specific databases.
* The SAP HANA XS advanced application server is a layer on top of SAP HANA that provides the platform for running SAP HANA-based Web applications. It is an integral part of the SAP HANA system.
* A system may consist of one host or a cluster of several hosts. This is referred to as a muliple-host, distributed system, or scale-out system and supports scalability and availability.
* The following sections provide overview information about these aspects of system architecture.



You can find a complete list of all SAP HANA components and the corresponding SAP HANA hardware and software requirements in the Product Availability Matrix (PAM), in the SAP HANA Hardware Directory, and in the SAP Community Network.





 Documentation of SAP HANA database 

SAP HANA hardware and software requirements are described in the SAP HANA Master Guide at:



* [SAP HANA Hardware and Software Requirements](https://help.sap.com/viewer/eb3777d5495d46c5b2fa773206bbfb46/2.0.01/en-US/d3d1cf20bb5710149b57fd794c827a4e.html)



SAP HANA support on SAP-certified Cloud IaaS:



* [Certified and Supported SAP HANA Hardware – Ceritified IaaS Platforms](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas;ve:28)



SAP HANA and SAP NetWeaver on UNIX/Linux:
* On Satellite clusters, the object store endpoint is now auto populated based on the cos.storageClass and s3provider options.







 Change log for version 2.2.8, released 23 January 2023 



* Updates the UBI Image to 8:8.7-1049
* Adds PriorityClasses for driver and plug-in pods
* Updates tolerations for plug-in pod
* Adds support for encrypting buckets with IBM Key Protect by using your root key CRN when creating buckets.
* Resolves the following CVEs: [CVE-2022-43680](https://nvd.nist.gov/vuln/detail/CVE-2022-43680), [CVE-2022-42010](https://nvd.nist.gov/vuln/detail/CVE-2022-42010), [CVE-2022-42011](https://nvd.nist.gov/vuln/detail/CVE-2022-42011), [CVE-2022-42012](https://nvd.nist.gov/vuln/detail/CVE-2022-42012), [CVE-2022-3821](https://nvd.nist.gov/vuln/detail/CVE-2022-3821), [CVE-2022-35737](https://nvd.nist.gov/vuln/detail/CVE-2022-35737), [CVE-2021-46848](https://nvd.nist.gov/vuln/detail/CVE-2021-46848).







 Change log for version 2.2.7, released 5 January 2023 



* Renames flex StorageClasses to Smart StorageClasses
* Updates GoLang to 1.18.9
},
}
client.CreateBucket(input)

d, _ := client.ListBuckets(&s3.ListBucketsInput{})
fmt.Println(d)
}

function createBucket(bucketName) {
console.log(Creating new bucket: ${bucketName});
return cos.createBucket({
Bucket: bucketName,
CreateBucketConfiguration: {
LocationConstraint: 'us-standard'
},
}).promise()
.then((() => {
console.log(Bucket: ${bucketName} created!);
}))
.catch((e) => {
console.error(ERROR: ${e.code} - ${e.message}n);
});
}

curl -X "PUT" "https://(endpoint)/(bucket-name)"
-H "Content-Type: text/plain; charset=utf-8"
-H "Authorization: Bearer (token)"
-H "ibm-service-instance-id: (resource-instance-id)"
-d "<CreateBucketConfiguration>
<LocationConstraint>(provisioning-code)</LocationConstraint>
</CreateBucketConfiguration>"

It isn't possible to change the storage class of a bucket once the bucket is created. If objects need to be reclassified, it's necessary to move the data to another bucket with the wanted storage class.
</Expiration>
</Rule>
</LifecycleConfiguration>

Example response The server responds with 200 OK.

We are completely uploaded and fine
HTTP/1.1 200 OK
Date: Fri, 28 Feb 2020 14:12:06 GMT
X-Clv-Request-Id: 587d909f-4939-41ef-8c16-80aea16a0587
Server: Cleversafe/3.14.9.53
X-Clv-S3-Version: 2.5
x-amz-request-id: 587d909f-4939-41ef-8c16-80aea16a0587
Content-Length: 0

Example request
{: token}


PUT /{bucket-name}?lifecycle HTTP/1.1
Authorization: Bearer {token}
Content-Type: text/plain
Host: s3.us.cloud-object-storage.appdomain.cloud
ibm-service-instance-id: {ibm-service-instance-id}
Content-Length: 123
Show more



Example request

PUT /{bucket-name}?lifecycle HTTP/1.1
Authorization: 'AWS4-HMAC-SHA256 Credential={access-key}/{date}/{region}/s3/aws4_request,SignedHeaders=host;x-amz-date;,Signature={signature}'
x-amz-date: {timestamp}
Content-Type: text/plain
Host: s3.us.cloud-object-storage.appdomain.cloud
Content-Length: 123

----
 Retrieve a bucket lifecycle configuration {: compatibility-api-get-config}

A GET operation uses the lifecycle query parameter to retrieve lifecycle settings for the bucket.

Syntax

 {: codeblock}
GET https://{endpoint}/{bucket-name}?lifecycle path style
GET https://{bucket-name}.{endpoint}?lifecycle virtual host style

Example request
Endpoints and storage locations 

Sending a REST API request or configuring a storage client requires setting a target endpoint or URL. Each storage location has its own set of URLs.

A bucket's resiliency is defined by the endpoint used to create it. Cross Region resiliency will spread your data across several metropolitan areas, while Regional resiliency will spread data across a single metropolitan area. Single Data Center resiliency spreads data across multiple appliances within a single data center. Regional and Cross Region buckets can maintain availability during a site outage.

Compute workloads co-located with a Regional Object Storage endpoint will see lower latency and better performance. For workloads requiring Cross Region resiliency, performance impacts are mitigated via geo endpoint routes connecting to the nearest Cross Region metropolitan area.

Some workloads may benefit from using a Single Data Center endpoint. Data stored in a single site is still distributed across many physical storage appliances, but is contained within a single data center. This can improve performance for compute resources within the same site, but will not maintain availability in the case of a site outage. Single Data Center buckets do not provide automated backup in the case of site destruction, so any applications using a single site should consider [using replication for disaster recovery](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-replication-overview) in their design.

All requests must use SSL when using IAM, and the service will reject any plaintext requests.

All IBM Cloud® Object Storage endpoints support TLS 1.2 encryption.



 Endpoint Types 

IBM Cloud® services are connected to a three-tiered network, segmenting public, private, and management traffic.



* Private endpoints are available for most requests originating from within IBM Cloud. Private endpoints provide better performance and do not incur charges for any outgoing or incoming bandwidth even if the traffic is cross regions or across data centers. Whenever possible, it is best to use a private endpoint.
* Public endpoints can accept requests from anywhere and charges are assessed on outgoing bandwidth. Incoming bandwidth is free.
Instead, users can refresh by using the browser refresh.





 06 November 2020 

Proxy protocol support for IBM Cloud Application Load Balancer for VPC (ALB)
: Proxy protocol provides a way to carry client information across proxies by appending it to the traffic your load balancer sends to the back-end servers. It also allows you to use listeners if incoming traffic contains proxy protocol information set by an intermediate proxy in between the client and your load balancer. See [Enabling proxy protocol](https://cloud.ibm.com/docs/vpc?topic=vpc-advanced-traffic-managementproxy-protocol-enablement) for details.





 02 November 2020 

Custom routing tables and routes (GA)
: IBM Cloud Custom Routes for VPC enables you to configure and control the routing behavior of traffic within your VPC. This functionality provides the ability to explicitly manage traffic between and among subnets within a VPC; thereby, supporting network topologies that incorporate the use of virtual appliances, such as firewalls. This service also allows IBM Cloud to continue to develop and deliver advanced features and capabilities for our IBM Cloud services. For more information, see [About routing tables and routes](https://cloud.ibm.com/docs/vpc?topic=vpc-about-custom-routes).

IP spoofing (GA)
: IP spoofing allows users to enable or disable the IP spoofing check on virtual server instances. This facilitates the traffic flow configured with custom routes functionality. For more information, see [About IP spoofing](https://cloud.ibm.com/docs/vpc?topic=vpc-ip-spoofing-about).







 October 2020 



 30 October 2020 

Virtual private endpoint gateways (GA)
: IBM Cloud Virtual Private Endpoints (VPE) for VPC is now generally available. This service enables you to connect to supported IBM Cloud services from your VPC network by using the IP addresses of your choosing, allocated from a subnet within your VPC.