vania-janet/multiturn-rag-retrieval-data huggingface dataset:
If possible, avoid waiting for the acknowledgment of a message before you publish the next message. Waiting prevents the producer from being able to batch together messages and also reduces the rate that messages can be published to below the round-trip latency of the network.





 Batching, throttling, and compression 

For efficiency purposes, the producer collects batches of records together for sending to the servers. If you enable compression, the producer compresses each batch, which can improve performance by requiring less data to be transferred over the network.

If you try to publish messages faster than they can be sent to a server, the producer automatically buffers them up into batched requests. The producer maintains a buffer of unsent records for each partition. There comes a point when even batching does not allow the rate that you want to be achieved.

There is another factor that has an impact. To prevent individual producers or consumers from swamping the cluster, Event Streams applies throughput quotas. The rate that each producer is sending data at is calculated and any producer that attempts to exceed its quota is throttled. The throttling is applied by slightly delaying the sending of responses to the producer. Usually, this just acts as a natural brake.

For more information about throughput guidance, see [Limits and quotas](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-kafka_quotaskafka_quotas).

In summary, when a message is published, its record is first written into a buffer in the producer. In the background, the producer batches up and sends the records to the server. The server then responds to the producer, possibly applying a throttling delay if the producer is publishing too fast. If the buffer in the producer fills up, the producer's send call is delayed, but ultimately might fail with an exception.





 Delivery semantics 

Kafka offers the following multiple different message delivery semantics:



* At most once: Messages might get lost and do not get redelivered.
* At least once: Messages are never lost but there might be duplicates.
* Exactly once: Messages are never lost and there are no duplicates.



The delivery semantics are determined by the following settings:
The appliance deployment method uses pre-defined validated SAP-optimized hardware by SAP-certified hardware partners that are running a specific operating system. These hardware options are offered in various configuration sizes.

Partners (such as Cloud Service Providers) offer appliances with multiple layers of redundant hardware, software and network components, which do not interrupt SAP HANA operations and defend against system outage. These components include:



* Redundant power supplies and fans and uninterrupted power supply (UPS)
* Enterprise grade error-correcting memories
* Fully redundant network switches and routers
* Disk storage systems that use batteries to guarantee writing even in the presence of power failure
* Disk storage systems that use striping and mirroring for redundancy and recovery from disk failures



In collaboration with SAP, a Cloud Service Provider defines the correct sizing when designing SAP-certified IaaS for SAP HANA with the appliance deployment method:



* Ensures maximum performance with the hardware capable of meeting specified workloads; providing dedicated memory for SAP HANA after the resident memory of OS and other programs is accounted for and with swapping to disk disabled.
* To maximize performance and throughput, SAP recommends that you scale up as far as possible (acquire the configuration with the highest processor and memory specification for the application workload) before scaling out (for deployments with greater data volume requirements).
* You can copy a database to machines from different SAP HANA appliance vendors with different hardware configurations, if both the source and target machines are compliant with the SAP HANA appliance specifications.









 TDI deployment method for Expert sizing 



 Expert sizing type 

Expert sizing type refers to a sizing exercise where customer-specific data is analyzed and used to put more detail on the sizing result for the hardware requirements of an SAP application (such as network, CPU, Memory, Storage).

According to SAP, expert sizing typically includes "exploring some business processes in more detail, both on functional and technical level" (quotation source: [Sizing Types - Expert Sizing](https://www.sap.com/about/benchmark/sizing.html)).

Therefore, with expert sizing, there are no standardized tools used to conduct the sizing it will often require significant effort and SAP expertise.
Host service in multiple regions IBM is responsible for hosting this service in multiple regions. The Customer is responsible for designing and deploying their workload in a way that achieves the wanted availability and Disaster Recovery capabilities by using provided tools. For example, deploy in different zones of a region, use at least two load balancers that are located in different zones, and either use DNS records to point to the load balancers, or ensure that your application can handle a list of IP addresses that it can connect to. 
 Backup user configuration data IBM is not responsible for backup of user configuration data. The Customer is responsible for backup of configuration data, as needed. 
 Backup and recovery of workload IBM is not responsible for backup or recovery of user configuration data. The Customer is responsible for backup and recovery of their workloads, as needed.
There are six private subnets in two zones; one subnet for each of the web, app, and database tiers in each zone. For more information on subnets, see [About networking](https://cloud.ibm.com/docs/vpc?topic=vpc-about-networking-for-vpc).

Bastion Host

The bastion host provides access to private networked hosts. It’s a single point of entry to all virtual servers in the farm. A bastion host is a VSI (Virtual server Instance) that acts like a gateway to these servers. Administrative tasks on all the Virtual Server Instances are proxied through a bastion host. While a single bastion host in enough for day to day operations, you can optionally create a second bastion host in another zone for High Availability.

NACLs and SG

NACLs are permissions that are defined with inbound and outbound rules. Both inbound and outbound rules must be explicitly defined as NACLs are stateless. NACLs are applied at the subnet level and the rules apply to all the instances within the subnet. NACLs are the first level of protection at the subnet.

Security Groups are a bit more granular, and can be applied at the instance level, and are stateful. You can define a different set of rules to two different SGs and apply the SGs to two different instances running in the same subnet and with the same NACLs. Most of the rules and limits in this tutorial are applied through Security Groups. For more information on VPC security groups, see [About security groups](https://cloud.ibm.com/docs/vpc?topic=vpc-using-security-groups).

Load Balancers

Load balancers distribute your traffic across multiple VMs, and you can scale your Virtual Server Instances up or down with minimal or no disruption of your traffic. You can Auto scale your servers as well. In the current architecture there are three load balancers: a public facing Load Balancer for the web servers, and a private load balancer each for both the app and database tiers. Application load balancers are configured in active-active mode.
Database details and connection credentials 

To configure your local development environment and to connect applications and tools to your Db2 Warehouse on Cloud database, you need to know the database details and connection credentials.



  Database details 

There are important database details that are needed for connecting applications and tools to your database:



*  Host name - The host name of the server.
*  Port number - Used by the database manager for TCP/IP communication. (Note that your service is provisioned with one port number for connections that use Secure Socket Layer (SSL) and a different port for non-SSL connection.)

If needed, you can get the IP address of your server by using the ping command or the nslookup command, specifying your host name.
*  Database name - The Db2 database name, usually BLUDB.







  Connection credentials 

There are three types of credentials:



*  IBMid - If you use IBM Cloud, this is the ID you would use to log in to IBM Cloud. This is not what you use to connect applications or tools to your Db2 Warehouse on Cloud database.
*  Db2 database credentials - Your service comes provisioned with a database user ID and password that can be used to connect applications and tools to your database.
*  Administrator-created users - Some Db2 Warehouse on Cloud plans allow administrative users to create new users. These administrator-created user IDs and passwords can be used to log directly into the web console URL and to connect to the Db2 database from applications or tools.







  Where to find database details and connection credentials 

You can collect this information from the following places:



*  Your administrator - If you are not the owner or administrator of your Db2 instance, you can get your database details and connect credentials from your administrator.
*  Db2 web console - Database details and credentials are available in the web console.
*  If you are using IBM Cloud:



*  IBM Cloud Dashboard - When you view your service on your IBM Cloud dashboard, in the "Service credentials" area, you can view the database details as well as the database user ID and password.
*  VCAP_SERVICES - VCAP_SERVICES is an environment variable in IBM Cloud that contains database details and credentials in JSON format. When you view the service credentials for your service in your IBM Cloud dashboard, the contents of VCAP_SERVICES is what is displayed. When you bind other IBM Cloud services or apps to your service, those other services or apps can access database details and credentials programmatically by reading VCAP_SERVICES.
You cannot toggle a public network off if there are no other defined networks.





 Detecting problems by using the System Reference Code (SRC) 

A system reference code (SRC) is a set of eight alphanumeric characters that identifies the name of the system component that detects, the error codes, and the reference code that describes the error condition. When the Power Systems Virtual Server instance detects a problem, an SRC number is displayed along with a timestamp in the Server details page. You can use the SRC to resolve the issue yourself. If you are contacting support to resolve a problem, the SRC number might help the hardware service provider better understand the problem and to provide the solution.

For an IBM i VM, the SRC number can be progress code, operation code, or software code. For more information, see the [System Reference Code list](https://www.ibm.com/support/knowledgecenter/ssw_ibm_i_73/rzahb/rzahbsrclist.htm) in IBM i documentation. For AIX VM instances, the SRC numbers are progress codes that provide information about the stages involved in powering on and performing initial program load (IPL). AIX SRCs refresh once in 2 minutes. For more information, see [AIX IPL progress codes](https://www.ibm.com/support/knowledgecenter/POWER9_REF/p9eai/aixIPL_info.htm).





 Use cases 

The different scenarios that you may face while requesting for resizing a Power Systems Virtual Server instance's memory are listed below.



 You request for resizing both memory and CPU fails 

When you attempt to resize the memory as well as CPU of a deployed virtual server instance through a single request it may fail due to the following reasons:



* There is no free memory available on the host.
* There is no free memory available on the logical partition as the resources on it are running.
* The free memory available on the logical partition is less than that of the desired value indicated in the resizing request.
* You have made multiple attempts for resizing.
* Currently, there is no preference for memory or CPU on what should be resized first.
You can log in to any node only through the bastion host by using the following command:

ssh -J ubuntu@<IP_address_bastion_host> vpcuser@<IP-address-of-nodes>

Although all of the nodes of each cluster have passwordless SSH set up among them, due to security constraints, you can't directly log in to a node from one cluster to another cluster.
* How many compute and storage nodes can I deploy in my Spectrum Scale cluster through this offering?

Before you deploy a cluster, it is important to ensure that the VPC resource quota settings are appropriate for the size of the cluster that you would like to create (see [Quotas and service limits](https://cloud.ibm.com/docs/vpc?topic=vpc-quotas)).

See the following minimum and maximum number of nodes that are supported in a cluster:



* Compute nodes: For all storage clusters, a minimum of 3 and a maximum of 64 virtual server instance compute nodes are supported.
* Scratch and evaluation cluster storage nodes: For a scratch and evaluation storage clusters, a minimum of 3 and a maximum of 18 virtual server instance storage nodes are supported.
* Persistent cluster storage nodes: For a persistent storage cluster, a minimum of 3 and a maximum of 10 bare metal server storage nodes are supported.



For more information, see [Deployment values](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-deployment-values).
* What storage types are available through this offering?

The Spectrum Scale solution offers three different storage types: scratch, persistent, and evaluation. For more information, see [Storage types](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-storage-types).
* Why are there two different resource group parameters that can be specified in the IBM Cloud catalog tile?

The first resource group parameter entry in the Configure your workspace section in the IBM Cloud catalog applies to the resource group where the Schematics workspace is provisioned on your IBM Cloud account.
For more information about access control for toolchains, see [Managing access to toolchains in resource groups](https://cloud.ibm.com/docs/services/ContinuousDelivery?topic=ContinuousDelivery-toolchains-iam-security).

You can specify the scripts to run in many of the job types that are provided by the pipeline, giving you direct control over what is run by the job. These scripts run in a Docker image that contains a number of standard development tools, including tools that are required for interacting with the IBM Cloud runtimes. For more information about what the standard Docker image contains, see [Preinstalled resources](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-deliverypipeline_environmentdeliverypipeline_resources). If your job requires development tools that are not available in the standard image, or you need different versions of those tools, you can use a custom image. For more information about custom images, see [Working with custom Docker images](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-custom_docker_imagescustom_docker_images).

When the pipeline runs scripts, properties that describe the context where the job is running are passed to the script by using environment variables. For example, the URL of the repo that is the input to the stage, the name of the stage and the job that is being run, the parameters specified by the job type, and so on. To view a list of the available environment variables, see [Preinstalled resources](https://cloud.ibm.com/docs/ContinuousDelivery?topic=ContinuousDelivery-deliverypipeline_environmentdeliverypipeline_resources).

You can define properties at both the pipeline level and the stage level. Pipeline properties are shared across all stages and jobs in a pipeline. Stage properties are unique to a particular stage, and shared across all jobs in that stage.
* If you create one more snapshot by using same disks, you will be charged for disk space that is used by M disks (600 GB) + (30% of 600 GB) + (30% of 600 GB) = 960 GB of disk space.
* The pricing that is charged for the snapshot is based on the storage tier that contains the source volumes: Tier 1 or Tier 3.







 Best practices 



* Before you take a snapshot, ensure that all of the data is flushed to the disk. If you take a snapshot on a running virtual machine (VM) and did not flush the file system, you might lose some content that is residing in memory.
* It is recommended that you quiesce all of the applications on the snapshot volume.







 Use cases 



* Taking a snapshot of a VM targeting only the boot volume
* Taking a snapshot of a VM targeting only data volumes
* Taking a snapshot of a VM targeting all volumes (both boot and data)
* Taking multiple snapshots of a single VM (the snapshots must be taken one at a time)
* Taking multiple snapshots of different VMs at the same time







 Restrictions and considerations 



* Parallel VM snapshot operations from different VM nodes for the same shared volume are not allowed.
* You cannot restore a VM if you are taking a snapshot and there are clone (full-copy)FlashCopy operations that are running in the background. The FlashCopy operations must first complete.
* Some of the attributes of source disks cannot be changed while the disks are in a snapshot relationship. For example, you cannot resize the source disks when there are snapshot relationships in place for those disks.
* Volumes that are in a snapshot relationship cannot be detached from the VM.









 Cloning a volume 

The clone operation creates a full copy of the volume. You can select multiple volumes and initiate a group clone operation. When multiple volumes are selected, the clone operation ensures that a consistent data copy is created.

The clone operation continues to copy data from the source disks to target disks in the background. Depending on the size of the source disks and the amount of data to be copied, the clone operation can take a significant amount of time.
After you reviewed the [overview](https://cloud.ibm.com/docs/vpc?topic=vpc-responsibilities-vpcoverview-by-resource), see what tasks you and IBM share responsibility for when you use IBM Cloud VPC.



 Incident and operations management 

Incident and operations management includes tasks such as monitoring, event management, high availability, problem determination, recovery, and full state backup and recovery.



Table 2. Responsibilities for incidents and operations

 Task IBM Responsibilities Your Responsibilities 

 Infrastructure IBM deploys a fully managed, highly available, secured, IBM-owned infrastructure. The Customer uses the provided API, CLI, or UI console to provision compute and storage, and to adjust networking configurations to meet the needs of their workload. To automate the provisioning and management of VPC service instances, you can use automation tools, such as IBM Cloud Schematics or Terraform. 
 Availability IBM fulfills requests for VPC infrastructure, such as VPCs, subnets, virtual server instances, block storage volumes, security groups, access control lists, floating IP addresses, public gateways, and SSH keys across multiple availability zones (AZs) and multi-zone regions (MZRs). The Customer designs and deploys their workload in a way that achieves high availability by using our provided tools, such as multiple availability zones. At a high level, you are responsible for deploying workloads in different zones of the region, by using at least two load balancers that are located in different zones, and either by using DNS records to point to the load balancers or ensuring that your application can handle the list of IP addresses that it can connect to. For more information, see [Deploy isolated workloads across multiple locations and zones](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-vpc-multi-region). 
 Bring your own CIDR IBM provides the ability to bring your own CIDR block to a subnet. The Customer ensures the CIDR blocks that they specify for their VPC do not conflict with CIDR blocks that are used by any other network that they plan to connect to their VPC. 
 Monitoring IBM provides built-in virtual server instance monitoring and IBM Cloud Monitoring.
Release notes 

The following new features and changes to IBM Watson™ Knowledge Studio for IBM Cloud Pak® for Data are available.

For details about what's new, see the IBM Cloud Pak for Data documentation.



* [4.5.x](https://www.ibm.com/docs/SSQNUZ_4.5.x/fixlist/wks-fixlist.html)





 Version 1.2.0 (9 December 2020) 



* Security and stability fixes



* Several security fixes and stability enhancements have been introduced.









 Version 1.1.3 (31 August 2020) 



* Security and stability fixes



* Several security fixes and stability enhancements have been introduced.









 Version 1.1.2 (19 June 2020) 



* Enhanced pre-annotation workflow.



* Run multiple pre-annotators at once and configure the order of pre-annotators to resolve annotation conflicts between them. Existing pre-annotator results are preserved unless you remove them with the Wipe option.
* Human annotations are preserved when running pre-annotators, even when using the Wipe option.









 Version 1.1.1 (28 February 2020) 



* New advanced rules tokenizer Advanced rules has introduced a new tokenizer which is not compatible with the previous one. The following extractor categories are affected:



* Parts of Speech extractor names are renamed automatically
* Cardinal is renamed to Numeral
* Preposition is renamed to Adposition
* Machine Data Analytics Splitter extractors have been removed and will disappear from the canvas. There is no substitute for these extractors that is provided by the new tokenizer.
* Generic Splitters:



* Date Time Splitter
* Day First Splitter
* Month First Splitter
* Year First Splitter



* Syslog Splitter:



* Date Time Splitter





* Cross-sentence relations (Experimental): English entities and relations workspaces can now support annotating relations between entities within spans of six sentences.