vania-janet/multiturn-rag-retrieval-data huggingface dataset:
To bust bureaucracy and transform government’s relationship with its constituency, they turned to open standards to build a platform for co-creation.



* OPEN DATA – data storage where citizens, government agencies, and businesses access, share, and enhance data freely
* OPEN APIs – a development platform where APIs are contributed by and reused with all community partners
* OPEN INNOVATION – a set of cloud services that allow developers to use plug-in innovation instead of manually coding it



To start, the government uses IBM Cloud Object Storage to store its public data in the cloud. This storage is free to use and reuse, shareable by anyone, and subject only to attribution and share alike. Sensitive data can be sanitized before it’s pushed to the cloud. Besides that, access controls are set up so that the cloud caps new data storage, where the community can demonstrate POCs of enhanced existing free data.

The government’s next step for the public-private partnerships was to establish an API economy that is hosted in IBM® API Connect. There, community and enterprise Developers make data easily accessible in API form. Their goals are to have publicly available REST APIs, to enable interoperability, and to accelerate app integration. They use IBM Secure Gateway to connect back to private data sources on-premises.

Finally, apps based on those shared APIs are hosted in IBM Cloud Kubernetes Service, where it’s easy to spin up clusters. Then, Developers across the community, private sector, and the government can co-create apps easily. In short, Developers need to focus on coding instead of managing the infrastructure. Thus, they chose IBM Cloud Kubernetes Service because IBM simplifies infrastructure management.



* Managing Kubernetes master, IaaS, and operational components, such as Ingress and storage
* Monitoring health and recovery for worker nodes
* Providing global compute, so Developers don’t have to stand up infrastructure in worldwide regions where they need workloads and data to be located



Moving compute workloads into the IBM Cloud isn't enough though. The government needs to go through a method transformation as well.
* The architecture fully separates compute, metadata, and storage to provide ultimate flexibility.
* Fit for purpose query engines designed to handle modern data formats that are highly elastic and scalable.
* Data sharing between watsonx.data, Db2® Warehouse on Cloud, and Netezza or any other data management solution through common Iceberg table format support, connectors, and a shareable metadata store.



* How can I create a watsonx.data service instance?

To create a watsonx.data service instance, see [Provision an instance](https://cloud.ibm.com/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/watsonxdata?topic=watsonxdata-getting-started).
* How can I access watsonx.data web console?

To access the watsonx.data web console, see [Open the web console](https://cloud.ibm.com/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/watsonxdata?topic=watsonxdata-getting-started).
* Which data formats for ingestion job are supported in watsonx.data?

Data ingestion in watsonx.data supports .CSV and .Parquet formats.
* How can I delete the watsonx.data instance?

To delete the watsonx.data instance, see [Deleting watsonx.data instance](https://cloud.ibm.com/watsonxdata?topic=watsonxdata-delete_lh).
* How can I manage IAM access for watsonx.data?

IBM Cloud® Identity and Access Management (IAM) controls access to watsonx.data service instances for users in your account. Every user that accesses the watsonx.data service in your account must be assigned an access policy with an IAM role.
Overview of Cloud Pak for Data data sources 

In Discovery for Cloud Pak for Data, you can crawl documents from a local source that you upload or from a remote data source that you connect to. Learn more about the supported data sources and how to configure them.

IBM Cloud Pak for Data

IBM Cloud Pak for Data only

This information applies only to installed deployments. For more information about IBM Cloud data sources, see [Overview of the IBM Cloud data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sources).

All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.

You can use Discovery for Cloud Pak for Data to crawl from the following data sources:



* [Box](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-box-cp4d)
* [Database](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-database-cp4d)
* [FileNet P8](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-filenet-cp4d)
* [LDAP directory](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-ldap-cp4d)
* [Local File System](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-lfs-cp4d)
* [Notes](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-notes-cp4d)
* [Salesforce](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-salesforce-cp4d)
* The cluster name must start with a lowercase alphabetic character.
* The cluster name must end with a lowercase alphabetic or numeric character.
* The maximum length of the cluster name is 30 characters.
* The cluster name must be unique within the vCenter Server instance.







 Licensing settings 

Review the following information and specify the licensing setting for the Cyber Recovery component in the cluster.



* For Business Partner users, the vSphere license (Enterprise Plus edition) is included and purchased on your behalf.
* For users who are not Business Partners, use the IBM-provided VMware licenses for this component, which are included with purchase.



Bring Your Own License (BYOL) is no longer supported except for migrations or upgrades of existing BYOL clusters. Select I will provide or Use existing license and enter your own license key only if you are performing an upgrade or migration of an existing BYOL cluster.



* The Use existing license option is available only if you are using a BYOL vSphere license for your instance. When the option is enabled, you can select the existing license only if the instance has enough capacity for the additional hosts.







 Bare metal server settings 

Options might differ depending on the version that your instance was initially deployed in. You can choose between the Cascade Lake or SAP-certified Cascade Lake server types.



 Data center location 

The IBM Cloud data center location of the cluster is set to the IBM Cloud data center of the Cyber Recovery instance by default. You can deploy the cluster to a different IBM Cloud data center than the deployed instance if you ensure that the network latency between the two IBM Cloud data centers is less than 150 ms.

If you deploy the cluster to a different IBM Cloud data center or IBM Cloud infrastructure pod, three extra VLANs are ordered for use with the ordered IBM Cloud bare metal servers.





 Cascade Lake 

For Cascade Lake servers, you can choose the following CPU models and a supported RAM size, which depends on the NSX networking solution.

NSX-T instances



Table 1. Options for Cascade Lake bare metal servers - NSX-T instances

 CPU model Cores GHz RAM 

 Dual Intel Xeon Silver 4210 processor 20 2.2 128 GB, 192 GB, 384 GB, 768 GB, 1.5 TB
Learning about watsonx.data architecture and workload isolation 



  Architecture 

IBM® watsonx.data is a new open architecture that combines the elements of the data warehouse and data lake models. The best-in-class features and optimizations available on the watsonx.data, make it an optimal choice for next generation data analytics and automation.

It is a multi-tenant system that consists of two major building blocks:



1.  Control plane - One control-plane cluster is created per IBM Cloud region that runs multi-tenant components such as service web console UI and API.
2.  Data plane - One or more data-plane clusters are connected and managed by a control-plane running single-tenant customer-dedicated components within a customer-isolated instance such as a Presto engine and Hive metastore.



Important components of watsonx.data are:



1.  Web console – Web console is the UI interface for watsonx.data from where customers, can perform various functional tasks. It is a multi-tenant component, which resides in the control plane.
2.  Presto - Presto is a distributed SQL query engine, with the capability to query vast data sets located in different data sources.
3.  Hive Metastore - Hive metastore (HMS) is a service that stores metadata that is related to presto and other services in a backend relational database. When a new table is created, information that is related to the schema such as column names, data types, are stored in the metastore’s backend database.







  watsonx.data workload isolation 

As mentioned in the architecture section, some components of IBM® watsonx.data are multi-tenant and shared across customers in a given IBM Cloud region and some are single-tenant where they are explicitly dedicated to customers. To separate the access to infrastructure resources and data, several levels of authentication and authorization checks are in place. IAM authentication and access policies checks are performed on a service level. Role-based access control checks are performed on a resource level to allow only authorized users to perform certain operations and access to data.
BI.S4.H4.1.5TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2516)<br><br><br> 
 BI.S4.H4.3TB RAM + 3TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2430)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2432)<br><br><br> 
 BI.S4.H4.3TB RAM + 6TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2510)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2519)<br><br><br>
BI.S3.H2.384 (VMware) IBM Cloud for VMware 36 72 384 71,487 Intel Skylake SP 5.33 TDI OLAP/OLTP and SAP Business One for HANA Classic 
 BI.S3.H2.768 (VMware) IBM Cloud for VMware 36 72 768 71,667 Intel Skylake SP 10.67 TDI OLAP/OLTP and SAP Business One for HANA Classic 
 BI.S4.H2.192 (VMware) IBM Cloud for VMware 32 64 192 74,223 Intel Cascade Lake SP 3.00 TDI OLAP/OLTP and SAP Business One for HANA Classic 
 BI.S4.H2.384 (VMware) IBM Cloud for VMware 32 64 384 76,617 Intel Cascade Lake SP 6.00 TDI OLAP/OLTP and SAP Business One for HANA Classic 
 BI.S4.H2.768 (VMware) IBM Cloud for VMware 40 80 768 101,547 Intel Cascade Lake SP 9.60 TDI OLAP/OLTP and SAP Business One for HANA Classic 
 BI.S4.H2.1500 (VMware) IBM Cloud for VMware 56 112 1,536 132,498 Intel Cascade Lake SP 13.71 TDI OLAP/OLTP Classic 
 BI.S4.H2.3000 (VMware) IBM Cloud for VMware 56 112 3,072 121,614 Intel Cascade Lake SP 27.43 TDI OLAP/OLTP Classic 
 BI.S4.H4.3000 (VMware) IBM Cloud for VMware 112 224 3,072 257,373 Intel Cascade Lake SP 13.71 TDI OLAP/OLTP Classic 
 BI.S4.H4.6000 (VMware) IBM Cloud for VMware 112 224 6,144 257,373 Intel Cascade Lake SP 27.43 TDI OLAP/OLTP Classic 
 BI.S4.H8.6000 (VMware) IBM Cloud for VMware 224 448 6,144 495,603 Intel Cascade Lake SP 13.71 TDI OLAP/OLTP Classic
* [Networking](https://cloud.ibm.com/docs/containers?topic=containers-limitationsclassic_networking_limit)
* [Storage](https://cloud.ibm.com/docs/containers?topic=containers-limitationsclassic_storage_limit)



* [VPC cluster limitations](https://cloud.ibm.com/docs/containers?topic=containers-limitationsks_vpc_gen2_limits)



* [Compute](https://cloud.ibm.com/docs/containers?topic=containers-limitationsvpc_gen2_compute_limit)
* [Networking](https://cloud.ibm.com/docs/containers?topic=containers-limitationsvpc_gen2_networking_limit)
* [Storage](https://cloud.ibm.com/docs/containers?topic=containers-limitationsvpc_gen2_storage_limit)
BI.S4.H2.1.5TB RAM + 1.5TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2426)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2429)<br><br><br> 
 BI.S4.H2.1.5TB RAM + 3TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLAP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2508)<br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2503)<br><br><br> 
 BI.S4.H4.768GB RAM + 3TB PMEM IBM Cloud Bare Metal (Local SSD + Intel Optane DC PMEM) <br><br> * [OLTP Record](https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN//solutions?filters=iaas&id=s:2515)<br><br><br>
For more information, see [Box](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sourcesconnectboxpublic).

New SharePoint 2016 On-Premises connector
: Crawl SharePoint 2016 On-Premises systems. For more information, see [SharePoint 2016 On-Premises](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cloud).

The Box connector does not run on Safari
: For more information, see [Box connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-box-cloud).

Metadata conversion
: If the metadata property is converted to an array in the index, the document cannot be deleted by using the Delete labeled data API method. For more information, see the [API reference](https://cloud.ibm.com/apidocs/discovery-datadeletedocument).





 30 October 2020 

New language support for Bosnian, Croatian, Hindi, and Serbian
: Basic language support now available for Bosnian, Croatian, Hindi, and Serbian. For more information, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

New beta Patterns enrichment
: The beta release of Patterns enrichment uses pattern induction to help you teach Discovery to recognize patterns in your data. Pattern induction generates extraction patterns from the examples you specify. After you specify a small number of examples, Discovery will suggest additional rules that you verify to complete the pattern. You can use pattern induction as an enrichment or to create a facet. For more information, see [Patterns](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern) and [Creating a facet by identifying a pattern](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facetsfacetpattern).
: You can now configure the data source crawl to store ACL information as metadata in the documents that are added to your SharePoint Online collection. For more information, see [Microsoft SharePoint Online](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-online-cloud).

You can add more documents to the training data of the beta entity extractor model
: If you added and labeled 20 documents to train a model, and now want to continue to improve the model's performance, you can add more documents. Add the additional documents to the collection that you are using to train the model. After you label the first 20 documents, and the model is up to date with any changes, you can choose to continue labeling documents. The new documents that you added to the collection are loaded. You can label them to augment the training data, and then retrain your model. For more information, see [Customizing the terms that Discovery can recognize](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor).

Log out of Discovery
: You can log out of the Discovery service instance at any time by clicking Log out from the user profile menu that is available from the page header of the product user interface.





 18 November 2021 

Enterprise plan is now available everywhere
: The Enterprise plan is available from all data center locations. Scale and secure your Discovery application with enterprise-grade support and performance, and address more use cases including contract analysis and content mining to explore insights across documents. For more information, see [Discovery pricing plans](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-pricing-plans).





 11 November 2021 

New locations for Enterprise plan now available
: The Enterprise plan is available from the Frankfurt, London, Sydney, and Tokyo locations in addition to the Dallas location.





 3 November 2021 

New Enterprise plan
: Scale and secure your Discovery application with enterprise-grade support and performance and address more use cases, including contract analysis and content mining to explore insights across documents. Currently, the Enterprise plan is available only from the Dallas location.
See also [training data](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx2860199), [blind data](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx7881128).





 third-party 

Pertaining to a product or service that is provided by a company other than IBM.





 tile 

A visual representation of a running application that provides status on a dashboard.





 train 

To set up a Watson instance with components that enable the system to function in a particular domain (for example: corpus content, training data that generates machine learning models, programmatic algorithms, annotators, or other ground truth components) and then making improvements and updates to these components based on accuracy analysis.





 training data 

A set of annotated documents that can be used to train machine learning models. See also [testing data](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx7736833), [blind data](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx7881128).





 transaction 

The mechanism that participants on the blockchain network use to interact with assets. A transaction either creates new chaincode or invokes an operation in an existing chaincode. See also [chaincode](https://cloud.ibm.com/docs/overview?topic=overview-glossaryx9829545).





 trigger 

A mechanism that initiates actions. Triggers can be explicitly fired by a user or fired on behalf of a user by an external event source.





 true negative 

An answer or annotation that is actually incorrect and is predicted to be incorrect.





 true positive 

An answer or annotation that is actually correct and is predicted to be correct.





 trusted profile 

A grouping of federated users, compute resources, or both, that are granted the same IAM access. When an identity applies a trusted profile, temporary security credentials are provided for the duration of a session. All identities that are allowed to apply a single profile inherit the same access.





 trusted root
<-- </section "id="section-validate-a-firewall-rule-expression-examples" "> --><-- </section "id="section-validate-a-firewall-rule-expression" "> --><-- </section "id="section-firewall-rules" "> --><-- <section "id="section-glb" "> --> Global load balancer Manipulate global load balancers by using the following glb commands.<-- <section "id="section-create-glb" "> --> ibmcloud cis glb-create Create a global load balancer under a given DNS domain. ibmcloud cis glb-create DNS_DOMAIN_ID (--json @JSON_FILE | JSON_STRING) -i, --instance INSTANCE] --output FORMAT]
Deprecated] ibmcloud cis glb-create DNS_DOMAIN_ID (-s, --json-str JSON_STR | -j, --json-file JSON_FILE) -i, --instance INSTANCE] --output FORMAT] ! !!!!]!!!!
<-- <section "id="section-create-glb-options" "> --> Command options DNS_DOMAIN_ID: The ID of DNS domain. Required.--json: The JSON file or JSON string used to describe a global load balancer. Required.
<-- <ul> -->

* The required fields in JSON data are name, fallback_pool, default_pools:

<-- <ul> -->

* name: the DNS hostname to associate with your Load Balancer.
* fallback_pool: the pool ID to use when all other pools are detected as unhealthy.
* default_pools: a list of pool IDs ordered by their failover priority.

<-- </ul> -->

* The optional fields are description, ttl, region_pools, proxied, enabled, session_affinity, session_affinity_ttl, steering_policy:
FAQs for billing and usage 

FAQs for billing and usage might include questions about credit cards, promo codes, or other billing-related issues. To find all FAQs for IBM Cloud®, see our [FAQ library](https://cloud.ibm.com/docs/faqs).



 What is a GB-hour? 

Runtime and container usage is charged based on the following variables:



* How much memory your runtime or container uses, in gigabytes (GB).
* The duration that memory is used.



Multiply the two values together, and the result is the GB-hour.





 How are bandwidth overage charges calculated for Bare Metal Servers and Virtual Servers? 

IBM Cloud systems monitor all inbound and outbound traffic for a server regardless of the type of traffic. Based on the allocated bandwidth for a server, overages are assessed for excess traffic, which is monitored at the network switch level. Monitor your bandwidth usage with bandwidth graphs. For information, see [Viewing bandwidth graphs](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-bm-view-bandwidth-graphs).





 How do I prevent bandwidth overages? 

If you provision multiple servers, you can potentially reduce future bandwidth overage charges by pooling your servers' bandwidth. For more information, see [Optimizing your bandwidth usage](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-bm-view-bandwidth-graphsbm-optimize-bandwidth-usage). Contact an [IBM Cloud Sales](https://cloud.ibm.com/catalog?contactmodule) representative to request a quote for additional server bandwidth.





 What's the difference between a Pay-As-You-Go and Subscription account? 

With Pay-As-You-Go accounts, you're billed monthly for your resource usage. Your resource usage consists of recurring and fluctuating costs. This account type is a good fit for developers or companies that want to explore the entire IBM Cloud catalog but have smaller or variable workloads. You pay only for what you use or commit to on a monthly basis, with no long-term contracts.