vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Then, you are charged as stated in the offering details page, see [Container Registry](https://cloud.ibm.com/registry/catalog).<br><br>Pull traffic. You are charged by Gigabyte usage per month. The first 5 GB are free. Then, you are charged as stated in the offering details page, see [Container Registry](https://cloud.ibm.com/registry/catalog). If you exceed your storage or pull traffic limits, you can't push or pull images to and from your namespace. For more information about storage, pull traffic, and the cost estimator, see [Quota limits and billing](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewregistry_plan_billing). 







 Quota limits and billing 

Find information and examples for how the billing process and quota limits work in Container Registry.

Every image is built from a number of layers that each represent an incremental change from the base image. When you push or pull an image, the amount of storage and pull traffic that is needed for each layer is added to your monthly usage. Identical layers are automatically shared between images in your IBM Cloud account and are reused when you create other images. The storage for each identical layer is charged only once, regardless of how many images in your account reference the layer. Layers that are only referenced by deleted images in the trash are not charged.

From 1 February 2022, both [tagged](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) and [untagged](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_untagged) images are charged for.

Quota limits and billing are scoped to the specific registry instance (one of the regional registries or the global registry) that you're currently working with. Quota settings must be managed separately for your account in each registry instance. For more information, see [Regions](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewregistry_regions).
Before you get started with deploying apps in Code Engine, learn the basics about Code Engine. The following table describes some high-level terminology differences between Cloud Foundry and Code Engine.



Table 1. Terminology

 Heroku Code Engine Description 

 N/A Resource group and projects A grouping of workloads. The specific choice of which workload goes into each grouping is defined by the user. "Resource groups" are an IBM Cloud concept, while "projects" are Code Engine specific. Projects provide a level of isolation between workloads. See [Managing projects](https://cloud.ibm.com/docs/codeengine?topic=codeengine-manage-project). 
 Dynos Containers A dyno is a light-weight container that runs your Heroku code and is managed for you. In Code Engine, the underlying containers are based on Kubernetes and are automatically managed for you. 
 Application Application (app) A workload that responds to HTTP requests from a REST API, a web page request, or an event, for example. Code Engine requires that applications include the HTTP server as part of the code. Code Engine applications automatically scale (up and down) based on the incoming load. You can configure the minimum and maximum scale if needed. By default, the application listens on port 8080. You can override this behavior by using the console or with the CLI --port flag. See [Working with apps in Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-application-workloads). 
 N/A Job or batch job A job runs one or more instances of your executable code. Unlike applications, which handle HTTP requests, jobs are designed to run one time and exit. When you create a job, you can specify workload configuration information that is used each time that the job is run. See [Working with jobs and job runs](https://cloud.ibm.com/docs/codeengine?topic=codeengine-job-plan). 
 heroku create and git push heroku main or heroku container:push web app create The process of creating an application.
* [Iceberg](https://prestodb.io/docs/current/connector/iceberg.html)
* [Db2](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-db2_connector)
* [Netezza](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-netezza_connector)
* [Hive](https://prestodb.io/docs/current/connector/hive.html)
* [Hudi](https://prestodb.io/docs/current/connector/hudi.html) (For more information about using Hudi connector with watsonx.data, see [Integrating Presto with Apache Hudi using a Hudi connector](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-hudi-conn).)
* [JMX](https://prestodb.io/docs/current/connector/jmx.html)
* [Kafka](https://prestodb.io/docs/current/connector/kafka.html)
* [Memory](https://prestodb.io/docs/current/connector/memory.html)
* [MongoDB](https://prestodb.io/docs/current/connector/mongodb.html)
* [MySQL](https://prestodb.io/docs/current/connector/mysql.html)
* [PostgreSQL](https://prestodb.io/docs/current/connector/postgresql.html)
* [TPCDS](https://prestodb.io/docs/current/connector/tpcds.html)
* [TPCH](https://prestodb.io/docs/current/connector/tpch.html)







 Presto SQL Language
Choose enrichments 

Add resources that can teach Discovery about terms or patterns that have special meaning to your application.

The following table describes the best resources to add to address different needs.



Domain tools overview

 Goal Resource Notes 

 Define categories by which text in your documents can be classified. [Classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifier) N/A 
 Recognize terms and synonyms for terms that are significant to you, such as the names of products that you sell. [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionary) Term suggestions are displayed if the Part of Speech enrichment is applied to the collection. 
 Define regular expressions that capture patterns of significance, such as that AB10045 is the syntax that is used for your order numbers. [Regular expressions](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-regex) N/A 
 Recognize and tag entities and relationships that are defined in a custom machine learning model. [Machine learning models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-ml) Requires a model that is built and exported from another IBM tool. 
 Apply rules to fields that are based on rules you defined by creating an advanced rules model in IBM Watson® Knowledge Studio. [Advanced rules models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-mladvanced-rules) Requires an advanced rules model that is built and exported from IBM Watson® Knowledge Studio or that uses an exported Patterns resource. 
 <br>IBM Cloud<br><br> Recognize terms that are mentioned in sentences that match a syntactic pattern that you teach Discovery to recognize. [Patterns (beta)](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern) Available as a beta feature for English-language collections in managed deployments only. The enrichment that is derived by defining patterns cannot be applied to Content Mining projects.
This family is hosted exclusively on the latest generation Intel® Xeon® Platinum Cascade Lake server hosts and is optimized for running memory intensive applications and in-memory database such as SAP HANA, Memcached, or Redis. Ultra High Memory profiles are available in the US South (Dallas) and EU Germany (Frankfurt) regions. For more information, see [Ultra HIgh Memory with instance storage profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-dh-profilesuhm-is-dh-pr).





 12 November 2021 

Fedora Core OS
: Fedora Core OS is now available in all regions as a stock image. The user login ‘root’ is disabled by default in Fedora Core OS. The user login ‘core’ can be used to log in to Fedora Core OS instances. For more information, see [Virtual server images](https://cloud.ibm.com/docs/vpc?topic=vpc-about-imagesx86-supported-os) and [User data examples for Fedora Core OS](https://cloud.ibm.com/docs/vpc?topic=vpc-user-datauser-data-examples-for-fed-core).





 02 November 2021 

Instance Metadata Service for VPC (LA)
: The metadata service is now available in all regions to customer accounts authorized to access this service. This service is enabled by default when you create new instances by using the UI, CLI, or API. You can also disable the service from these interfaces. For more information, see [Enable or disable the metadata service](https://cloud.ibm.com/docs/vpc?topic=vpc-imd-configure-service&interface=uiimd-metadata-service-enable).
: The metadata service also provides a new API call to generate an IAM token from the instance identity access token by using trusted profile information.
Finnish [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2)
Streaming data 

Stream data from an IBM Log Analysis instance to other corporate tools such as Security Information and Event Management (SIEM) tools.

When you stream data to data lakes, other analysis tools, or other SIEM tools, you can add additional capabilities to the ones provided by the IBM Log Analysis service:



* You can gain visibility into enterprise data across on-premises and cloud-based environments.
* You can identify and prioritize security threats that might affect your organization.
* You can detect vulnerabilities by using Artificial Intelligence (AI) to investigate threats and incidents.



You can stream data to an Event Streams instance or to an IBM Log Analysis instance. For example, when you enable streaming on an IBM Log Analysis instance, you configure Log Analysis to send data to an Event Streams instance. Then, you can configure Kafka Connect to consume the data and forward it to your destination tool. Once the data is persisted within Event Streams, you can configure any application or service to create a subscription and take action on log data being streamed.

Zoom

![Streaming example with Event Streams](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/la_streams.svg)

Figure 1. Streaming example with Event Streams

You can also also configure streaming from one IBM Log Analysis instance to a second IBM Log Analysis instance.

Zoom

![Log Analysis to Log Analysis streaming](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/la_to_la.svg)

Figure 2. Log Analysis to Log Analysis streaming

You can only stream from one IBM Log Analysis instance to one other IBM Log Analysis instance. You cannot stream from the second IBM Log Analysis instance to another IBM Log Analysis instance.
FAQs 

This is a collection of frequently asked questions (FAQ) about the IBM® watsonx.data service.



 What is watsonx.data? 

watsonx.data is an open, hybrid, and governed fit-for-purpose data store optimized to scale all data, analytics and AI workloads to get greater value from your analytics ecosystem.





 What can I do with watsonx.data? 

You can connect to data in multiple locations and get started in minutes with built-in governance, security and automation. You can leverage multiple query engines to run analytics and AI workloads, reducing your data warehouse costs by up to 50%.





 What are the key features of watsonx.data? 

Key features of watsonx.data are:



* The architecture fully separates compute, metadata, and storage to provide ultimate flexibility.
* Fit for purpose query engines designed to handle modern data formats that are highly elastic and scalable.
* Data sharing between watsonx.data, Db2® Warehouse on Cloud, and Netezza or any other data management solution through common Iceberg table format support, connectors, and a shareable metadata store.







 How can I create a watsonx.data service instance? 

To create a watsonx.data service instance, see [Provision an instance](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started).





 How can I access watsonx.data web console? 

To access the watsonx.data web console, see [Open the web console](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started) in [Getting started with watsonx.data](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-getting-started).





 Which data formats for ingestion job are supported in watsonx.data? 

Data ingestion in watsonx.data supports .CSV and .Parquet formats.
* [Auditing events for Satellite](https://cloud.ibm.com/docs/satellite?topic=satellite-at_events)



* Setting up storage



* [Understanding Satellite storage](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-template-ov)
* [Storage template feature overview](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-template-features)
* AWS storage templates



* [AWS EBS](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-aws-ebs-csi-driver)
* [AWS EFS ](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-aws-efs-csi-driver)



* Azure storage templates



* [Azure Disk CSI driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azuredisk-csi-driver)
* [Azure File CSI Driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-azurefile-csi-driver)



* Google storage templates



* [Google Compute Engine persistent disk Container Storage Interface (CSI) Driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-gcp-compute-persistent-disk-csi-driver)



* IBM storage templates



* [IBM Systems Block Storage CSI driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-ibm-system-storage-block-csi-driver)
* [IBM Cloud Object Storage Driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-ibm-object-storage-plugin)
* [Block Storage for VPC Container Storage Interface (CSI) Driver](https://cloud.ibm.com/docs/satellite?topic=satellite-storage-ibm-vpc-block-csi-driver)



* NetApp storage templates
Storage DRS collects resource usage information for the datastores in a datastore cluster. vCenter Server uses this information to generate recommendations for placement of virtual disks on datastores.

When low aggressiveness level is set for a datastore cluster, Storage DRS recommends Storage vMotion migrations only when necessary. For example, if I/O load, space usage, or their imbalance is high, Storage DRS recommends a migration. If high aggressiveness level is set for a datastore cluster, Storage DRS recommends migrations whenever the datastore cluster can benefit from space or I/O load balancing.

The following threshold categories are available in the datastore cluster.



* Space Utilization - Storage DRS generates recommendations or performs migrations when the percentage of space utilization on the datastore is greater than the threshold you set in the vSphere Web Client.
* I/O Latency - Storage DRS generates recommendations or performs migrations when the 90th percentile I/O latency measured over a day for the datastore is greater than the threshold.



In this design, the Storage DRS Runtime Settings are enabled and thresholds are kept to their respective default values. Change these values based on the workload I/O requirements and latency sensitivity.

The following table shows the settings in the VMware vSphere® Web Client.



Table 1. Storage DRS runtime settings

 Setting Value 

 Enable I/O metric for SDRS recommendations Selected 
 Utilized space Selected, set to 80% 
 I/O latency threshold 15 ms 



For more information about configuring these settings in the vSphere Web Client, see [Set storage DRS runtime rules in the vSphere Web Client](https://docs.vmware.com/en/VMware-vSphere/5.5/com.vmware.vsphere.resmgmt.doc/GUID-AD2D13CE-539B-48C3-BBC9-E55A834874F0.html).





 Storage I/O Control for NFS v3 

When SIOC (Storage I/O Control) is enabled in the environment, it changes the device queue length for the single VM.
These workloads are typically characterized by having a large percentage of data inactive at a time. Example applications include storing mailboxes or departmental level file shares.
* 2 IOPS per GB is designed for most general-purpose usage. Example applications include hosting small databases that are backing web applications or virtual machine disk images for a hypervisor.
* 4 IOPS per GB is designed for higher-intensity workloads. These workloads are typically characterized by having a high percentage of data active at a time. Example applications include transactional and other performance-sensitive databases.
* 10 IOPS per GB is designed for the most demanding workloads such as those created by NoSQL databases, and data processing for Analytics. This tier is available in [all data centers](https://cloud.ibm.com/docs/BlockStorage?topic=BlockStorage-selectDC) for storage that is provisioned up to 4 TB.
* The available custom IOPS range depends on the volume capacity. The following table shows the available IOPS ranges based on volume size.





Table 1. Available IOPS based on volume size

 Volume size (GB) IOPS range 

 10 - 39 100 - 1,000 
 40 - 79 100 - 2,000 
 80 - 99 100 - 4,000 
 100 - 499 100 - 6,000 
 500 - 999 100 - 10,000 
 1,000 - 1,999 100 - 20,000 
 2,000 - 2,999 200 - 40,000 
 3,000 - 3,999 200 - 48,000 
 4,000 - 7,999 300 - 48,000 
 8,000 - 9,999 500 - 48,000 
 10,000 - 12,000 1,000 - 48,000 



8. In the side panel, review your order summary, and apply your Promo Code if you have one.

Discounts are applied when the order is processed.
9. Acknowledge that you reviewed the terms and conditions bycheck the appropriate box.
10. Click Create. Your new storage allocation is available in a few minutes.







 Ordering Block Storage for Classic through the SLCLI
* Storage disk options - 960 GB SSD, 1.9 TB SSD, 3.8 TB SSD, or 7.68 TB SSD.
* Disk quantity options - 2, 4, 6, 8, or 10.

Additionally, two cache disks of 960 GB are also ordered per host.

3.8 TB SSD (Solid-State Disk) drives are supported when they are made generally available in a data center.
* High Performance with Intel Optane - this option provides two extra capacity disk bays for a total of 10 capacity disks. It's available only for vSphere 6, and for Skylake and dual-processor Cascade Lake CPU models.









 Technical specifications for VMware vSphere expansion nodes 

Each VMware vSphere instance expansion node deploys and incurs charges for the following components in your IBM Cloud infrastructure customer portal account.



 Hardware for expansion nodes 

One IBM Cloud bare metal server with the hardware configuration presented in [Technical specifications for VMware vSphere](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverviewvs_vsphereclusteroverview-specs).





 Networking for expansion nodes 

One IBM Cloud bare metal server with the networking configuration presented in [Technical specifications for VMware vSphere](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverviewvs_vsphereclusteroverview-specs).





 VMware components for expansion nodes 



* One IBM Cloud bare metal server with VMware vSphere Enterprise Plus 7.0 or 6.7u3 for V4.7 and earlier.
* Optional VMware components presented in [Technical specifications for VMware vSphere](https://cloud.ibm.com/docs/vmwaresolutions?topic=vmwaresolutions-vs_vsphereclusteroverviewvs_vsphereclusteroverview-specs).
See [Encrypting your data](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-encryptionencryption).

In addition, or as an alternative to using IBM Cloud Object Storage storage encryption in analytic scenarios for large-scale data, you can use Parquet modular encryption, especially when fine-grained access control is important. See [Working with Parquet modular encryption](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-parquet-encryption-serverless).





 Encrypting endpoints 

All service endpoints to the cluster are SSL encrypted (TLS 1.2 enabled). In addition, when you use IBM Analytics Engine with IBM Cloud Object Storage, the link between the Object Storage service instance and IBM Analytics Engine is encrypted.





 Isolation and network access 

Each IBM Analytics Engine serverless instance gets is own isolated sandbox that is disconnected from other instances from a network and security stand point.

Spark workloads deployed in an instance can:



* Communicate with other Spark workloads deployed in the same instance.
* Communicate with public internet
* Can connect with other IBM Cloud® services over private end points



Spark workloads in one IBM Analytics Engine instance cannot communicate with Spark workloads in another instance. See [Instance architecture](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsserverless-architecture) for more on instance isolation.





 Ensuring code security 

You are advised to be cautious when applying libraries or package customization to your instance. You must use secure code from trusted sources only, so as not to compromise the overall security of the instances.

IBM recommends that you scan any source code, libraries, and packages you use before uploading them to your instance. While the use of non-trusted code will not impact others, it might impact you.