vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Adding facets 

Add more facets that you can use to filter your data.

When you apply custom enrichments to your collection, annotations are added to its documents. The annotations feed into new facets that you can use to sort your data.

The following table describes the types of facets that you can create from annotations.



Custom facet types

 Information to recognize Annotator type 

 Commonly understood terms, such as organization or people names. [Built-in Natural Language Processing models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-edit-collectioncm-enrichments) 
 Phrases that express an opinion and evaluate whether the opinion is positive or negative. [Phrase sentiment](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-phrase-sentiment) 
 Alternative words that share a meaning with terms in a finite list. [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotator) 
 Terms that match a syntactical pattern [Regular expression](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotator) 
 Custom terms by the context in which they are used. [Machine learning model](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotator) 
 Documents that fit into categories that you define. [Document classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifier) 





 Grouping facets 

To organize your facets, you can group them in folders.

Grouping facets does not combine the data from the facets. It merely makes the facets easier to find because they are organized in named folders.

To associate facets such that you can combine data from multiple facets, the facets must have a facet and subfacet relationship.
Annotation-based entites are those for which you annotate occurrences of the entity in sample sentences to teach your assistant about the context in which the entity is typically used.

In order to train a contextual entity model, you can take advantage of your intent examples, which provide readily-available sentences to annotate.

Using an intent's user examples to define contextual entities does not affect the classification of that intent. However, entity mentions that you label are also added to that entity as synonyms. And intent classification does use synonym mentions in intent user examples to establish a weak reference between an intent and an entity.



1. From your dialog skill, click the Intents tab.
2. Click an intent to open it.

For this example, the intent place_order defines the order function for an online retailer.

![Select #place_order intent](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent.png)
3. Review the intent examples for potential entity mentions. Highlight a potential entity mention from the intent examples.

In this example, computer is the entity mention.

![Review intent examples](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-review.png)

The Edit icon ![Edit icon](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/oe-intent-edit.png) is used to edit an intent user example; it is not related to adding annotations.
4. A Search box opens that you can use to search for the entity that the highlighted word or phrase is a mention of.

!
For more information, see the [String variation operator](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operatorsvariation).

For more information about the languages for which the stemmer is supported, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).





 Collection limits 

The number of collections that you can create per project differs by project type.



Collections per project limits

 Project type Collections per project 

 Document Retrieval 5 
 Document Retrieval for Contracts 5 
 Conversational Search 5 
 Content Mining 1 
 Custom 5 



The number of collections you can create per service instance depends on your Discovery plan type.



Plan details

 Plan Collections per service instance 

 Cloud Pak for Data 300 
 Premium 300 
 Enterprise 300 
 Plus (includes Trial) 40 



IBM Cloud Pak for Data

The number of collections you can create depends on your hardware configuration. Discovery supports a maximum of 300 collections per instance and installation, but that number depends on many factors, including memory.





 Supported file types 

Discovery can ingest specific file types. For all other types of files, a warning message is displayed and the file is not ingested.

The following table shows the supported file types and information about feature support that varies by file type.



Supported file types
This table has row and column headers. The row headers identify supported file types. The column headers identify different product features. To understand which features are available for a file type, go to the row that describes the document type, and find the columns for the feature you are interested in.

 File type Text extraction support Smart Document Understanding (SDU) support Optical Character Recognition (OCR) support 

 CSV ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 DOC, DOCX !
This figure includes all words, both OOV words and words that are already part of the service's base vocabulary, that are included in corpora or grammars. For corpora, the service uses these additional words to learn the context in which OOV words can appear, which is why corpora are a more effective means of improving recognition accuracy.



A large words resource can increase the latency of speech recognition, but the exact effect is difficult to quantify or predict. As with the amount of data that is needed to produce an effective custom model, the performance impact of a large words resource depends on many factors. Test your custom model with different amounts of data to determine the performance of your models and data.





 Working with corpora for previous-generation models 

You use the POST /v1/customizations/{customization_id}/corpora/{corpus_name} method to add a corpus to a custom model. A corpus is a plain text file that contains sample sentences from your domain. The following example shows an abbreviated corpus for the healthcare domain. A corpus file is typically much longer.

Am I at risk for health problems during travel?
Some people are more likely to have health problems when traveling outside the United States.
How Is Coronary Microvascular Disease Treated?
If you're diagnosed with coronary MVD and also have anemia, you may benefit from treatment for that condition.
Anemia is thought to slow the growth of cells needed to repair damaged blood vessels.
What causes autoimmune hepatitis?
A combination of autoimmunity, environmental triggers, and a genetic predisposition can lead to autoimmune hepatitis.
What research is being done for Spinal Cord Injury?
The National Institute of Neurological Disorders and Stroke NINDS conducts spinal cord research in its laboratories at the National Institutes of Health NIH.
NINDS also supports additional research through grants to major research institutions across the country.
Some of the more promising rehabilitation techniques are helping spinal cord injury patients become more mobile.
What is Osteogenesis imperfecta OI?
. . .
Show more

Speech recognition relies on statistical algorithms to analyze audio.
/dev/mapper/hana_vg-hana_shared_lv /hana/shared xfs defaults,inode64 0 0
/dev/mapper/hana_vg-hana_data_lv /hana/data xfs defaults,largeio,swalloc,inode64 0 0





 mx2-48x384 profile 

For Virtual Server created based on the mx2-48x384 profile there are:



* 3x 500 GB volumes; three block storage volumes of 500 GB size, with a custom volume profile that supports up to 10,000 Max IOPS attached to the Virtual Server is required
* 4x 100 GB volumes; four block storage volumes of 100 GB size, with a custom volume profile that supports up to 6,000 Max IOPS attached to the Virtual Server is required
* Optional: 1x 2,000 GB volume; one block storage volume of 2,000 GB size, with a lower 4,000 IOPS (medium performance) attached to the Virtual Server for backups



After attaching the seven data volumes, seven new virtual disks will appear in the Virtual Server, see the table that follows. In this example, those disks are vdd, vde, vdf, vdg, vdh, vdi, vdj.

These three disks must be managed under the Linux® Logical Volume Manager (LVM), and deployed as logical volumes. In order to achieve that, first put the three devices under LVM control. For example, make them physical volumes:

[root@hana384-vsi ] pvcreate /dev/vd[d,e,f,g,h,i,j]

Then, two different volume groups need to be created:

[root@hana384-vsi ] vgcreate hana_vg /dev/vdh /dev/vdi /dev/vdj
[root@hana384-vsi ] vgcreate hana_log_vg /dev/vdd /dev/vde /dev/vdf /dev/vdg

Next, three logical volumes need to be defined on top. These logical volumes reflect the file system size requirements for SAP HANA. The following commands are for a 384 GB virtual server:
How many Red Hat OpenShift on IBM Cloud clusters can I run before I need to attach capacity to the location control plane? 

The number of clusters depends on the size of your clusters and the size of the hosts that you use for the Satellite location control plane. You must scale up the control plane hosts in multiples of 3, such as 6, 9, or 12.

The following tables provide examples of the number of hosts that the control plane must have to run the masters for various combinations of clusters and worker nodes, for informational purposes only.



* The size of the hosts that run the control plane, 4 vCPU and 16GB RAM or 16 vCPU and 64GB RAM, affect the numbers of clusters and worker nodes that are possible in the location. Keep in mind that actual performance requirements depend on many factors, such as the underlying CPU performance and control plane usage by the applications that run in the location.
* You can assign hosts to the control plane in groups of 3. The table presents examples up to 12 hosts as common configurations to give you an idea of how you might size the control plane for your host and application environment. Note that you can add more than 12 hosts to your control plane in groups of 3. For example you might create a control plane with 18 or 27 hosts.







 What types of changes can increase my location sizing requirements? 

Your sizing requirements can increase depending on the amount of workload that is running in a cluster. The following examples can cause your sizing requirements for your location to increase.



* Large amounts of a dynamic pod workload, such as more storage required to hold all pod, service, or app metadata.
* Large amounts of configuration information, such as ConfigMaps and Secrets, which can lead to increased memory or CPU of control plane that is holding or processing that information.
* Aggregated kube-apiserver request workload and response sizes of data gathered. For example, if your cluster contains many ConfigMaps and an application queries for the full list of that data, that request can cause the control plane to require more resources.
Headers are not expected for the CSV file.





 Classifications training data requirements 



* Classifications training data consists of an array containing multiple JSON objects.
* Each of these JSON objects, needs to contain, 1 text and 1 labels field.
* text consists of the training examples and labels consists of 1 or more labels associated with an example.
* labels are case-sensitive
* Minimum number of unique labels required: 2
* Maximum number of unique labels allowed: 3000
* Minimum number of examples required per label: 5
* Maximum size of each example (training and predict): 2000[codepoints](https://en.wikipedia.org/wiki/Code_point)
* Maximum number of examples: 20000







 Classifications training parameters 

Passing in the optional training_parameters object allows you to specify characteristics of your classifier. Not passing in the object or an empty object into the request will train the model using default values.

Supported training parameters:



 Keys Default Value Optional Values 

 model_type multi_label single_label 



Description:



* model_type: Passing the single_label value will result in a single-label classifier, capable of handling training datasets with only one label per example. The single-label classifier will output normalized confidence scores so that the scores sum up to one. Passing the multi_label value will result in a multi-label classifier, capable of handling training datasets with multiple labels per example. The multi-label classifier will not output normalized confidence scores, in order to account for the added flexibility of associating multiple labels with examples.









 Training a custom classifications model 

When your training data is ready, use the Create classifications model method to create your custom classifications model. Make sure to substitute your credentials for {apikey} and {url}, and use the path to your training data file in the training_data parameter. Optionally, you can also specify characteristics of your classifier using training_parameters.
The field includes the string user if you add or modify the word directly.



After adding or modifying a word in a model's words resource, it is important that you verify the correctness of the word's definition; for more information, see [Validating a words resource for previous-generation models](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWordsvalidateModel). You must also train the model for the changes to take effect during transcription; for more information, see [Train the custom language model](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-languageCreatetrainModel-language).





 How much data do I need? 

Many factors contribute to the amount of data that you need for an effective custom language model. It is not possible to indicate the exact number of words that you need to add for any custom model or application. Depending on the use case, even adding a few words directly to a custom model can improve the model's quality. But adding OOV words from a corpus that shows the words in the context in which they are used in audio can greatly improve transcription accuracy.

The service limits the number of words that you can add to a custom language model:



* You can add a maximum of 90 thousand OOV words to the words resource of a custom model. This figure includes OOV words from all sources (corpora, grammars, and individual custom words that you add directly).
* You can add a maximum of 10 million total words to a custom model from all sources. This figure includes all words, both OOV words and words that are already part of the service's base vocabulary, that are included in corpora or grammars. For corpora, the service uses these additional words to learn the context in which OOV words can appear, which is why corpora are a more effective means of improving recognition accuracy.



A large words resource can increase the latency of speech recognition, but the exact effect is difficult to quantify or predict. As with the amount of data that is needed to produce an effective custom model, the performance impact of a large words resource depends on many factors.
The amount of data that is scanned depends on the amount of data that Data Engine must read to run your query, and not on the actual size of your data. Several factors play a role when it comes to how much data needs to be accessed to run a query. First, data layout is important. Columnar formats, such as Parquet, lead to less data to be scanned, as Data Engine can selectively read ranges and single columns. Furthermore, the actual object layout determines how many objects need to be scanned. Read [How to lay out big data in IBM Cloud Object Storage for Spark SQL](https://www.ibm.com/cloud/blog/big-data-layout) for more details on how to lay out big data on Cloud Object Storage to improve cost and performance of SQL queries. Each successful query is charged with at least 10 MB.



 Example 

Assume you have 1 PB of data that is stored on Cloud Object Storage that is laid out as described in the [blog post](https://www.ibm.com/cloud/blog/big-data-layout) and is optimized for the queries you want to run. If you run a single query, the most expensive query possible is SELECT * FROM, as reading 1 PB of data is required. Any other query is much cheaper and faster. For example, a 1 PB data set consists of audit events for users of a system (user A performed action B in system X at time T) and the data is laid out in a way that it is partitioned by time (one file per day and system). So to answer a query like SELECT DISTINCT user FROM WHERE System='X' AND Day >= (TODAY - 30), Data Engine must access all objects for system X that contain data for the last 30 days. The sum of the size of these objects is the maximum estimate of data that is scanned that you would be charged for. But as Data Engine accesses only one field, and data is stored as Parquet, it is much less. Calculating the precise price of the query is not possible in advance because much of it depends on the data itself. Parquet, for example, stores compressed columns, so if the column can be compressed effectively, even less data needs to be read.
FAQs 



 How many instances can share the use of a Block Storage for Classic volume? 

The default limit for the number of authorizations per block volume is eight. That means that up to eight hosts can be authorized to access the Block Storage for Classic LUN. Customers who use Block Storage for Classic in their VMware® deployment can request the authorization limit to be increased to 64. To request a limit increase, contact your sales representative or raise a [Support case](https://cloud.ibm.com/unifiedsupport/cases/add).

If multiple hosts mount the same Block Storage for Classic volume without being cooperatively managed, your data is at risk for corruption. Volume corruption can occur if changes are made to the volume by multiple hosts at the same time. You need a cluster-aware, shared-disk file system to prevent data loss such as Microsoft® Cluster Shared Volumes (CSV), Red Hat Global File System (GFS2), VMware® VMFS, and others. For more information, see your host's OS Documentation.





 Our compute hosts have multiple network cards with different IP addresses for network redundancy and expanded bandwidth. How can we authorize them all to access the same Storage volume? 

It is possible to authorize a subnet of IP addresses to access a specific Block Storage for Classic volume through the console, SLCLI, or API. To authorize a host to connect from multiple IP addresses on a subnet, complete the following steps.



 Console UI 



1. Go to [Classic Infrastructure](https://cloud.ibm.com/gen1/infrastructure/devices).
2. Click Storage > Block Storage for Classic.
3. Locate the volume and click the ellipsis ![Actions icon](https://cloud.ibm.com/docs-content/v1/content/14cf2f2f32b430cf4ed67ad14b3cecc010f91c45/icons/action-menu-icon.svg).
4. Click Authorize Host.
5. To see the list of available IP addresses, select IP address as the host type.
This will not affect the enrichments, but it is best to use only one of them to select and apply the enrichment to one or more fields. * If a web crawl appears to be stuck processing at a fixed number of documents, and the message displayed on the Logs page is The ingestion job <jobid> is terminated incorrectly, contact IBM support for assistance restarting the crawl. * If one or more of your collections is trained, the training data from one of those collection may display on the Train page of an untrained collection. Refresh the page to clear that training data. * The following types of documents will not be processed if they do not have the proper file extension: .docx, .pptx, .xlsx.<-- </ul> -->Also see the issues identified in the previous release.<-- <section "id="section-30aug2019ki-fixed" "> --> 2.0.1 issues that were fixed in subsequent releases <-- <ul> --> * [Fixed in version 2.1.2 When you upload documents to a collection with existing documents, a Documents uploaded message displays on the Activity page, but no further processing status displays until the number of documents increases.<-- </ul> --><-- </section "id="section-30aug2019ki-fixed" "> --><-- </section "id="section-30aug2019ki" "> --><-- <section "id="section-known-issues-ga" "> --> General Availability (GA) release, 28 June 2019 <-- <ul> --> * If you are working in the Discovery for Cloud Pak for Data tooling, and your IBM Cloud Pak® for Data session expires, you will receive a blank page. To return to the tooling, refresh the browser and log back in. * All JSON files ingested into Discovery should include the .json file extension. * When querying on the collection_id of a trained collection, the training_status.notices value may occasionally display as 0 instead of the correct value. * Not all query limitations are enforced in this release. See query limitations]] ! ! !
/hana/data hana_data_lv min. 1152 vg1 nvme4n1- <br>nvme7n1- 11.6 
 /hana/log 384 vg0 
 bx2d-metal-192x768 /hana/shared hana_shared_lv 768 vg0 nvme0n1- <br>nvme3n1- 11.6 
 /hana/data hana_data_lv min. 2304 vg1 nvme4n1- <br>nvme7n1- 11.6 
 /hana/log 512 vg0 
 mx2d-metal-96x768 /hana/shared hana_shared_lv 768 vg0 nvme0n1- <br>nvme3n1- 11.6 
 /hana/data hana_data_lv min. 2304 vg1 nvme4n1- <br>nvme7n1- 11.6 
 /hana/log 512 vg0 



Note, that both volume groups vg0 and vg1 are not fully used. Remaining space can be used to extend the listed sizes, which are only minimum sizes, or can be used for other purposes. However, do not point I/O load at the remaining space since that impacts SAP HANA’s performance.

To ensure a higher level of availability and failure resilience, RAID10 logical volumes are built on-top the under-laying NVMEs, based on Linux’s logical volume manager.

These steps show a step-by-step guide for setting up the volume groups, logical volumes, and file systems. Size information differs and can be retrieved from the storage table.



1. Log in to the OS and install the lvm2 package, if not installed already.

[root@mx2d-metal-96x768 ] yum install lvm2

This command applies to RHEL, on SLES use ‘zypper in’ instead.
2. Create the volume groups.