vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Lesson 1: Pre-annotating new documents with a machine learning model 

In this lesson, you will learn how to use a machine learning model to pre-annotate documents in Knowledge Studio.



 About this task 

After you train a machine learning model, you can use it to pre-annotate new documents that you add to the corpus.

Do not run a pre-annotator on documents that have been annotated by humans, but not been added to the ground truth yet. If you do, all current annotations will be stripped from the documents.

In this tutorial, you can add a second set of documents by using the documents-ml.csv file. Do not re-add the documents-new.csv file, since this addition would result in duplicate documents in the ground truth. Duplication causes the following problems:



* If annotations on each document do not match, they lower the quality of the machine learning model.
* If annotations on each document match, they over-train the machine learning model on the duplicated files.



For more information about pre-annotating documents, see [Bootstrapping annotation](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-preannotation). You can also read about other pre-annotation methods.





 Procedure 



1. Log in to Knowledge Studio as the administrator.
2. Upload more documents to the workspace. You can use the [documents-ml.csv](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-ml.csv) file.

For more information about adding documents to a workspace, see [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotation).
3. Create an annotation set that uses the documents-ml.csv file as the base set, and assign it to yourself, the administrator.
A topic is made up of one or more partitions. The messages on a partition are ordered by a number that is called the offset. With multiple partitions distributed across the brokers, the scalability of a topic is increased.

If a topic has more than one partition, it allows data to be fed through in parallel to increase throughput by distributing the partitions across the cluster. The number of partitions also influences the balancing of workload among consumers.

For more information, see [Partition leadership](https://cloud.ibm.com/docs/EventStreams?topic=EventStreams-partition_leadershippartition_leadership).

Zoom

![Topics and partitions diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/concepts_topics_and_partitions.png)

Figure 3. The diagram shows one topic with three partitions that are spread across three brokers.





 Replication 

To improve availability, each topic can be replicated onto multiple brokers. For each partition, one of the brokers is the leader, and the other brokers are the followers.

Replication works by the followers repeatedly fetching messages from the leader.

Zoom

![Replication diagram.](https://cloud.ibm.com/docs-content/v1/content/e0c3acba1296ff980e76d5db456d53adefaabba4/EventStreams/concepts_replication.png)

Figure 4. The diagram shows a topic partition that is replicated across three brokers.





 In-sync replicas 

A follower replica that is keeping up with the partition leader is in-sync. Any follower with an in-sync replica can become the leader without losing any messages.

If the partition leader fails, another leader is chosen from the followers. All the replicas are to be usually in-sync. It is acceptable for a replica to be temporarily not in-sync while it is catching up after a failure.

Zoom
If your training data includes documents with a combination of sections that are labeled and others that are not, the system might sample some examples from unlabeled sentences. Subsampling helps to balance the number of positive and negative examples that are used for training. Balancing the examples in the training set improves the training performance.







 Publishing the entity extractor as an enrichment 

When you think the entity extractor is ready, publish the entity extractor. How do you know when it's ready? If the score doesn't change after several test runs in which you make improvements, the model is ready. You can return to update and retrain the model after you publish it.



1. From the Evaluate extractor page, click Publish extractor.
2. Click Apply to data.
3. Choose a collection, and then select the document field where you want the entity extractor enrichment to be applied.
4. Click Apply.







 Exporting the entity extractor 

For installed deployments, the export capability was added with the 4.6.2 release.

An entity extractor model that you create and deploy in one project is available as an enrichment that can be applied to a collection from any project in the same service instance.

If you want to use the entity extractor model in a project from another service instance, you can export the entity extractor. To use it elsewhere, follow the steps to create a machine learning model from [Use imported ML models to find custom terms](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-ml). You cannot continue to edit an entity extractor that you import into another project.

The entity extractor that you want to export must be fully trained.

To export an entity extractor, complete the following steps:



1. Open the project with the entity extractor that you want to export.
2. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Extract entities.
3. From the Entity extractors list, find the entity extractor that you want to export.
4.
As this number stays small, increasing data size has no effect on query latency, unlike global queries.







 Partitioned databases tutorials 

You can see two examples of using partitioned databases:



1. Read about [partitioned databases and Node.js](https://blog.cloudant.com/2019/05/24/Partitioned-Databases-with-Cloudant-Libraries.html) in this blog article that includes how to create a partitioned database, search, views, and a global index.
2. Read the following example about using views and the _all_docs endpoint.







 Example - Partitioning IoT reading data 

This discussion is abstract; let's make it concrete with an example. We take the Internet of Things domain and look at using IBM Cloudant as a historian for device readings. Say that the devices provide sensor readings on pieces of infrastructure like roads or bridges.

Review the following assumptions:



* Hundreds or thousands of devices that report readings.
* Each device has a unique ID.
* Each piece of infrastructure has a unique ID.
* Devices aren't moved between pieces of infrastructure.
* Each device writes a reading to IBM Cloudant every 10 seconds. Likely this reading is delivered by using a message bus to IBM Cloudant.



In a non-partitioned database, you might allow IBM Cloudant to generate document IDs. Another alternative is to name documents by device ID and record timestamp.

Using the second approach, we'd end up with document IDs like the following example:

device-123456:20181211T11:13:24.123456Z

The timestamp could also be an epoch timestamp.

This approach allows the data for each device to be queried efficiently by using partitioned indexes. However, global indexes might need to be used to create views over multiple devices, for example, all devices on a specific piece of infrastructure.

For illustrative purposes, let's make the scenario a bit more complicated. Assume that the application mostly needs to read all sensor data for a specific piece of infrastructure rather than for individual devices.
For all of the following layouts consider that the volume names might differ - we assume that the naming follows the sequence of ordering the storage, that is, 1st order -> vdd, 2nd order -> vde, and so on. All block storage volumes must be ordered with the predefined profile of 10 IOPS/GB (high performance). One exception might be /hana/shared partition where 5 IOPS/GB (medium performance) are sufficient - but ONLY IF you assigned a dedicated volume for this partition. For all profiles optional: one appropriately sized block storage volume or several equally sized volumes that are gathered to a volume group, with the predefined profile of 5 IOPS/GB (medium performance) attached to the Virtual Server for backups.

[SAP's recommended file system layout](https://help.sap.com/viewer/2c1988d620e04368aa4103bf26f17727/2.0.latest/en-US/4c24d332a37b4a3caad3e634f9900a45.html) must be available for SAP HANA deployment.





 mx2-* profiles - Storage Layouts 

The following table shows the required physical volumes, related volume groups, logical volumes, and their characteristics:



Table 5. Storage layout for mx2-* profiles based virtual servers

 Profile File <br>system Logical <br>Volume LV Size <br>(GB) Volume Group Physical <br>Volume PV Size <br>(GB) 

 mx2-16x128 <br>and <br>mx2-32x256 /hana/shared hana_shared_lv 256 hana_vg vdd 500 
 /hana/data hana_data_lv 256 vde 500 
 /hana/log hana_log_lv 988 vdf 500 
 mx2-48x384 /hana/shared hana_shared_lv 384 hana_vg vdd 500 
 /hana/data hana_data_lv 616 vde 500 
 vdf 500 
 /hana/log hana_log_lv 400 hana_log_vg vdg 100 
 vdh 100 
 vdi 100 
 vdj 100
SAP HANA Data and SAP HANA Log directory file systems must be striped over 4 or 8 storage volumes.

Striping for other file systems (SAP executable files, HANA Shared, HANA Backup, HANA Export) is not required. See the SAP TDI documents for general storage requirements.



Table 6. Example of a storage layout for an SAP HANA system

 HANA file system Total Space Required Size of Volumes Number of Volumes 

 /hana/data 560 70 8 
 /hana/log 640 80 8 
 /hana/shared 200 200 1 
 /usr/sap 130 130 1 
 /backup 560 560 1 
 /export 560 560 1
Similarly, when an alphanumeric word is used in the dictionary entry, the synonyms are handled with Hankaku characters, except for the same alphanumeric word, which is represented by Zenkaku characters. The Zenkaku word is treated as a separate term from the Hankaku words.

Dictionary enrichments that you add to one project can be applied to collections in other projects in the same service instance. In fact, you can apply them to collections in a Content Mining project from the deployed Content Mining application.





 Dictionary limits 

The number of dictionaries and term entries you can create per service instance depends on your Discovery plan type.



Dictionary plan limits

 Plan Number of dictionaries per service instance Number of term entries per dictionary Number of terms for which suggestions can be generated 

 Cloud Pak for Data Unlimited Unlimited 1,000 
 Premium 100 10,000 1,000 
 Enterprise 100 10,000 1,000 
 Plus (includes Trial) 20 1,000 50
[root@vx2d-144x2016 ] vgcreate hana_data_vg /dev/vde /dev/vdf /dev/vdg /dev/vdh
[root@vx2d-144x2016 ] vgcreate hana_log_vg /dev/vdi /dev/vdj /dev/vdk
2. After the volume group is created, two logical volumes need to be defined on top. These logical volumes reflect the file system size requirements for SAP HANA.

[root@vx2d-144x2016 ] lvcreate -i 4 -I 64 -l 100%VG -n hana_data_lv hana_data_vg
[root@vx2d-144x2016 ] lvcreate -i 3 -I 64 -l 100%VG -n hana_log_lv hana_log_vg
3. Now proceed with the same instructions that are listed in steps 3 and 4 [profile vx2d-16x224](https://cloud.ibm.com/docs/sap?topic=sap-hana-iaas-offerings-profiles-intel-vs-vpcvx2d-16x224).









 ux2d-* profiles - Storage Layouts 

The following table shows the required volumes and related volume groups, if necessary, and their characteristics:



Table 7. Storage for ux2* profile based virtual servers

 Profile File <br>system Logical <br>Volume LV Size <br>(GB) Volume Group Physical <br>Volume PV Size <br>(GB) 

 ux2d-8x224 /hana/shared hana_shared_lv 224 hana_vg vde 1,120 
 /hana/data hana_data_lv 672 
 /hana/log hana_log_lv 224 
 ux2d-16x448 /hana/shared hana_shared_lv 448 hana_vg vde 2,240 
 /hana/data hana_data_lv 1,344 
 /hana/log hana_log_lv 448
In development instances, the limit is 1,000 open shards, and in production instances, the limit is two data nodes, which is equal to 2,000 open shards, or 1,000 open shards per data node. After you reach either limit, you cannot create any more projects and collections on your cluster, and if you try to create a new project and collection, you receive an error message.

This limit is due to the fact that, when you install Discovery version 2.2.0, Elasticsearch version 7.8.0 automatically runs on your clusters. Because this version of Elasticsearch runs on your clusters, a new cluster stability configuration becomes available that limits the number of open shards to 1,000 for each Elasticsearch data node.

If you are unable to create new projects and collections and you receive errors, first check the status of your Elasticsearch cluster and the number of shards on that cluster. Consider increasing the number of data nodes on your cluster to support more shards. This method is optimal for maximizing performance. However, an increased number of nodes uses more memory. If the number of shards reaches the limit, you can also increase the limit in a data node. For more information about increasing the shard limit in a node, see [Increasing the shard limit](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-troubleshootincrease-shards).

This limit of 1,000 shards does not apply to versions of Discovery that are earlier than 2.2.0.



 Increasing the shard limit 



1. Log in to your Discovery cluster.
2. Access your data node.
3. Enter the following command:

oc exec -it $(oc get pod -l app=elastic,ibm-es-data=True -o jsonpath='{.items[0].metadata.name}') -- bash
4. Enter the following command, replacing the <> and the content inside with your port number:

curl -X POST http://localhost:<port_number>/_cluster/health?pretty

If you do not know what your port number is, enter the following command to find it:
Through training and refinement of example input data, the model can deliver accurate, repeatable results when it analyzes new data.
* machine learning annotator

See [machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-glossarygloss_M).
* machine learning model

A component that identifies entities and entity relationships according to a statistical model that is based on ground truth. The model applies past experience, such as training data, to determine or predict the correct outcome of future experiences based on characteristics of the data. These past experiences are captured in the form of a model by calculating feature scores for each candidate answer or evidence and combining that with known outcomes. Sometimes referred to as machine learning annotator.
* mention

A span of text that you consider relevant in your domain data. For example, in a type system about automotive vehicles, occurrences of terms like "airbag", "Ford Explorer", and "child restraint system" might be relevant mentions.







 N 



* named entity

A concept in a domain that falls in to a well-defined category, such as names of organizations, locations, authors, or diseases.
* natural language processing

A field of artificial intelligence and linguistics that studies the problems inherent in the processing and manipulation of natural language, with an aim to increase the ability of computers to understand human languages.







 O 



* ontology

An explicit formal specification of the representation of the objects, concepts, and other entities that can exist in some area of interest and the relationships among them.







 P 



* part of speech (POS)

In a dictionary, individual lexical items are assigned part of speech (POS) tags. For example, the word 'fly' can be identified as a verb or a noun.
* performance

The measurement of a Watson system in terms of accuracy, precision, and recall, for example, when answering questions, discovering relationships, or annotating text.
* pre-annotation

The process of annotating a set of documents prior to human annotation.
Upload one CSV file at a time. The first column in the CSV file specifies the file name of the document. The second column in the file contains the document text. For an example of the required format, see the [documents-new.csv](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-new.csv) file in the tutorial sample files.





 PDF files 

Text cannot be extracted from a PDF in some cases, depending on how the PDF was created. Typically, text can't be extracted from embedded fonts that don't map to unicode characters. If you are unsure whether text from a PDF can be extracted, you can try copying the text from the PDF and then pasting it into a text editor. If you do not see the same characters that are visible in the PDF itself, then the text extraction would likely fail.





 Formatted documents 

When formatted documents are converted to plain text, it's possible that losing the formatting could result in poor tokenization of words. For example, if a table row in a DOCX file contains cell values that do not end with a period, the values might be converted as one sentence. As another example, if a PDF document contains a very long word that is hyphenated at the end of a line, that word might be converted as two words. In cases like these, the documents might not be suitable for machine learning unless you pre-process the files to fix formatting limitations.





 Documents from another Watson Knowledge Studio workspace 

If you previously downloaded documents from a Knowledge Studio workspace, you can upload the ZIP file that you downloaded. An option lets you specify whether you want the ground truth annotations to be included in the imported files.

After documents are annotated, the annotated documents are stored in JSON format. The markup language in these files, which shows how the original document text was parsed and tokenized, includes elements for all of the annotations that a human annotator added. To improve model accuracy over time, you can upload these files into another workspace, thus preserving all of the existing annotations.
Through training and refinement of example input data, the model can deliver accurate, repeatable results when it analyzes new data.
* machine learning annotator

See [machine learning model](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-glossarygloss_M).
* machine learning model

A component that identifies entities and entity relationships according to a statistical model that is based on ground truth. The model applies past experience, such as training data, to determine or predict the correct outcome of future experiences based on characteristics of the data. These past experiences are captured in the form of a model by calculating feature scores for each candidate answer or evidence and combining that with known outcomes. Sometimes referred to as machine learning annotator.
* mention

A span of text that you consider relevant in your domain data. For example, in a type system about automotive vehicles, occurrences of terms like "airbag", "Ford Explorer", and "child restraint system" might be relevant mentions.







 N 



* named entity

A concept in a domain that falls in to a well-defined category, such as names of organizations, locations, authors, or diseases.
* natural language processing

A field of artificial intelligence and linguistics that studies the problems inherent in the processing and manipulation of natural language, with an aim to increase the ability of computers to understand human languages.







 O 



* ontology

An explicit formal specification of the representation of the objects, concepts, and other entities that can exist in some area of interest and the relationships among them.







 P 



* part of speech (POS)

In a dictionary, individual lexical items are assigned part of speech (POS) tags. For example, the word 'fly' can be identified as a verb or a noun.
* performance

The measurement of a Watson system in terms of accuracy, precision, and recall, for example, when answering questions, discovering relationships, or annotating text.
* pre-annotation

The process of annotating a set of documents prior to human annotation.
Can be increased for individual pods through the cluster-autoscaler.kubernetes.io/pod-scale-up-delay annotation. 0s 
 balancingIgnoreLabel The node labels to ignore during zone balancing. Apart from the fixed node labels, the autoscaler can ignore an additional 5 node labels during zone balancing. For example: balancingIgnoreLabel:key1=value1 and balancingIgnoreLabel:key2=value2 
 workerPoolsConfig.json The worker pools that you want to autoscale, including their minimum and maximum number of worker nodes per zone.<br><br><br><br> * {"name": "<pool_name>","minSize": 1,"maxSize": 2,"enabled":false}.<br> * <pool_name>: The name or ID of the worker pool that you want to enable or disable for autoscaling. To list available worker pools, run ibmcloud oc worker-pool ls --cluster <cluster_name_or_ID><br> * maxSize: <number_of_workers>: The maximum number of worker nodes per zone that the cluster autoscaler can scale up to. The value must be equal to or greater than the value that you set for the minSize: <number_of_workers> size.<br> * min=<number_of_workers>: The minimum number of worker nodes per zone that the cluster autoscaler can scale down to. The value must be 2 or greater so that your ALB pods can be spread for high availability. If you [disabled](https://cloud.ibm.com/docs/openshift?topic=openshift-kubernetes-service-clics_alb_configure) all public ALBs in each zone of your standard cluster, you can set the value to 1. Keep in mind that setting a min size does not automatically trigger a scale-up. The min size is a threshold so that the cluster autoscaler does not scale to fewer than this minimum number of worker nodes per zone.
Can be increased for individual pods through the cluster-autoscaler.kubernetes.io/pod-scale-up-delay annotation. 0s 
 balancingIgnoreLabel The node labels to ignore during zone balancing. Apart from the fixed node labels, the autoscaler can ignore an additional 5 node labels during zone balancing. For example: balancingIgnoreLabel:key1=value1 and balancingIgnoreLabel:key2=value2 
 workerPoolsConfig.json The worker pools that you want to autoscale, including their minimum and maximum number of worker nodes per zone.<br><br><br><br> * {"name": "<pool_name>","minSize": 1,"maxSize": 2,"enabled":false}.<br> * <pool_name>: The name or ID of the worker pool that you want to enable or disable for autoscaling. To list available worker pools, run ibmcloud ks worker-pool ls --cluster <cluster_name_or_ID><br> * maxSize: <number_of_workers>: The maximum number of worker nodes per zone that the cluster autoscaler can scale up to. The value must be equal to or greater than the value that you set for the minSize: <number_of_workers> size.<br> * min=<number_of_workers>: The minimum number of worker nodes per zone that the cluster autoscaler can scale down to. The value must be 2 or greater so that your ALB pods can be spread for high availability. If you [disabled](https://cloud.ibm.com/docs/containers?topic=containers-kubernetes-service-clics_alb_configure) all public ALBs in each zone of your standard cluster, you can set the value to 1. Keep in mind that setting a min size does not automatically trigger a scale-up. The min size is a threshold so that the cluster autoscaler does not scale to fewer than this minimum number of worker nodes per zone.