vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Connecting to a local file system data source 

From your Discovery project, complete the following steps:



1. From the navigation pane, choose Manage collections.
2. Click New collection.
3. Click Local File System, and then click Next.
4. Name the collection.
5. If the language of the documents that you want to crawl is not English, select the appropriate language.

For a list of supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).
6. Optional: Change the synchronization schedule.

For more information, see [Crawl schedule options](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionscrawlschedule).
7. In the Specify what you want to crawl section, enter the file path that you want to crawl in the Path field, and then click Add.

The file path is case-sensitive. Remember, only files in the /mnt directory or one of its subdirectories can be accessed by the crawler.
8. Optionally, add more file paths.
9. If you want to limit the types of files to add to the collection, you can list the file extensions for file types to either include or exclude.

For a list of supported file types, see [Supported file types](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionssupportedfiletypes).

Support for this option was added with the 4.7.0 release.
10. If you want the crawler to extract text from images in documents, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly.
* Id: The alphanumeric ID that Discovery uses when you submit a text query. For example, ans1.
* Fields: The text fields that have the Analyzable text content setting applied. Select one or multiple text fields that you want to group for facet analysis.



4. Click Add.

Repeat this task if you want to add more text fields that you want to group for facet analysis.
5. Click Save.



Now you can return to the data analysis page for your collection. From the Facet analysis panel, you can click Contextual view selection to see the text fields that you grouped. You can select one of the text fields to view the facet analysis for that field.





 Enrich your collection 

Discovery provides built-in natural language processing models, such as the Entities enrichment that can recognize mentions of commonly known things, such as business or location names and other types of proper nouns. You can apply these built-in NLP enrichments to your collection.

You can also apply a document classifier enrichment that you created in the Content Mining application to your collection.

Alternatively, you can apply enrichments that were built in other projects in the same service instance to the collection in your content mining project. For example, you can apply a dictionary or text classifier that was built in another project in the same service instance to your collection.

To apply enrichments to your collection, complete the following steps:



1. Complete the steps in [Editing a collection](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-edit-collectioncm-edit-collection-task) to get the collection into edit mode.
2. Click the Enrichment tab, and then select the enrichments that you want to apply to your collection.
3. Click Save.
[Shows a search result in the Preview](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/search-preview-example.png)

To show the exact answer highlighted in bold font, enable the Emphasize the answer feature that is available with Discovery v2 instances.

Watch a 4-minute video that provides an overview of the search skill:

To learn more about how search skill can benefit your business, [read this blog post](https://medium.com/ibm-watson/adding-search-to-watson-assistant-99e4e81839e5).

Find out how to keep your assistant data current by reading the blog post, [COVID-19: Are Your Virtual Assistant’s Answers Up-To-Date?](https://medium.com/ibm-watson/covid-19-are-your-virtual-assistants-answers-up-to-date-c9e1ba70eb65).



 How it works 

The search skill searches for information from a data collection that you create by using the Discovery service.

Discovery is a service that crawls, converts, and normalizes your unstructured data. The product applies data analysis and cognitive intuition to enrich your data such that you can more easily find and retrieve meaningful information from it later. To read more about Discovery, see the [product documentation](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-about).

Typically, the type of data collection you add to Discovery and access from your assistant contains information that is owned by your company. This proprietary information can include FAQs, sales collateral, technical manuals, or papers written by subject matter experts. Mine this dense collection of proprietary information to find answers to customer questions quickly.

The following diagram illustrates how user input is processed when both a dialog skill and a search skill are added to an assistant.

![Diagram that shows how some user input is answered by dialog and other questions are answered by search.]
Download[iris_initial.csv](https://github.com/IBM-Cloud/ml-iris-classification/raw/master/data/iris_initial.csv) which consists of 40 instances of each species. Make sure the downloaded file is named iris_initial.csv.



1. Under Data in this project, click Drop data files here or browse for files to upload.
2. Upload the downloaded iris_initial.csv.
3. Once added, you should see iris_initial.csv under the All assets section of the project.









 Step 2: Associate the Machine Learning service 



1. In the top navigation menu, click on iris-project, click on Manage in the top bar and select the Services & integrations section.
2. Click Associate Service.
3. If you have an existing Watson Machine Learning service instance, skip to the next step. Otherwise continue with the following steps to create a new instance.



1. Click New service and then click on the Watson Machine Learning tile.
2. Select a region same as the Watson Studio service and choose a Lite plan.
3. Enter machine-learning-tutorial as the Service name and select a resource group.
4. Click Create to provision a Machine Learning service.



4. Check the checkbox next to the Machine Learning service and click Associate service.







 Step 3: Build a machine learning model 



1. In the top navigation menu, click on iris-project, click on Assets in the top bar.
2. Click on New asset + and select AutoAI.



1. On the left pane, click +New.
2. Set the name to iris_model.
3. Under Watson Machine Learning service instance, select the service previously associated.



3. Click Create.



Once the model is created,



1. Add training data by clicking Select data from project.



1. Choose the iris_initial.csv file under Data asset.
2. Click Select asset.



2.
The creation of a model is an iterative multiple-step process that involves several stages: knowledge curation, ground truth generation, model development, model evaluation, and runtime deployment.



 End-to-end domain adaptation 

The following diagram summarizes the interactions between these five stages of model development and the typical activities that occur at each stage.

![A summary of the five stages of model development and the activities that occur at each stage.](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/watson-knowledge-studio/images/wks_flow.svg) Figure 2. A summary of the five stages of model development and the activities that occur at each stage.





 Knowledge curation 

This stage, which is external to Knowledge Studio, refers to the process of selecting, collecting, preserving, and maintaining content relevant to a specific domain. Curation adds value to data; it transforms data into trusted information and knowledge.





 Ground truth generation 

This stage refers to the use of Knowledge Studio tools and best practices to produce a collection of vetted data that can be used to adapt a Watson solution to a particular domain. The accuracy of this vetted data, called ground truth or gold standard documents, is critical because inaccuracies in the ground truth will correlate to inaccuracies in the applications that rely on it.

An essential part of teaching Watson about a new domain involves providing it with knowledge about entities of interest in your domain content, the relationships between them, and how the entities co-reference each other. Collecting this knowledge includes the following activities:



* Involving domain subject matter experts to create the following resources, or to identify existing resources that can be re-used or modified for your domain:



* Annotation guidelines and examples to help human annotators learn how words and passages in your domain content are to be annotated.
* Type systems that define the domain-specific types (objects) and features (data classifications) that can be discovered in your domain content through text analysis. The type system controls the types of annotations that a human annotator can add to documents.
Gathering Ingress logs 

Run the following commands to gather the required logs for debugging Ingress.



1.  List ALBs.

ibmcloud oc ingress alb ls -c CLUSTERID
2.  Get the Ingress status.

ibmcloud oc ingress status-report get -c CLUSTERID
3.  Get nodes and node labels.

oc get nodes --show-labels
4.  Get endpoints.

oc get endpoints -o wide
5.  Describe your Ingress resource.

oc describe ing <ingress name> -n <namespace>
6.  Get the logs for the nginx-ingress container in your Ingress pods.

oc logs <ingress pods> -n kube-system -c nginx-ingress
7.  List host names for network load balancer (NLB).

ibmcloud oc nlb-dns ls -c CLUSTERID
8.  List the alb pods in the kube-system namespace.

oc get pods -n kube-system | grep alb
9.  Run an nslookup on your Ingress subdomain.

nslookup ingresssubdomain.com



Review the output for error messages, then review the list of [Ingress and routers troubleshooting topics](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_ingress_and_routers) for help resolving common Ingress issues.
source-files:<SOURCE_DATA_FILE>
source-s3-creds:AWS_SECRET_ACCESS_KEY=<YOUR_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_ACCESS_KEY_ID>,AWS_REGION=<YOUR_REGION>, BUCKET_NAME=<YOUR_BUCKET>, ENDPOINT_URL=<YOUR_ENDPOINT_URL>
staging-location:<STAGING_LOCATION>
staging-s3-creds:AWS_SECRET_ACCESS_KEY=<YOUR_TARGET_SECRET_ASSESS_KEY>,AWS_ACCESS_KEY_ID=<YOUR_TARGET_ACCESS_KEY_ID>,AWS_REGION=<YOUR_TARGET_REGION>, BUCKET_NAME=<YOUR_TARGET_BUCKET>, ENDPOINT_URL=<YOUR_TARGET_ENDPOINT_URL>

For example:

[global-ingest-config]
target-tables:iceberg_cat.ice_schema.customer1_tab
ingestion-engine:hostname=localhost,port=8080
create-if-not-exist:true

[ingest-config1]
source-files:s3://cust-bucket/warehouse/a_source_file1.csv, s3://cust-bucket/warehouse/a_source_file2.csv
source-s3-creds:AWS_ACCESS_KEY_ID=xxxxxx,AWS_SECRET_ACCESS_KEY=yyyyyy,AWS_REGION=us-east-1,BUCKET_NAME=cust-bucket
staging-location:s3://cust-bucket/warehouse/staging/
staging-s3-creds:AWS_ACCESS_KEY_ID=zzzzzz,AWS_SECRET_ACCESS_KEY=vvvvvv,ENDPOINT=http://some_site/xxx.com?addsf:dfsdf,AWS_REGION=us-east-1,BUCKET_NAME=cust-bucket

[global-ingest-config]
The options that are displayed differ depending on the data source type.



* For a Salesforce data source, you select the object types that you want to extract from the source documents. You might select a [Case object type](https://developer.salesforce.com/docs/atlas.en-us.object_reference.meta/object_reference/sforce_api_objects_case.htm!) that represents a case, which is a customer issue or problem, for example.
* For a Sharepoint data source, you specify paths.
* For file repositories, you specify directories or files.
* For a web crawl data source, specify the base URL of a website that you want to crawl. The web page that you specify and any pages that it links to are crawled and a document is created per web page.



Give Watson a few minutes to start creating documents. As soon as the source starts to be ingested, the number of documents displayed on the Discovery details page increases. You might need to refresh the page.

To get help with creating data sources, see [Troubleshooting](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addskill-search-add-troubleshoot).
5. Click Finish.

The data collection is created. After the process completes, a summary page is displayed in Discovery.



* To create a collection by uploading documents, click Upload data, and then click Next.



1. First, you define the collection, and then you upload the documents. Provide the following information:



* Collection name. The name must be unique for this service instance.
* Language. Select the language of the files that you are adding to this collection.

For information about the languages supported by Discovery, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

If you are uploading a PDF document and want to extract party, nature, and category information from it, then expand the Advanced section and click Use the Default Contract Configuration with this collection.



2. Click Next.
After training data is uploaded, review this [blog post](https://developer.ibm.com/dwblog/2017/get-relevancy-training/) to ensure you meet the previous accuracy numbers. Keep in mind that old document ids must be transferred to the new training data, or updated to reflect the new document ids in your new collection.







 Restoring connections to external data sources 

In case of an unanticipated loss of data, you might lose your scheduled crawls of external data sources. See [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for the list of available sources.

To restore your external data, re-establish your connections to these data sources, and then re-crawl them.

To find the data source credentials that you stored, follow the instructions for your chosen data source in [Connecting to Data Sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources). These instructions explain how you can reconnect to your data sources and get the data imported into Discovery.





 Restoring Smart Document Understanding models 

To import a previously exported Smart Document Understanding (SDU) model, see [Importing and exporting models](https://cloud.ibm.com/docs/discovery?topic=discovery-sduimport). SDU models have the file extension of .sdumodel.

When importing an SDU existing model into a new collection, it is a good best practice to create the new collection and add one document, then import the model and upload the remainder of your documents.
globalSearchService.search(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err)
});
2. Extract the value of the CRN field from the response.

{
"items": [
{
"crn": "crn:v1:bluemix:public:cf:au-syd:a/000af2ea12345aabb1234567801fffab::cf-organization:aaaf4100-0011-2233-1111-11aaffee0011"
}
],
"limit": 1,
"search_cursor": "e34535305339jg0rntt39nt3nu5tt9yn3u5ntt329nutt92u4ntt92u4t9u5gt2u5gt92u4n5g982458hg2t45hg9u42rg9t2u49gh285ght28h5t2835th85ht318h4tg9138h4tg91u3hgt931hg45918h34tg18h43hgt31hgt3rng0fnefvodnfvpojdpbojarfdbpojeafrbeafbjeqpnrqngrpqgrhbHNlIiwiY2FuVGFnIjoiZmFsc2UiLCJsdfshfksdhfkshfdkshfdkhkhewrkfehrkhkwhfkwrhkfhrgk3h5k3h45k3hk45hgk3hgk3hfk3h4k3hfkrfh3rkgh3krghk3rgh3kghk3hgk3hgk3rhgdWJsaWM6Y2Y6YXUtwriretoeiroteito4i5ot4i5oyti45ito4tio45ito45io5ogno5iogn5oin5oingoi5o5ngogo4ngo4ngro3jrong3gor3g3gno3jrgo3jrngo3ngro3g32ODQiXX0t"
}
3. Call the following command:



* Curl
* Java
* Python
* Go
* Node



curl -X POST -H "Authorization: {iam_token}" -H "Accept: application/json" -H "Content-Type: application/json" -d '{ "resources": [{ "resource_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/59bcbfa6ea2f006b4ed7094c1a08dcdd:1a0ec336-f391-4091-a6fb-5e084a4c56f4::" }], "tag_names": ["project:myproject"] }' "https://tags.global-search-tagging.cloud.ibm.com/v3/tags/detach?tag_type=access"

Resource resourceModel = new Resource.Builder().resourceId(resourceCRN).build();
DetachTagOptions detachTagOptions = new DetachTagOptions.Builder()
.addResources(resourceModel)
.addTagNames("project:myproject")
Export a sample set of the logs from ineffective conversations, and then analyze and annotate them.

For example, indicate whether a response is correct. If correct, mark whether it is helpful. If a response is incorrect, then identify the root cause, the wrong intent or entity was detected, for example, or the wrong dialog node was triggered. After identifying the root cause, indicate what the correct choice would have been.
6. Feed the annotated spreadsheet to the Analyze Watson Assistant Effectiveness notebook.



* Effectiveness: Performs a deeper analysis of your logs to help you understand the steps you can take to improve your assistant.



7. Use the Dialog Flow Analysis for Watson Assistant notebook to review your dialog. The notebook can help you pinpoint the dialog nodes where customers most frequently abandon the conversation.

For more information about how this notebook can help you analyze and assess abandonment, read this [Medium.com blog post](https://medium.com/ibm-watson/do-you-know-where-and-why-users-drop-off-the-conversation-6246e99baddc).



This process helps you to understand the steps you can take to improve your assistant.





 Using the notebooks with standard Python tools 

If you choose to use standard Python tools to run the notebooks, you can get the notebooks from GitHub.



* [Dialog Skill Analysis for Watson Assistant](https://github.com/watson-developer-cloud/assistant-dialog-skill-analysis)
* [Watson Assistant Recommendation notebooks (Measure and Analyze Effectiveness)](https://github.com/watson-developer-cloud/assistant-improve-recommendations-notebook)
* [Watson Assistant Dialog Flow Analysis notebook](https://github.com/watson-developer-cloud/assistant-dialog-flow-analysis)
[Shows that the first node in the dialog asks which type of cupcake the user wants, gluten-free or regular, and has two child nodes that provide a different response depending on the user's answer.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/node1-children.png)





 Dialog flow 

The dialog that you create is processed by your assistant from the first node in the tree to the last.

![Arrow points down next to 3 nodes to show that dialog flows from the first node to the last](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/node-flow-down.png)

As it travels down the tree, if your assistant finds a condition that is met, it triggers that node. It then moves along the triggered node to check the user input against any child node conditions. As it checks the child nodes it moves again from the first child node to the last.

Your assistant continues to work its way through the dialog tree from first to last node, along each triggered node, then from first to last child node, and along each triggered child node until it reaches the last node in the branch it is following.

![Shows arrow 1 pointing from the first root node to the last, arrow 2 pointing from along the length of a triggered node, and arrow 3 pointing from the first to the last child nodes of the triggered node.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/node-flow.png)

When you start to build the dialog, you must determine the branches to include, and where to place them. The order of the branches is important because nodes are evaluated from first to last.
To extract the training data, use the API to download the queries and the ratings from Discovery.



1. [List the training data](https://cloud.ibm.com/apidocs/discoverylist-training-data)
2. [List the examples](https://cloud.ibm.com/apidocs/discoverylist-examples-for-a-training-data-query) for each query.
3. [Get the details](https://cloud.ibm.com/apidocs/discoveryget-details-for-training-data-example) for a training data examples.



The document IDs that you use in your training data point to the documents in your current collection. Use the same IDs in your new collections to ensure that the correct documents are referenced. If the IDs do not match, your restored relevancy training will not work.





 Queries 

By default, Discovery stores the queries that you send to it, unless you opt out.

If you want to be able to restore your queries for [statistical purposes](https://cloud.ibm.com/apidocs/discoverynumber-of-queries-over-time), it is recommended that you store those queries separately.

You can [extract queries](https://cloud.ibm.com/apidocs/discoverysearch-the-query-and-event-log) from Discovery, however a maximum of 10,000 queries are stored. There is no paging API. Restoring queries is not recommended; we recommend starting from scratch.





 Feedback/clicks 

If you are storing clicks in the form of feedback events, there is currently no easy way to back up and restore this information. The recommendation is to start from scratch with the [clicks/feedback data](https://cloud.ibm.com/apidocs/discoverycreate-event) API.





 Expansion lists 

If you are using expansions for query modification, back up your expansion list, and store it locally.
Procedure 

To get analyzed documents from a Watson Explorer Content Analytics collection, follow these steps:



1. Open the Content Analytics administration console in a web browser.
2. On the Collections view, expand the collection that you want to export documents from. In the Parse and Index pane, ensure that the parse and index process is running, and then click the arrow icon for Export analyzed document content and metadata.
3. In the Analyzed document export options area, select Export documents as XML files, select the Enable CAS as XMI format export check box, specify the output path for where the exported data is to be written, and click OK.
4. Stop and restart the parse and index services for the collection, and then do one of the following steps:



* If the collection already contains indexed documents that you want to use for training the machine learning model in the document cache, restart a full index build.
* If the collection does not contain indexed documents that you want to use for training the machine learning model, upload documents, configure at least one crawler to crawl the documents, and start the crawler.



5. In the Export area, check the status of the export request. The progress indicates how many documents are exported.
6. Go to the output folder that you specified when you configured export options. When documents are exported as XML files, the output folder name is based on the time stamp when the export occurs. The output folder contains XMI files (.xmi) and the UIMA TypeSystem descriptor file (exported_typesystem.xml).







 What to do next 

You must define a mapping between the UIMA types and Knowledge Studio entity types. You must also create a .zip file that contains all the files that are required to upload the analyzed data into a Knowledge Studio workspace.

Related information:



* [Exporting crawled or analyzed documents](https://www.ibm.com/support/knowledgecenter/SS8NLW_11.0.0/com.ibm.discovery.es.ad.doc/iiysatexportca.htm)