vania-janet/multiturn-rag-retrieval-data huggingface dataset:
* For more information about how to create a Content Mining project, see [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects).
* For more information about how to apply enrichments to a collection in the Content Mining application, see [Applying the annotator](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-custom-annotatorcm-custom-annotator-deploy).







 Importing dictionaries from Watson Explorer Analytical Components 

You can import [user dictionaries](https://www.ibm.com/support/knowledgecenter/en/SS8NLW_12.0.0/com.ibm.discovery.es.ad.doc/iiysatauserdict.html) from IBM Watson Explorer Analytical Components.

The default file location and name for dictionaries that are saved in Watson Explorer Analytical Components is ${primary_server_node}/{primary_configuration}/{collection_ID}/{dictionary_name}.fdic.xml.



1. Download your user dictionaries from Watson Explorer Analytical Components.
2. From your Discovery Content mining project, open the Content Mining application.
3. From the analysis view of your collection, click the Collections link in the breadcrumb to open the Create a collection for your analytics solutions page of the Content Mining application.
4. To create an annotator, click collection, and then select custom annotator from the list.

Zoom

![Shows the collection menu](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-collection-menu.png)

Figure 1. Collection menu
5. Click Create custom annotator.
6. Name your annotator, and then optionally add a description.
7. From the Annotator Type menu, select Dictionary, and then click Next.
8.
(https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/dr-search.png)

Figure 1. Search bar with search results

If pinpointing answers is your goal, complete the steps that are listed in the following table.



Checklist for getting answers

 Step Task Related information 

 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Create a Document Retrieval project. [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Add up to 5 collections that connect to external data sources or contain uploaded files. [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Run test queries to assess the quality of the initial results. [Previewing the default query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-results) 
 !
"nodes_visited_details":
{
"dialog_node": "node_1_1547675028546",
"title": "order drink",
"user_label": "I'd like to order a drink.",
"conditions": "order_drink"
}
]
},
"source_dialog_node": "root"
},
{
"label": "I need a drink refill.",
"value": {
"intents":
{
"intent": "refill_drink",
"confidence": 0.2529746770858765
}
],
"entities": ],
"input": {
"suggestion_id": "6583b547-53ff-4e7b-97c6-4d062270abcd",
"text": "I need a drink refill"
}
},
"output": {
"text":
"I'll get you a refill."
],
"generic":
{
"response_type": "text",
"text": "I'll get you a refill."
}
],
"nodes_visited_details":
{
"dialog_node": "node_2_1547675097178",
"title": "refill drink",
"user_label": "I need a drink refill.",
"conditions": "refill_drink"
}
]
},
"source_dialog_node": "root"
}
]
}
],
},
"user_id": "faf4a112-f09f-4a95-a0be-43c496e6ac9a"
}
Show more
(https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-manage-collection.png)

Figure 3. Collections page in the Sample project
2. Click Sample Collection.

The Activity page is displayed. This page shows the status of the collection. For example, it shows the total number of documents and when it was last updated. If Discovery encounters a problem when a document is uploaded or a data source is crawled, any associated messages are displayed here.

Zoom

![Shows the Activity page for the Sample Collection.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-sample-activity-tab.png)

Figure 4. Activities page in the Sample project

After you create a collection, you can come to this page to find information about the processing status of the data in the collection.
3. Click the Enrichments tab.

The Enrichments page shows you a list of available enrichments. Enrichments make meaningful information easier to find and return in searches. You can apply built-in enrichments to your collection to leverage powerful Natural Language Understanding models that tag terms, such as commonly known keywords.

Zoom

![Shows the Enrichments page for the Sample Collection.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-sample-enrichments.png)

Figure 5. Enrichments page of the Sample project

The following enrichments are applied to the sample collection:

Entities
: Recognizes proper nouns such as people, cities, and organizations that are mentioned in the content.

Part of Speech
For the steps to perform ingestion, see the [Ingesting data through the command-line interface (CLI)](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-ingest_data_cli)





 Step 7: Querying data 

In this section of the tutorial, you learn how to navigate to the Query workspace, and create SQL queries to query your data from the bucket.

To run the SQL queries, do the following steps:



1. From the navigation menu, select SQL. The Query workspace page opens.
2. Select the Presto engine from the Engine drop-down.
3. Select a catalog, for example, default schema, and a table, for example, order_detail, to run the query.
4. Click the overflow menu and select the required query.



* For a catalog and schema, you can run the Generate Path query.
* For a table, you can run the Generate path, Generate SELECT, Generate ALTER, and Generate DROP query.
* For a column, you can run the Generate path, Generate SELECT, and Generate DROP query.



Consider the following sample query to view the details from the table:

Example:

!/bin/bash
SELECT * FROM "iceberg-beta"."default"."order_detail" LIMIT 10;
5. Click Run on to run the query.
6. Select Result set or Details tab to view the results. If required, you can save the query.
7. Click Saved queries to view the saved queries.
8. Click [Explain](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-explain_sql_query) to view the logical or distributed plan of execution for a specified SQL query.







 Step 8: Keep exploring 



1. Explore the other tutorials in the documentation.
2. Monitor the promo code consumption to decide whether to buy, build on (default), decline or manually delete your instance.
The tasks in this topic describe how to use the application in guided mode.





 Look for relevant keywords 

To analyze keyword relevance, complete the following steps:



1. From the initial search page, submit a keyword search to filter the documents.
2. From the search results page in guided mode, click Analyze cause or characteristics.

After the characteristic words pane, a pane with relevancy information for each facet type is displayed.

Zoom

![Facet relevancy graphs](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-facet-relevancy.png)

Figure 2. Facet relevancy

Each relevancy pane shows a list of the keywords that occur in the documents that match the facet type.



The Count column shows the number of documents in the current result set that contain the keyword. The Relevancy column shows the level of uniqueness of the frequency count compared to other documents that match your query. High relevancy values are shown in shades of color with increasing intensity. The color begins at yellow, then increases to orange, and then to red.





 Find trends 

Use Trends analysis to find trends in your data. For example, you might see that a new product release aligns with an uptick in customer interest. Or that a new customer care approach is followed by an increase in customer satisfaction.

Your documents must contain at least one date field for trend information to be available.

To find trends, complete the following steps:



1. From the initial search page, enter a keyword or select a facet with number values to filter the documents.
2. Click Find trends and anomaly from the list of suggested next steps that is displayed in the guided mode view.

The resulting bar graph shows the number of documents that mention the term or facet value that you specified in the search query over time.

Zoom

!
How your data is processed 

When you connect to a data source, Discovery processes the information from the data source to create a collection.

The goal of processing a data source is to identify meaningful information and tag it as it is added to the collection so it is easier to find and retrieve the information later.

The processing that is applied to all data sources includes the following steps:



* Identify individual documents in the data source
* Find fields in the documents
* Index the fields



You can see a list of the fields that were indexed from the Manage fields page.



1. Go to the Manage collections page, and then choose the collection.

Make sure that the processing of the collection is finished first. The Activity page shows the processing status.
2. Click the Manage fields tab.



The fields that are shown can differ based on your data. However, one subset of fields is always listed. These fields, with names such as footer and header, are derived from the Smart Document Understanding (SDU) tool, and are listed even when you don't explicitly apply an SDU model to the collection. (For the full list of SDU-generated fields, see [Available fields](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-default-fields).) Only the fields with a data type specified are stored in the collection's index.

One of the SDU-generated fields that is stored in the index is the text field. The text field typically contains the main body of text from the original document. Most of the content that is returned in search results that you submit from the Improve and customize page originates from this one field. How to parse and return only relevant chunks of information from this field is determined by the query result configuration that is used by the project. For more information, see [Previewing the default query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-results).

More processing adds more fields. And more processing is applied automatically depending on the project type.
After you choose a data collection type, the collection field values are prepopulated with source fields that are considered most likely to contain useful information given the collection's data source type. However, you know your data better than anyone. You can change the source fields to ones that contain the best information to meet your needs.

To learn more about the structure of the documents in your collection, including the names of fields that contain information you might want to extract, open the collection in Discovery.

The source fields are created when the collection is created. To learn more about fields that are generated for you, such as enriched_text.concepts.text, see [Creating enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-create-enrichments).







 Troubleshooting 

Review this information for help with performing common tasks.



* Creating a Web crawl data collection: Things to know when you create a web crawl data source:



* To increase the number of documents that are available to the data collection, click add a URL group where you can list the URLs for pages that you want to crawl but that are not linked to from the initial seed URL.
* To decreate the number of documents that are available to the data collection, specify a subdomain of the base URL. Or, in the web crawl settings, limit the number of hops that Watson can make from the original page. You can specify subdomains to explicitly exclude from the crawl also.
* If no documents are listed after a few minutes and a page refresh, then make sure that the content you want to ingest is available from the URL's page source. Some web page content is dynamically generated and therefore cannot be crawled.



* Configuring search results for uploaded documents: If you are using a collection of uploaded documents and cannot get the correct search results or the results are not concise enough, consider using Smart Document Understanding when you create the data collection.

This feature enables you to annotate documents based on text formatting. For example, you can teach Discovery that any text in 28-point bold font is a document title.
For example, to view the logs of a peer node simply add the text pod:<pod_name_of_peer_node> to the search input box, replacing <pod_name_of_peer_node> with the name of the peer node pod. The pod name of your node can be obtained by using kubectl commands such as kubectl get pod, or it is visible in your Kubernetes dashboard.

kubectl get po

The output looks similar to: NAME READY STATUS RESTARTS AGE ibp-operator-6558fdc54f-c5zmn 1/1 Running 0 65m org1ca-74b8f49fb6-z8gvx 1/1 Running 0 60m peerorg1-5765d88c85-jglk2 4/4 Running 0 25m

In the IBM Log Analysis search box, prepend the name of the pod with the text pod: such that the resulting search string in the tracking dashboard resembles:

pod:peerorg1-5765d88c85-jglk2

This same process can be used to view the logs of ordering nodes, peers and CAs. Simply add the pod name of the node to the search bar as shown in the following screenshot:

Zoom

![Filtering on node](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/logDNAPod.png)

Figure 1.Filtering logs by node

If you are not getting any results, you might need to filter on your namespace for the blockchain nodes to be visible in the Kubernetes dashboard. To find the namespace, open any Certificate Authority tile in your blockchain console and click the Info and usage tab. View the value of the API URL. For example: https://nf85a2a-znorg10524.ibpv2-cluster.us-south.containers.appdomain.cloud:7054.
* [Contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schema)
* [Table Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables)



For more information about how to create custom enrichments, see [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).

For more information about how to get the most from enrichments, read the [Enriching your documents can make search more effective](https://community.ibm.com/community/user/watsonai/blogs/bill-murdock1/2022/01/14/enriching-your-documents-can-make-search-more-effe) blog post.

For more information about how to apply enrichments by using the API, see [Applying enrichments by using the API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichmentsenrichments-api-task).





 Add enrichments 

To add an NLP enrichment, complete the following steps:



1. Open your project and go to the Manage collections page.
2. Click to open the collection that you want to enrich.
3. Open the Enrichments tab.
4. Scroll to find the NLP enrichment that you want to apply to your documents.

Both built-in enrichments and custom enrichments are listed. Built-in enrichments have a type value of System.
5. Choose one or more fields to apply the enrichment to.

You can apply enrichments to the text and html fields, and to custom fields that were added from uploaded JSON or CSV files or from the Smart Document Understanding (SDU) tool.
6. Click Apply changes and reprocess.



Enrichments that you enable are applied to the documents in random order.
[More icon](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon.



* To create a peer node that is checked next if the condition for the existing node is not met, select Add node below.
* To create a peer node that is checked before the condition for the existing node is checked, select Add node above.
* To create a child node to the selected node, select Add child node. A child node is processed after its parent node.
* To copy the current node, select Duplicate.



For more information about the order in which dialog nodes are processed, see [Dialog overview](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-builddialog-build-flow).
10. Test the dialog as you build it.

See [Testing your dialog](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-build-test) for more information.







 Conditions 

A node condition determines whether that node is used in the conversation. Response conditions determine which response to return to a user.



* [Condition artifacts](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-condition-artifacts)
* [Special conditions](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-special-conditions)
* [Condition syntax details](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-condition-syntax)



For tips on performing more advanced actions in conditions, see [Condition usage tips](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-tipsdialog-tips-condition-usage).



 Condition artifacts
This documentation is for the classic Watson Assistant experience. To see the documentation for the new Watson Assistant, please go [here](https://cloud.ibm.com/docs/watson-assistant).



 Conversation building tips 

Get tips about ways to address common tasks.



 Adding nodes 



* Add a node name that describes the purpose of the node.

Today, you know what the node does, but months from now you might not remember. Your future self and any team members will thank you for adding a descriptive node name. And the node name is displayed in the log, which can help you debug a conversation later.
* To gather the information that is required to perform a task, try using a node with slots instead of a bunch of separate nodes to elicit information from users. See [Gathering information with slots](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-slots).
* For a complex process flow, tell users about any information they will need to provide at the start of the process.
* Understand how your assistant travels through the dialog tree and the impact that folders, branches, jump-tos, and digressions have on the route. See [Dialog flow](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-builddialog-build-flow).
* Do not add jump-tos everywhere. They increase the complexity of the dialog flow, and make it harder to debug the dialog later.
* To jump to a node in the same branch as the current node, use Skip user input instead of a Jump-to.

This choice prevents you from having to edit the current node's settings when you remove or reorder the child nodes being jumped to. See [Defining what to do next](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-jump-to).
* Before you enable digressions away from a node, test the most common user scenarios. And be sure that likely digressed-to nodes are configured to return.
Step 1: Create the Document Retrieval project 

Create a project. Choose to create a Document Retrieval project type. This type is optimized for finding answers that are returned as passages from large documents.

For more information about project types, see [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects).



1. From the Discovery Plus plan service page in IBM Cloud, click Launch Discovery.
2. From the My Projects page, click New Project.
3. Name your project Finance tutorial project, and then click the Document Retrieval tile.

Zoom

![Shows the project type options.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-new-project.png)

Figure 1. Project type options
4. Click Next.



You'll configure the data source for the project in the next step.





 Step 2: Upload a PDF file 

We want the search application to be able to answer questions about algorithmic trading. Therefore, we are adding the “Staff Report on Algorithmic Trading in US Capital Markets” PDF that was created on 5 August 2020 as a data source for the project.



1. Get a copy of the PDF so that you can upload it to your project. You can download the file from the [US Securities and Exchange Commission](https://www.sec.gov/tm/reports-and-publications/special-studies/algo_trading_report_2020) website.
2. From the Select data source page, click Upload data, and then click Next.

Zoom

![Shows the data source options.]
For more information, see [Use NeuralSeek to return polished answers from existing help content](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-tutorial-neuralseek).





 Find trends 

Uncover patterns, trends, and relationships in structured and unstructured data. Use text analytics to gain insights into social media, e-commerce trends, and user behavior. Or start to address problems by finding their root cause.

Only users of installed deployments (IBM Cloud Pak for Data) or Premium or Enterprise plan-managed deployments can create this type of project.

Zoom

![Shows the content mining application user interface.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/mining-search.png)

Figure 4. Analyzing data with the Content Mining application

If finding trends in your data is your goal, complete the steps that are listed in the following table.



Checklist for finding trends

 Step Task Related information 

 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Create a Content Mining project. [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Add a collection that connects to an external data source or contains uploaded files. [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections) 
 !