vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Select the project you just created from the list of projects, and then click Configure.





 Data collection creation example 

For example, you might have a JSON file like this one:

{
"Title": "About",
"Shortdesc": "IBM Watson Assistant is a cognitive bot that you can customize for your business needs, and deploy across multiple channels to bring help to your customers where and when they need it.",
"Topics": "overview",
"url": "https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-index"
}

If you upload a JSON file that contains repeating name values, then only the first occurrence of the name and value pair is indexed and returned by search. Break up the file into multiple JSON files and upload the set.







 Configure the search 



1. From the Discovery for IBM Cloud Pak for Data instance, click Back to Watson Assistant.

If you are not returned to the Watson Assistant user interface, then go to the Watson Assistant instance page, launch the tool again, and then return to your skill.
2. On the Watson Assistant search skill page, click Configure.
3. Choose the Discovery collection fields from which you want to extract text to include in the search result that is returned to the user.

The fields that are available differ based on the data you ingested.

Each search result can consist of the following sections:



* Title: Search result title. Use the title, name, or similar type of field from the collection as the search result title. The field title is used by default.
* Body: Search result description. Use an abstract, summary, or highlight field from the collection as the search result body. The field text is used by default.
* URL: This field can be populated with any footer content that you want to include at the end of the search result. When you use a web crawl data collection, metadata.source.url is specified by default.

For example, you might want to include a hypertext link to the original data object in its native data source.
For more information about the elements, see [Understanding contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schema).





 Content Mining 

Facets based on the Part of Speech enrichment are shown.



1. To analyze your data, open the Content Mining application. Click Launch application.



For more information, see [Analyzing your data](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contentminerapp).





 What to do next 



* For more information about how to enrich your documents so that you can find key information, see [Choosing enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).
* To explore ways to improve the query results, see [Improving your query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-improvements).
[Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Add a collection that connects to an external data source or contains uploaded files. [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Run test queries to assess the quality of the initial results. [Previewing the default query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-results) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Take actions to improve your results. [Improving your query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-improvements) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Connect your project to a virtual assistant that is built with Watson Assistant.
Data analysis methods 

Use tools from the Content Mining application to analyze your data.

You can analyze your data in the following ways:



* [Look for relevant keywords](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-relevance)
* [Find trends](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-trends)
* [Identify anomalies in cyclical patterns](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-topic)
* [Find characteristic words](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-characteristic-words)
* [Analyze relationships](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-facets-compare)
* [Analyze relationships between many facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-facets-network)



As you review the results of your analysis, you can flag documents that you want to research further later. For more information, see [Flagging documents](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-datacm-flags).

When you find important insights, you can take a snapshot of the view, and then add it to a report to share with others. For more information, see [Creating a report](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-report).



 Start your analysis 

Use the content mining application to analyze documents in your collection based on the document text and any annotations or enrichments that are stored in the documents.

To start your analysis, complete the following steps:



1.
Dialog building tips 

Get tips about ways to address common tasks.



 Adding nodes 



* Add a node name that describes the purpose of the node.

You currently know what the node does, but months from now you might not. Your future self and any team members will thank you for adding a descriptive node name. And the node name is displayed in the log, which can help you debug a conversation later.
* To gather the information that is required to perform a task, try using a node with slots instead of a bunch of separate nodes to elicit information from users. See [Gathering information with slots](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-slots).
* For a complex process flow, tell users about any information they will need to provide at the start of the process.
* Understand how your assistant travels through the dialog tree and the impact that folders, branches, jump-tos, and digressions have on the route. See [Dialog flow](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-builddialog-build-flow).
* Do not add jump-tos everywhere. They increase the complexity of the dialog flow, and make it harder to debug the dialog later.
* To jump to a node in the same branch as the current node, use Skip user input instead of a Jump-to.

This choice prevents you from having to edit the current node's settings when you remove or reorder the child nodes being jumped to. See [Defining what to do next](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-overviewdialog-overview-jump-to).
* Before you enable digressions away from a node, test the most common user scenarios. And be sure that likely digressed-to nodes are configured to return. See [Digressions](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtimedialog-runtime-digressions).







 Adding responses
Table Understanding 701db916-fc83-57ab-0000-000000000012 All supported languages 



For more information about all of the supported languages, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).





 Apply enrichments by using the API 

To apply an enrichment by using the API, complete the following steps:



1. Determine the base URL for the endpoint and the token or API key for your deployment.

For more information, see [Building custom applications with the API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-api-use).
2. Get the project ID for your project.

From the product user interface, go to the Integrate and Deploy > View API information page, and then copy the project ID.
3. If you don't know the ID of the collection that you want to apply the enrichment to, get a list of your collections to find it.

For example:

GET $authentication $url/v2/projects/$project_id/collections?version=2019-11-22

The collection_id is returned.
4. Send a GET request to return the configuration of the collection that lists the applied enrichments.

For example:

GET $authentication $url/v2/projects/$project_id/collections/$collection_id?version=2019-11-22

For the enrichment IDs for the prebuilt enrichments, see [Enrichment IDs](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichmentsenrichments-ids).
5. Add the enrichment that you want to apply.

For example, to add the Keywords enrichment, you can include the enrichment in the enrichments list. First, get its ID from the table.

The Keywords enrichment ID is 701db916-fc83-57ab-0000-000000000018.
[External link icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/test-doc4.html)

In some browsers, the link open in a new window instead of saving locally. If this occurs, select Save As in your browser's File menu to save a copy of the file.





 Step 1: Quick tour of the Discovery data schema 

Let's start out by getting to know the Discovery JSON. To understand how to build a query using the Discovery Query Language, it helps to be familiar with the JSON produced by Discovery after it enriches the documents in your collection.



1. [Launch the Discovery tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-startedlaunch-the-tooling). On the Manage data screen, choose the IBM Press Releases collection.
2. Review the insights Watson discovered in your enriched documents.



* Sentiment Analysis displays the percentage breakdown of documents tagged as positive, neutral, and negative discovered by the Sentiment Analysis enrichment.
* Entity Extraction displays persons, places, and organizations discovered in your documents by the Entity Extraction enrichment.
* Category Classification displays the hierarchical taxonomies discovered in your documents by the Category Classification enrichment.
* Concept Tagging displays the concepts discovered in your documents by the Concept Tagging enrichment.



3. To get familiar with the data schema of your documents, let's look at the View data schema screen. It displays the fields and values in your transformed documents two ways: by document (Document view), or by field (Collection view). Collection view displays all fields in your collection.

Click the View data schema icon. In the Collection view, under enriched_text, you can examine the enrichments you applied to your collection. Click on categories, concepts, entities, and sentiment to see how your collection was enriched with Watson insights.
[A sample dialog tree with example content](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/dialog-depiction-full.png)

The dialog tree in this diagram contains two root dialog nodes. A typical dialog tree would likely have many more nodes, but this depiction provides a glimpse of what a subset of nodes might look like.



* The first root node conditions on an intent value. It has two child nodes that each condition on an entity value. The second child node defines two responses. The first response is returned to the user if the value of the context variable matches the value specified in the condition. Otherwise, the second response is returned.

This standard type of node is useful to capture questions about a certain topic and then in the root response ask a follow-up question that is addressed by the child nodes. For example, it might recognize a user question about discounts and ask a follow-up question about whether the user is a member of any associations with which the company has special discount arrangements. And the child nodes provide different responses based on the user's answer to the question about association membership.
* The second root node is a node with slots. It also conditions on an intent value. It defines a set of slots, one for each piece of information that you want to collect from the user. Each slot asks a question to elicit the answer from the user. It looks for a specific entity value in the user's reply to the prompt, which it then saves in a slot context variable.

This type of node is useful for collecting details you might need to perform a transaction on the user's behalf. For example, if the user's intent is to book a flight, the slots can collect the origin and destination location information, travel dates, and so on.







 Ready to get started? 

For more information, see [Creating a dialog](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overview).
The project must have at least one collection with documents that are representative of your domain data.
2. From the Improvement tools panel of the Improve and customize page, expand Teach domain concepts, and then click Extract entities.
3. Click New.

If you want to create an entity extractor that is based on the entity type system from a IBM Watson® Knowledge Studio corpus, click the arrow, and then choose Import a Knowledge Studio corpus. For next steps, see [Importing a Knowledge Studio corpus](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-import-wks).
4. Add an extractor name and optionally a description.

This name is used as the model name and as the name of the enrichment that is created when you publish the model. The name is displayed as the enrichment name in the Enrichments page where you and others can apply it to collections. It also is displayed as the model name in the JSON representation of documents where custom entities are found. The name is stored with the capitalization and spacing that you specify.
5. Choose a collection with documents that are representative of your domain data.
6. Choose fields from the document to show in the document view where you will label documents from the collection.



* Document title is shown in the page header as the document name. Choose a field that has a unique value per document, such as the file name, which is stored in the extracted_metadata.filename field.
* Document body is where you label entity examples. Choose a field that contains the bulk of the document content, such as the text field.



Zoom

![Shows that PP3.docx is the document title and the main text panel shows the body field.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/ee-title-body.png)

Figure 2. Label documents page
7. Click Create.
[Shows the data source options](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-web-crawl-0.png)

Figure 19. Data source options
2. In the Collection name field, add FRED papers.

Zoom

![Shows the Web crawl name field](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-web-crawl-1.png)

Figure 20. Web crawl connector
3. In the Starting URLs field, add the following URL:

https://research.stlouisfed.org/wp

Zoom

![Shows the Starting URLs field](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-convo-web-crawl-2.png)

Figure 21. Starting URL field

You will add only one starting URL. In a real scenario, you might add multiple URLs that go to other pages with information about the same topic. By adding more URLs, you can expand the breadth of the expertise of your assistant.
4. Click Add.
5. Click the Edit icon for the URL that you just added.

Zoom

!
The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.



 Prerequisite steps for prior releases 

If you are using version 2.2.1 or earlier, then you must complete these extra steps before you can connect to the data source:



* Obtain a web services package from your Discovery cluster. This web services package is a custom module that the crawler uses to obtain the necessary information to crawl successfully. For more information, see [Get the web services package](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cp4dconnector-sharepoint-onprem-cp4d-prereq1).
* Deploy the web service package on the SharePoint server. For more information, see [Deploy the web services on the SharePoint server](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cp4dconnector-sharepoint-onprem-cp4d-prereq2).







 Get the web services package 

To obtain the web services package from your Discovery cluster, complete the following steps:



1. Log in to your Discovery cluster.
2. Enter the following command to obtain your crawler pod name:

oc get pods | grep crawler

You might see output that is similar to the following message:

wd-discovery-crawler-57985fc5cf-rxk89 1/1 Running 0 85m
3. Enter the following command to obtain the ESSPSolution.wsp file, replacing {crawler-pod-name} with the crawler pod name that you obtained in the previous step:

oc exec {crawler-pod-name} -- ls -l /opt/ibm/wex/zing/resources/ | grep ESSPSolution

You might see output that is similar to the following message:
Select the html field from the field list.

Choose the field that contains HTML representations of the tables.



After the enrichment is applied, you can get valid results when you submit queries that require Discovery to find information that is stored in tables.

A developer can query tables by using the API. For more information, see [Query parameters](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameterstable_retrieval).

For more information about how to apply the table understanding enrichment by using the API, see [Applying enrichments by using the API](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-enrichmentsenrichments-api-task).



 Working with tabular data in Python 

Use [Text Extensions for Pandas](https://text-extensions-for-pandas.readthedocs.io/en/latest/io.htmlmodule-text_extensions_for_pandas.io.watson.tables), an open-source library from IBM, to read the tables that were parsed from documents in Discovery into pandas DataFrame objects. A pandas DataFrame is an object that represents two-dimensional tabular data in a form that can be transformed and manipulated for downstream analysis in Python.

For example, you can extract content from tables in many annual report documents and reconstruct it into a single table that includes multiyear data points of interest. For more information, read the [Structured Information Extraction from Tables in PDF Documents with Pandas and IBM Watson](https://medium.com/ibm-data-ai/structured-information-extraction-from-tables-in-pdf-documents-with-pandas-and-ibm-watson-fac302fd25bd) blog post on Medium.com.







 Output schema 

The output schema from the Table Understanding enrichment is as follows.

{
"tables": [
{
"location" : {
"begin" : int,
"end" : int
},
"text": string,
"section_title": {
"text": string,
"location": {
The table understanding enrichment must be applied to your collection for information from tables to be searchable. The table understanding enrichment is applied to collections automatically in some situations. If it isn't and your collection has an HTML field in its index, you can apply the table understanding enrichment yourself.

For more information, see [Understanding tables](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables).





 Information from diagrams is not represented 

Text from diagrams and other images is not captured unless you enable the optical character recognition (OCR) setting for the collection. You can apply the setting to a collection after its initial creation. For more information, see [Managing data collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-collections).





 Search does not recognize significant terms 

If the results suggest that keywords, common nouns, or domain-specific terms in the query are not being recognized as significant, enrich your collection.

Use Watson Natural Language Understanding to find and tag terms that are generally understood to have special meaning, such as locations or company names. For more information, see [Applying prebuilt enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu).

Teach Discovery about terms and patterns that have special meaning to your use case. For more information, see [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain).





 Default facets aren't useful 

You can add facets that categorize documents based on data from enrichments that you apply to a collection. For example, you might want to show facets based on keywords or dictionary categories. For more information, see [Facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facets).





 Explore other search features 

When you test your project from the Discovery user interface, you submit a natural language query. Search features are available that you can enable to influence how the natural language query search is done.
Figure 1. Watson Discovery Content Mining application home page

From this starting point, you can determine other ways to filter the data that might be useful.

If your data consists of traffic reports, for example, the Part of Speech facet might show that high frequency keywords include terms such as engine, brake, fire, smoke, and spark. Given this common terminology, you can create dictionaries to help you categorize and filter the data. The keywords from the example might lead you to create the following dictionaries:



* component dictionary for terms such as engine and brake
* phenomenon dictionary for terms such as fire, smoke, and spark



When you apply the dictionary enrichment to your data, it generates annotations. You can think of annotations as tags that you add to words or phrases, where the tag categorizes or identifies the meaning of the word or phrase. The resulting annotations function as new facets that you can use to filter and dissect your data further.

With your new component and phenomenon facets, for example, you can look for correlations between components and phenomena that are involved in traffic incidents.

[Learn about the ways that you can analyze your data](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-analyze-data).





 Digging deeper 

To dig even deeper into your data, apply or create AI models that can find different types of information in your documents. You can apply built-in natural language processing models, such as the Entities enrichment that can recognize mentions of commonly known things, such as business or location names and other types of proper nouns. Or you can apply a custom model that recognizes terms and categories that are unique to your data.

[Extend your analysis by adding your own facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-add-facets).





 Getting started 

Before you can use the application, you must create a Discovery Content Mining project. After the project is created and data is uploaded, you can open the Content Mining application.