vania-janet/multiturn-rag-retrieval-data huggingface dataset:
![Shows how the keyword facet looks.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-keyword-facet-added.png)

Figure 19. Keywords facet



You successfully added a built-in NLU enrichment that recognizes keywords in the sample collection documents. Then, you added a facet that uses the keywords enrichment to let you filter the documents by keyword.





 Step 6: Share the sample project 



1. Click Integrate and deploy from the navigation panel.

From here, you can share your project with colleagues and deploy it.
2. Follow the on-screen instructions to add a user, and then send login credentials and the provided link to your colleague.

Zoom

![Shows the Integrate and deploy page.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-deploy.png)

Figure 20. Integrate and deploy page

After you build your own search application and are ready to deploy it, you can use prebuilt user interface components or build a custom application.



* Click API Information. From this page, you can get the project ID for your project. You need the project ID to use the Discovery API. You also need the service instance URL and API key. The credential details are available from the Manage page of your service instance in IBM Cloud.
* Click UI Components to find links to ready-to-use code that you can use to create a full-featured search application faster.









 Step 7: Add your own content 

Now that you know more about some of the product features, you're ready to evaluate the data you want to search.

It's all about the data.
In the Catalogs configuration page, complete the following steps:



1. Select the table format for managing your data. Apache Hive and Apache Iceberg catalogs are selected by default.
2. Click Next.







 Step 3: Configure an engine 

Your watsonx.data needs a query engine to work with your data.

In the Engine configuration page, complete the following steps:



1. Select the engine to run and process the data that you attached.
2. Select the size of the engine based on the requirements of your workload.



Table 2. Engine size

 Size Description 

 Starter (5.6 RUs/hour) Includes 1 coordinator node and 1 worker node. All nodes are storage-optimized. 
 Small (11.2 RUs/hour) Includes 1 coordinator node and 3 worker nodes. All nodes are storage-optimized. 
 Medium (19.6 RUs/hour) Includes 1 coordinator node and 6 worker nodes. All nodes are storage-optimized. 
 Large (36.4 RUs/hour) Includes 1 coordinator node and 12 worker nodes. All nodes are storage-optimized. 



3. Click Next.







 Step 4: Review the configuration details 

In the Summary page, complete the following steps:



1. Review the configurations before you finish setting up your watsonx.data.
2. Click Finish and go.



When the setup is complete, the watsonx.data home page appears.





 Next steps 

You are all set to use the watsonx.data or you can configure it further.
[Screenshot of the field in the node edit view where you add the node purpose summary.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/disambig-node-purpose.png)

Do not add an external node name to the root node that you created in Step 2. When an escalation occurs, your assistant looks at the external node name of the last processed node to learn which user goal was not met successfully. If you include an external node name in the node with the connect to human agent intent, then you will prevent your assistant from learning the last real, goal-oriented node that the user interacted with before escalating the issue.



4. If a child node in the branch conditions on a follow-up request or question that you do not want the assistant to handle, add a Connect to human agent response type to the node.

For example, you might want to add this response type to nodes that cover sensitive issues only a human should handle or that track when an assistant repeatedly fails to understand a user.

At run time, if the conversation reaches this child node, the dialog is passed to a human agent at that point. Later, when you set up the Intercom integration, you can choose a human agent as a backup for each branch.



Your dialog is now ready to support your assistant in Intercom.



 Dialog considerations 

Some rich responses that you add to a dialog are displayed differently within the "Try it out" pane from how they are displayed to Intercom users. The following table describes how the response types are treated by Intercom.



 Response type How displayed to Intercom users 

 Option The options are displayed as a numbered list. In the title or description field, provide instructions that explain to the user how to choose an option from the list. 
 Image The image title, description, and the image itself are rendered. 
 Pause Whether or not you enable it, a typing indicator is not displayed during the pause.
3. Select one of the available API products to display the available APIs and Plans for the product.
![Select Product](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/13-product.png)
4. Select an API to explore the details of the available APIs.
![Products APIs](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/14-api.png)
5. On the details page of an API, you can view the available operations along with their parameters and the responses returned. At the end of the page, you can view the definitions that are used by the API.
![API Details](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/15-details.png)
6. In the Code examples panel, you can view examples in different coding languages of how to invoke the requests and their responses. Select one of the examples, such as Node, to see an example in that coding language.
![API Details](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/16-examples.png)



--------------------





 View and test the APIs 

This section shows how a portal user would view and test the available APIs for a product.



1.
Providing a good search experience depends on the alignment of your users' search needs with structure in the data. Running lots of unstructured data with an indexing engine gets you only so far. If you can add further structure to unstructured data, then the search experience benefits since fewer "false positives" are returned. Look at this example:

"Edinson Cavani scored two superb goals as Uruguay beat Portugal to set up a World Cup quarter-final meeting with France. Defeat for the European champions finished Cristiano Ronaldo's hopes of success in Russia just hours after Lionel Messi and Argentina were knocked out, beaten 4-3 by Les Bleus."

Source: BBC News https://www.bbc.co.uk/sport/football/44439361

From this snippet, I would manually extract the following "entities".



* Edinson Cavani - a footballer
* Uruguay - a country
* Portugal - another country
* World Cup - a football competition
* Cristiano Ronaldo - a footballer
* Russia - a country
* Lionel Messi - a footballer
* Argentina - a country
* Les Bleus - a nickname of the French national football team



Entity extraction is the process of locating known entities (given a database of such entities) and storing the entities in the search engine instead of, or as well as, the source text. The [Watson Natural Language and Understanding API](https://www.ibm.com/watson/services/natural-language-understanding/) can be fed raw text and returns entities it knows about (you can provide your own entity model for your domain-specific application):

Zoom

![Analyzers extract information about people, companies, organizations, cities, geographic features, and other information from the content.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/images/analyzers.png)

Figure 1. Analyzers

As well as entities, the API can also place the article in a hierarchy of categories.
Your documents might not have a text field if you uploaded a CSV file that doesn't have a column named text, or uploaded a JSON file that doesn't have an object named text, or if you used the Smart Document Understanding tool to define fields with other names in which the bulk of the content of your documents now are stored.

When you train a project from the API, results are taken from all of the root-level fields and they are all considered to have equal significance. Unlike Discovery Query Language queries, with natural language queries you cannot specify which fields from the document you care about or how much significance to give to each one. When you teach Discovery with examples, the service figures out for you how much weight to give to each field.

Discovery builds a model that assigns different weights to term, bigram, and skip-gram matches for each of the root-level fields and balances them against matches from all of the other document fields. With enough examples, Discovery can return better answers because it knows where the best answers are typically stored.

Relevancy training cannot be used to give more weight to nested fields. Nested fields are grouped and assigned one overall score. No matter how much you train, Discovery never gives a nested field more weight than it gives to a root-level field. For more information about nested fields, see the [FAQ](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-faqsfaq-nested-fields).





 Training a project 

The training data that is used to train the relevancy model includes these parts:



* A natural language query that is representative of a query that your users might submit
* Results of the query which are returned by the service
* The rating that you apply to the result that indicates whether the result is relevant or not relevant



To apply relevancy training to a project, complete the following steps:



1. Go to the Improve and customize page. On the Improvement tools panel, select Improve relevance, then Relevancy training.
2. Enter a natural language query in the Enter a question to train field.

Do not include a question mark in your query. Use the same wording as your users.
This documentation is for the classic Watson Assistant experience. To see the documentation for the new Watson Assistant, please go [here](https://cloud.ibm.com/docs/watson-assistant).



 Gathering information with slots 

Add slots to a dialog node to gather multiple pieces of information from a user within that node. Slots collect information at the user's pace. Details that a user provides up front are saved, and your assistant asks only for the missing details it needs to fulfill the request.



 Why add slots? 

Use slots to get the information you need before you can respond accurately to the user. For example, if users ask about operating hours, but the hours differ by store location, you could ask a follow-up question about which store location they plan to visit before you answer. You can then add response conditions that take the provided location information into account.

![Asks for location information before answering the question, When do you open?.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/op-hours.png)

Slots can help you to collect multiple pieces of information that you need to complete a complex task for a user, such as making a dinner reservation.

![Shows four slots that prompt for the information needed to make a dinner reservation.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/reservation.png)

The user might provide values for mutliple slots at once. For example, the input might include the information, There will be 6 of us dining at 7 PM. This one input contains two of the missing required values: the number of guests and time of the reservation. Your assistant recognizes and stores both of them, each one in its corresponding slot. It then displays the prompt that is associated with the next empty slot.

!
},
{
"name": "resource_group_id",
"metadata": {}
}
],
"last_job": {}
},
{
"module_type": "terraform",
"name": "basic-cos-storage",
"source": {
"source_type": "git_hub",
"git": {
"git_repo_url": "https://github.com/Cloud-Schematics/blueprint-example-modules/tree/main/IBM-Storage",
"git_branch": "main"
},
"catalog": {},
"cos_bucket": {}
},
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"inputs":
{
"name": "cos_instance_name",
"value": "$blueprint.cos_instance_name",
"metadata": {}
},
{
"name": "cos_storage_plan",
"value": "$blueprint.cos_storage_plan",
"metadata": {}
},
{
"name": "cos_single_site_loc",
"value": "ams03",
"metadata": {}
},
{
"name": "resource_group_id",
"value": "$module.basic-resource-group.outputs.resource_group_id",
"metadata": {}
}
],
"outputs":
{
"name": "cos_id",
"metadata": {}
},
{
"name": "cos_crn",
"metadata": {}
"source": {
"source_type": "git_hub",
"git": {
"git_repo_url": "https://github.com/Cloud-Schematics/blueprint-example-modules/tree/main/IBM-Storage",
"git_branch": "main"
},
"catalog": {},
"cos_bucket": {}
},
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"inputs":
{
"name": "cos_instance_name",
"value": "$blueprint.cos_instance_name",
"metadata": {}
},
{
"name": "cos_storage_plan",
"value": "$blueprint.cos_storage_plan",
"metadata": {}
},
{
"name": "cos_single_site_loc",
"value": "ams03",
"metadata": {}
},
{
"name": "resource_group_id",
"value": "$module.basic-resource-group.outputs.resource_group_id",
"metadata": {}
}
],
"outputs":
{
"name": "cos_id",
"metadata": {}
},
{
"name": "cos_crn",
"metadata": {}
}
],
"last_job": {}
}
],
"flow": {},
"blueprint_id": "Blueprint-Basic-Test-API.soB.347a",
This documentation is for the classic Watson Assistant experience. To see the documentation for the new Watson Assistant, please go [here](https://cloud.ibm.com/docs/watson-assistant).



 Creating a dialog 

The dialog defines what your assistant says in response to customers.



 Creating a dialog 

To create a dialog, complete the following steps:



1. From the Skills menu, click Dialog.

The following nodes are created for you automatically:



* Welcome: The first node. It contains a greeting that is displayed to your users when they first engage with your assistant. You can edit the greeting.



This node is not triggered in dialog flows that are initiated by users. For example, dialogs used in integrations with channels such as Facebook or Slack skip nodes with the welcome special condition.



* Anything else: The final node. It contains phrases that are used to reply to users when their input is not recognized. You can replace the responses that are provided or add more responses with a similar meaning to add variety to the conversation. You can also choose whether you want your assistant to return each response that is defined in turn or return them in random order.



For more information about these built-in nodes, see [Starting and ending the dialog](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-start).
2. To add more nodes to the dialog tree, click Add node.

Your new node is added after the Welcome node and before the Anything else node.
3. Add a name to the node.

Use a short, customer-friendly description of what the node does as its name. For example, Open an account, Get policy information, or Get a weather forecast.

The name can be up to 512 characters in length.

This node name is shown to customers or service desk personnel to express the purpose of this branch of the dialog, so take some time to add a name that is concise and descriptive.
4. In the If assistant recognizes field, enter a condition that, when met, triggers your assistant to process the node.

To start off, you typically want to add an intent as the condition.
[Views icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/alerts/images/views.png).
2. Select Everything or a view.
3. Expand Sources to see the list of event sources that are identified in the events. Then, choose the ones that you want.
4. Expand Levels to see the list of event levels that are identified in the events. Then, choose the ones that you want. Example values are critical, warning, and normal.



3. Filter event data by adding a search query.

When you search event data, the search applies any event filters and time queries configured in that view.

You can do simple searches (single term string search), compound search (multiple search terms and operators), field searches if the event line can be parsed, and others. For more information, see [Select the set of events to display through a view by applying a search query](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-viewsviews_step2).

AND and OR operators are case-sensitive and must be capitalized.
4. Create a view.

Click the Views icon ![Views icon](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker/alerts/images/views.png).

Select Everything or a view.

Filter event data then click Save as new view. The Create new view page opens.

Enter a name for the view in the Name field.

Optionally, add a category. Enter a name and then click Add this as new view category.

Click Save View







 Step 2. Create the alert 

Next, you must attach an alert to the view.

You can choose any of the following options:
[Shows another preview of the assistant where the test question is answered](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-answer2.png)

Figure 20. Web chat returns a detailed answer
5. Optionally ask the assistant other questions.

If the assistant doesn't know the answer, reword the question to include “in Watson Discovery” to make it clearer that you are asking about how something works in Discovery specifically.



Congratulations! You successfully created an assistant that can answer questions about Discovery by retrieving information from the product documentation by way of the NeuralSeek service.



 Summary 

In this tutorial, you created a Watson Discovery Document Retrieval project with uploaded PDF files that contain the Discovery product documentation. Separately, you created a Watson Assistant virtual assistant with a single action that can recognize user questions about Discovery. You added a custom extension to your assistant that connects to a third-party service called NeuralSeek that gets the correct answer from Discovery and rewords the response. Finally, you tested your virtual assistant by asking a question and getting an accurate and well-written response.





 Next steps 

The assistant that you created is available from the draft environment. Next, you can publish your assistant to a production environment and deploy it. You can deploy the assistant in various ways. For more information, see [Overview: Previewing and publishing](https://cloud.ibm.com/docs/watson-assistant?topic=watson-assistant-publish-overview).
* [Step five: Run a script](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-run-script)
* [Hints and Tips](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-hints)
* [Next steps](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-next-steps)



* [Generated identities](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-import-identities)



* [Summary](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ansible-buildansible-build-summary)









 Monitoring your network 

[IBM Log Analysis](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking)



* [Step one: Configure cluster-level logging](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking-kubernetes)
* [Step two: View the logs for your IBM Blockchain Platform nodes](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking-ibp)



* [View node logs](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking-ibp-nodes)
* [View smart contract logs](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking-ibp-smart-contract)



* [Summary](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-trackingibp-tracking-summary)
For more about the Discovery Query Language, see [Query concepts](https://cloud.ibm.com/docs/discovery?topic=discovery-query-concepts).

All private collections return a confidence score in the query results in most cases. For more information, see [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence).

Adding a custom stopwords list can improve the relevance of results for natural language queries. See [Defining stopwords](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsstopwords) for more information.

See [Training data requirements](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-apireqs) for the minimum requirements for training, as well as the training limits.

See [Usage monitoring](https://cloud.ibm.com/docs/discovery?topic=discovery-usage) for details on tracking usage and using this data to help you understand and improve your applications.



 Adding queries and rating the relevancy of results 

Training consists of three parts: a natural language query, the results of the query, and the ratings you apply to those results.



1. There are two ways to access the training page in the Discovery tooling:



* For an individual collection, on the Build queries screen, click Train Watson to improve results on the upper right. You don't need to enter a query on the Build queries screen to start training.
* From the Performance dashboard. Click on the View data metrics icon on the left to open the dashboard. You are prompted to choose a collection to train.



2. On the Train Watson screen, click Add a natural language query, for example: "IBM Watson in healthcare", and add it. Make sure your queries are written the way your users would ask them. Also, it is recommended that training queries be written with some term overlap between the query and the desired answer. This overlap improves initial results, when the natural language query is run.
workspace_activity_apply_result = schematics_service.apply_workspace_command(
    w_id='<workspace id>',
    refresh_token='<refresh-token>'
).get_result()

print(json.dumps(workspace_activity_apply_result, indent=2))

logging.info("Completed Applying Schematic Plan")





 Example Python response 

INFO:root:Started Applying Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 985
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "PUT /v1/workspaces/us-south.workspace.Schematic-Sunil-Test-Workspace.5a4cbf11/apply HTTP/1.1" 202 49
{
  "activityid": "33bade478b4e93f5c613e4f62943ad9a"
}
INFO:root:Completed Applying Schematic Plan







 Next steps 

After you've successfully applied a plan, you can begin [Accessing the GUI](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-access-gui) to monitor the resources and status of the service for both the compute and storage clusters.