vania-janet/multiturn-rag-retrieval-data huggingface dataset:
[Search icon](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/search_icon.png)
2. Enter a search term or phrase.

The first time you search, an index is created. You might be asked to wait while the text in your dialog nodes is indexed.



Nodes containing your search term, with corresponding examples, are shown. Select any result to open it for editing.

Intent search return]





 Copying a dialog node 

You can duplicate a node to create an exact copy of it as a peer node directly after it in the dialog tree. The copied node itself is given the same name as the original node, but with - copyn appended to it, where n is a number that starts with 1. If you duplicate the same node more than once, then the n in the name increments by one for each copy to help you distinguish the copies from one another. If the node has no name, it is given the name copyn.

When you duplicate a node that has child nodes, the child nodes are duplicated also. The copied child nodes have the exact same names as the original child nodes. The only way to distinguish a copied child node from an original child node is the copy reference in the parent node name.



1. On the node you want to copy, click the More(images/search_dialog.png)More icon][!!] icon, and then select Duplicate.
2. Consider renaming the copied nodes or editing their conditions to make them distinct.







 Moving a dialog node 

Each node that you create can be moved elsewhere in the dialog tree.

You might want to move a previously created node to another area of the flow to change the conversation. You can move nodes to become siblings or peers in another branch.



1. On the node you want to move, click the More!
[Shows the different ways that query and natural language query requests are handled by Discovery](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/query-flow.png)

Figure 1. Flow chart that shows the processes that are used for Natural Language Queries versus Discovery Query Language queries

The following processes are shown in the flow diagram:

BM25
: Uses Best Match 25 (a probabilistic information retrieval algorithm) to compute a relevance score for each document returned by search. The diagram shows that BM25 is applied to document results from the query requests, but it is not limited to query requests. It also is used along with other techniques as part of the relevancy training ranker process that is applied to natural language query results.

Curations
: If the natural language query matches a predefined curation query, then certain documents and possibly a hardcoded snippet are returned. There is no query parameter to enable a curation. For curations to be used, you must define them programmatically ([Create curation method](https://cloud.ibm.com/apidocs/discovery-datacreatecuration)). The output of any curations is merged with the output of the Relevancy training ranker or QPP results.

Relevancy training
: A model that you can optionally define and apply to a project to score documents for relevance. There is no query parameter to enable relevancy training. For relevancy training to be used, you must successfully train the project either programmatically ([Create training query method](https://cloud.ibm.com/apidocs/discovery-datacreatetrainingquery)) or by using the product user interface.

QPP
: A Query Performance Prediction algorithm that, given a query and a list of top results, produces a score that determines how relevant a document is. Used only if no Relevancy training ranker is available.

filter
However, the specific commands that are used in this example for creating, listing, and restoring snapshots apply only if you are running on IBM Cloud. For other cloud providers or when running in your own datacenter, follow the specific instructions for underlying snapshot creation and management found in the documentation of those providers.

When a snapshot is being taken the node continues to function as normal.



 Node snapshots 

Because the process for taking a snapshot is the same for all nodes, only one set of instructions is shown here.

After your node is provisioned, you can get the persistent volumes that are associated with it and the namespace the component is deployed into by issuing:

kubectl get pvc --all-namespaces

Which produces a list of nodes and their persistent volumes similar to this example:

NA.M.ESPACE NA.M.E STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
n3392fd orderer1node1-pvc Bound pvc-34723fe1-c8e9-4fe7-bdf8-75b8a2f22912 100Gi RWO ibmc-file-bronze 49d
n3392fd orderer2node1-pvc Bound pvc-11b34473-bfdf-4fac-962b-5999bb961a50 100Gi RWO ibmc-file-bronze 49d
n3392fd orderer3node1-pvc Bound pvc-33ed8cce-21ec-44a8-8a0f-190052b985f9 100Gi RWO ibmc-file-bronze 49d
n3392fd orderer4node1-pvc Bound pvc-5fd2b807-2361-406b-802a-a18dbaabe23e 100Gi RWO ibmc-file-bronze 49d
n3392fd orderer5node1-pvc Bound pvc-01da603e-3d2b-4b77-a0ac-af3e70e2d2f5 100Gi RWO ibmc-file-bronze 49d
After the pre-annotation is complete, create a human annotation task that includes the annotation set you created.

For more information about creating an annotation task, see [Annotation setup](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documents).
7. To view the annotations that were applied by the machine learning model to the new documents, open the annotation task.

Because the new documents were pre-annotated with the machine learning model, human annotation requires less time. For more information about adding annotations by human annotators, see [Annotating documents](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-user-guide).







 Results 

By using your machine learning model to pre-annotate new document sets, you can expedite human annotation tasks for those documents.
Enter enriched_text.concepts.text:"Cloud computing" into the Enter query here field.



3. Click Run query. There is one match ("matching_results": 1). Copy the Query URL at the top of the Summary or JSON tab to use in your application.



Under More options, you have the option to turn on passage retrieval with the Include relevant passages radio button. Passages are short, relevant excerpts extracted from the full documents that your query returns. For more information, see [Passages](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages). Passage retrieval is not available for the Discovery News collection.

If you'd like to check out a few pre-built queries, click the Use a sample query button.





 Step 3: Experiment with different queries 

Try out these queries:

To return all documents that have a positive sentiment: Click Search for Documents, Use the Discovery Query Language then:



* Click the Field drop-down and choose enriched_text.sentiment.document.label, for the Operator choose contains, then enter the Value of positive.

The query enriched_text.sentiment.document.label:positive displays under the Visual Query Builder.



To return all documents in the health and fitness category: Click Search for Documents, Use the Discovery Query Language then:



* Click the Field drop-down and choose enriched_text.categories.label, for the Operator choose is, then enter the Value of "health and fitness".

The query enriched_text.categories.label::"health and fitness" displays under the Visual Query Builder. The operator :: specifies an exact match.



To return all documents that contain the entity IBM, but not the entity Watson: Click Search for Documents, Use the Discovery Query Language then:



* Click the Field drop-down and choose enriched_text.entities.text, for the Operator choose contains, then enter the Value of IBM. Click Add rule, then for the Field choose enriched_text.entities.text, for the Operator choose does not contain, then enter the Value of Watson.

The query enriched_text.entities.text:IBM,enriched_text.entities.text:!Watson displays under the Visual Query Builder.
[checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Submit test queries to assess the results. [Testing your project](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-test) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Create a facet that surfaces the enriched data from your documents. [Facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facets) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Take actions to improve your results. [Improving your query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-improvements) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Deploy your solution. [Deploying your project](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-deploy) 







 Enhance your chatbot 

Delight your customers by fortifying your chatbot with an answer to every question. Discovery is designed to work seemlessly with Watson Assistant to search and deliver answers from help content that you already own.

Zoom
logging.info("Completed Generating Schematic Plan")





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 985
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "POST /v1/workspaces/us-south.workspace.Schematic-Sunil-Test-Workspace.5a4cbf11/plan HTTP/1.1" 202 49
{
  "activityid": "8c2ce5f031d23a60d26690d3cb398bb1"
}
INFO:root:Completed Generating Schematic Plan







 Next steps 

After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/ibm-spectrum-lsf?topic=ibm-spectrum-lsf-applying-plan&interface=api) to deploy your IBM Cloud resources to build a Spectrum LSF cluster instance.
* Add the metadata catalogs to manage your table schemas.
* Select the query engine to work with your data.



Complete the following steps:



1. In the Bucket configuration page, select Provision new IBM-managed bucket.
2. Click Next. This displays the Catalogs configuration page.
3. In the Catalogs configuration page, select a catalog.
4. Click Next. This displays the Engine configuration page.
5. In the Engine configuration page, select the Presto engine and smallest starter (1 coordinator node 1 worker node memory optimized).
6. Click Next. This displays the Summary page.
7. In the Summary page, review the configurations before you finish setting up your data infrastructure.

When the setup is complete, the watsonx.data home page is displayed. You are all set to use the watsonx.data or you can configure it further.
8. Click Finish and go. This displays the Infrastructure manager page.



Although you can add more engines, catalogs, buckets, and databases in the Infrastructure manager page if you want to, this would change the consumption rate of your promo code. When you add items in the Infrastructure manager, you can see the resource unit consumption per hour.

The promotion credits consumption begins immediately after you configure and the support services are created for your metadata. Ensure that you pause the Presto engine when it is not used. This helps in optimizing the usage credit.





 Step 6: Ingesting data 

The data files are ingested into watsonx.data using CLI. For the steps to perform ingestion, see the [Ingesting data through the command-line interface (CLI)](https://cloud.ibm.com/docs/watsonxdata?topic=watsonxdata-ingest_data_cli)





 Step 7: Querying data 

In this section of the tutorial, you learn how to navigate to the Query workspace, and create SQL queries to query your data from the bucket.

To run the SQL queries, do the following steps:



1.
For more information about Contract filter options, see [Understanding contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schemacontracts-elements).
5. You can learn more about information that is identified by the enrichments that are applied to your documents by reviewing the JSON representation of a document that is returned in a search result.

IBM Cloud



1. Click the Display options menu from the advanced view header, and then select JSON.

Zoom

![Shows the overflow menu options from the advanced text view.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/gs-display-options-menu.png)

Figure 1. Overflow menu options from the Advanced text view



IBM Cloud Pak for Data



1. Click JSON.

For a Document Retrieval for Contracts project, click the Contract Data tab. For more information about Contract filter options, see [Understanding contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schemacontracts-elements).







 Sharing your project 

Try out your project and share it with others on your team for testing purposes. A test implementation of your project is created and hosted by IBM. Use this application preview to test your search results.

To preview and share your project, complete the following steps:



1. From the Integrate and Deploy > Preview Link page, follow the instructions to give your team members access to your project. (In Content Mining projects, the page is named Share Link.)

IBM Cloud

For more information about access in IBM Cloud, see [Managing access to resources](https://cloud.ibm.com/docs/account?topic=account-assign-access-resources&interface=ui).
2. Click the copy icon for the Copy Link field to copy the URL of the preview application.
3.
For example, you might group nodes that address questions about user accounts in a User account folder and nodes that handle payment-related queries in a Payment folder.
* To group together a set of nodes that you want the dialog to process only if a certain condition is met. Use a condition, such as $isPlatinumMember, for example, to group together nodes that offer extra services that should only be processed if the current user is entitled to receive the extra services.
* To hide nodes from the runtime while you work on them. You can add the nodes to a folder with a false condition to prevent them from being processed.



These characteristics of the folder impact how the nodes in a folder are processed:



* Condition: If no condition is specified, then your assistant processes the nodes within the folder directly. If a condition is specified, your assistant first evaluates the folder condition to determine whether to process the nodes within it.
* Customizations: Any configuration settings that you apply to the folder are inherited by the nodes in the folder. If you change the digression settings of the folder, for example, the changes are inherited by all the nodes in the folder.
* Tree hierarchy: Nodes in a folder are treated as root or child nodes based on whether the folder is added to the dialog tree at the root or child level. Any root level nodes that you add to a root level folder continue to function as root nodes; they do not become child nodes of the folder, for example. However, if you move a root level node into a folder that is a child of another node, then the root node becomes a child of that other node.



Folders have no impact on the order in which nodes are evaluated. Nodes continue to be processed from first to last. As your assistant travels down the tree, when it encounters a folder, if the folder has no condition or its condition is true, it immediately processes the first node in the folder, and continues down the tree in order from there. If a folder does not have a folder condition, then the folder is transparent to your assistant, and each node in the folder is treated like any other individual node in the tree.



 Adding a folder 

To add a folder to a dialog tree, complete the following steps:



1.
If analyzing contracts is your goal, complete the steps that are listed in the following table.



Checklist for analyzing contracts

 Step Task Related information 

 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Create a Document Retrieval for Contracts project. [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Add up to 5 collections that connect to external data sources or contain uploaded files. [Creating collections](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Run test queries to assess the quality of the initial results. [Previewing the default query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-results) 
 ![checkbox](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/checkbox.png) Take actions to improve your results. [Improving your query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-improvements) 
 !
Step 6. Search logs 

When you search log data, the search applies any log filters and time queries configured in that view.

You can do simple searches (single term string search), compound search (multiple search terms and operators), field searches if the log line can be parsed, and others. For more information, see [How to Search Logs](https://docs.logdna.com/docs/search).

Note: AND and OR operators are case-sensitive and must be capitalized.





 Step 7. Create views 

Complete the following steps to create a view:



1. Click the Views icon ![Views icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/views.png).
2. Select EVERYTHING or a view.
3. Filter log data then click Save as new view.

The Create new view page opens.
4. Enter a name for the view in the Name field.
5. Optionally, add a category. Enter a name and then click Add this as new view category.
6. Optionally, attach an alert. A new section is displayed for you to configure the alert.
7. Click Save View
Setting up infrastructure and querying data 

In this tutorial, you learn to provision the IBM® watsonx.data instance with a Presto engine, catalog, and a bucket. You also explore how the watsonx.data service interacts with the data you bring in.

Sample Scenario : Your team is working on developing an automated data pipeline that requires querying data from the data bucket that you have. Your manager requests you to retrieve data from the data bucket by using the watsonx.data instance.

For this scenario, you must create the Presto query engine, establish connection, read data from the data bucket and display the result in watsonx.data instance.

This scenario assumes that you already have a data bucket that is associated with data for querying.



 Objective 



* Creating infrastructure within the watsonx.data service.
* Establishing connection with the customer data bucket.
* Querying from the bucket



Zoom

![Workflow diagram](https://cloud.ibm.com/docs-content/v1/content/4c3c4670556d1cad9f530a5c8a9edd2e63919b83/watsonxdata/images/customerbucket.svg)

Figure 1. Workflow diagram





 Before you begin 

This tutorial requires:



* Subscription of watsonx.data on cloud.
* The configuration details of data bucket that you bring in. This is required for establishing connetion with the watsonx.data.
* Ensure that the data bucket has data.







 Step 1: Provisioning watsonx.data service instance 

In this section of the tutorial, you are going to deploy a watsonx.data service, create a watsonx.data service instance from the service's catalog page or by specifying the service ID to the command line or to the API.



1. Log in to the [IBM Cloud console](https://cloud.ibm.com).
2. Click Catalog. The list of services appears.
3. To filter the list of services that are displayed, select the Databases category.
4. Click the watsonx.data tile. The service provisioning page opens.
If you use a database as your data source and want to apply enrichments to the nested fields that are indexed from the database, you must use a Content Mining project type.

If your goal is to create a search application by using a Document Retrieval project type, create a Content Mining project type first. From the Content Mining project, you can connect to the database and enrich the data. Then, you can reuse the enriched collection from a Document Retrieval project.

To enrich database content for use in a Document Retrieval project, complete the following steps:



1. Create a Content Mining project.

For more information, see [Creating a project](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects).
2. Connect to a database data source.

For more information, see [Configuring a data source: Database](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-typesdatabaseconnect).
3. Apply enrichments.

For more information, see the following topics:



* [Adding domain-specific resources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain)
* [Applying prebuilt enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu).



4. Create a Document Retrieval project.

For more information, see [Creating a project](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects).

When you are prompted to choose a collection, choose Reuse data from an existing collection. If necessary, scroll to see this option.
5. Select the collection that you created and enriched by using the Content Mining project, and then click Finish.
You can get this value from the same page as the endpoint.
* crawler name: (Optional) Name that you specified for the crawler.
* {built_connector_zip_file} is the name of the file you created in [Compiling and packaging the example connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-assemblecompile-package-connector).



For example:

bash scripts/manage_custom_crawler.sh --endpoint https://mycpd.wd40.example.com/discovery/zen40-wd/instances/1638165624521059/api --token eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCIsImtpZCI6ImVKcV9HY29NcHF5WUFJcVByZ0x0cERRZDNQcmRiTWo5TGg0X09WOEU4MlkifP.eyJ1aXQiOiIxMDAwMzMwOTk5IiwidXNlcm5hbWUiOiJhZG1pbiIsInJvbGUiOiJBZG1pbiIsInBlcm1pc3Npb25zIjpbImFkbWluaXN0cmF0b3IiLCJjYW5fcHJvdmlzaW9uIl0sImdyb3VwcyI6WzEwMDAwXSwic3ViIjoiYWRtaW4iLCJpc3MiOiJLTk9YU1NPIiwiYXVkIjoiRFNYIiwiaWF0IjoxNjQyNzAyMDA3fQ.5oymGw7pi6tAbTMW9rcdb62G95teR2-tKyznA_wjk_G698fbx1Zl73KZKyEWcTKtyX7IJ1Px5DPdophcqS9i3bPJowHy-ioVp6DML02mscZImhvZPra-e6gwUdhSB64KArmMClo1-kZG20EclNh6-oxR447Bjdsgp7IYpkmynmw0K6vPIqmzwEhr9gAK1vWLOoVd4EoiYNuxZaSFL5byJ0mnQxXzM14w3lKQHZ91WYVKc4JnuJiSVsdpGqVz1JNFmT8D9FBqJQ4uxtshnii0f1Yh-USKCbJmMPXicU8cDtJIfheBejwenfvejUTz5rgZgymYWrGvw3G2oOx_L1Yg-Q deploy -n awesome_crawler -f awesome_crawler.zip

Instead of specifying an access token for authentication, you can specify username and password parameters. For more information, see [Understanding the manage_custom_crawler.sh script](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-install-connectorccs-deploy-script).



When the custom crawler is deployed, a resource ID is assigned to the connector.





 Verifying an installed connector 

Verify that the connector has been deployed to the Discovery instance by logging into the Discovery tooling and ensuring that your connector is displayed as an option on the Configure collection page.





 Using an installed connector on Discovery