vania-janet/multiturn-rag-retrieval-data huggingface dataset:
To improve the quality of the search results, build a Smart Document Understanding model for this document. The model helps Discovery understand the document structure. You can then instruct Discovery about which sections of the document to search and which sections to ignore.



1. From the Improvement tools panel of the Improve and customize page, expand Define structure, and then click New fields.

Zoom

![Shows the New fields tool in the Improvement tools panel.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-new-fields.png)

Figure 8. New fields tool in the Improvement tools panel
2. The Identify fields tab is displayed, where you can choose the type of Smart Document Understanding model that you want to use.

Zoom

![Shows the Identify fields tab.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-model-options.png)

Figure 9. Identify fields tab



* The pretrained model applies a noncustomizable model that extracts text and identifies tables, lists, and sections. The pretrained model is a great choice to save time.
* For the purposes of this tutorial, where we want to explore how the Smart Document Understanding tool works, we'll choose to use the user-trained model.



If you don't choose a model, the text extraction model is applied automatically. With the text extraction model, most of the document content is treated as standard text and is indexed in the text field.
3. Click User-trained models, and then click Submit.

Zoom

![Shows the confirmation dialog after you choose the User-trained models option.]
logging.info("Completed Generating Schematic Plan")





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 985
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "POST /v1/workspaces/us-south.workspace.Schematic-Sunil-Test-Workspace.5a4cbf11/plan HTTP/1.1" 202 49
{
  "activityid": "8c2ce5f031d23a60d26690d3cb398bb1"
}
INFO:root:Completed Generating Schematic Plan







 Next steps 

After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/ibm-spectrum-symphony?topic=ibm-spectrum-symphony-applying-planapply-plan-api) to deploy your IBM Cloud resources to build a IBM Spectrum Symphony cluster instance.
1. Create a collection. See [Preparing the service for your documents](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicepreparing-the-service-for-your-documents).
2. Click on the collection to open it.
3. Click the Upload documents button and start uploading your documents via drag and drop or browse.



Your documents are now enqueued to be converted and enriched. The time this takes depends on the size of your collection. After it is indexed and enriched, the details of the collection are displayed in the Overview section.



* Fields identified in your collection.
* Created and Last updated dates (Click Use this collection in API to see the collection_id, configuration_id, and environment_id.)
* Number of documents in your collection
* Errors and Warnings



For information on how to connect to various data sources, or do a web crawl with the Discovery tooling, see [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources).





 Uploading documents with the API 

See [Getting started with the Discovery API](https://cloud.ibm.com/docs/discovery?topic=discovery-gs-api) for a step-by-step tutorial.

For more information about the API, see the [API reference](https://cloud.ibm.com/apidocs/discovery/).



1. Use the POST /v1/environments/{environment_id}/collections method to create a collection.
2. Then use the POST /v1/environments/{environment_id}/collections/{collection_id}/documents method to add documents to your collection.







 Crawling URLs 

You can crawl URLs and index them using the Web Crawl connector. See [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sourcesconnectwebcrawl).





 Troubleshooting ingestion issues 

Learn about steps you can take to mitigate issues that you might encounter when you are adding documents.
[More icon](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon.



* To create a peer node that is checked next if the condition for the existing node is not met, select Add node below.
* To create a peer node that is checked before the condition for the existing node is checked, select Add node above.
* To create a child node to the selected node, select Add child node. A child node is processed after its parent node.
* To copy the current node, select Duplicate.



For more information about the order in which dialog nodes are processed, see [Dialog overview](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-builddialog-build-flow).
10. Test the dialog as you build it.

See [Testing your dialog](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-tasksdialog-tasks-test) for more information.







 Conditions 

A node condition determines whether that node is used in the conversation. Response conditions determine which response to return to a user.



* [Condition artifacts](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-condition-artifacts)
* [Special conditions](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-special-conditions)
* [Condition syntax details](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-overviewdialog-overview-condition-syntax)



For tips on performing more advanced actions in conditions, see [Condition usage tips](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-tipsdialog-tips-condition-usage).



 Condition artifacts 

You can use one or more of the following artifacts in any combination to define a condition:
* [Building with Watson: Passage Retrieval and Relevancy Training with Watson Discovery](https://www.youtube.com/watch?v=8BiuQKPQZJk&list=PLZDyxLlNKRY_GJskIreh9sQgExJ4z8oZO&index=9&t=0s)
* [Building with Watson: Network Visualizations with Watson Discovery Service](https://www.youtube.com/watch?v=pcNwV9prfmY&list=PLZDyxLlNKRY_GJskIreh9sQgExJ4z8oZO&index=10&t=0s)
* [Building with Watson: A technical intro to the Watson Discovery Service](https://www.youtube.com/watch?v=FikHwoJ6_FE&list=PLZDyxLlNKRY_GJskIreh9sQgExJ4z8oZO&index=11&t=417s)
Figure 1. Architecture diagram of the tutorial



1. The admin uploads a CSV file from a local machine.
2. The uploaded CSV file is stored in IBM Cloud Object Storage service as a dataset.
3. The dataset is then used to build and deploy a machine learning model. The deployed model is exposed as an API (scoring-endpoint).
4. The user makes an API call to predict the outcome with the test data.
5. The deployed machine learning model is monitored for quality, accuracy and other key parameters with the test data.







 Step 1: Import data to a project 

A project is how you organize your resources to achieve a particular goal. Your project resources can include data, collaborators, and analytic tools like Jupyter notebooks and machine learning models.

You can create a project to add data and open a data asset in the data refiner for cleansing and shaping your data.



 Create a project 



1. If you do not have an existing Object Storage service, go to the [IBM Cloud® catalog](https://cloud.ibm.com/catalog) and create an instance of [Object Storage](https://cloud.ibm.com/objectstorage/create).
2. From the [catalog](https://cloud.ibm.com/catalog), create [Watson Studio](https://cloud.ibm.com/catalog/services/data-science-experience?taxonomyNavigation=app-services)



1. Select a region
2. Select a Lite pricing plan
3. Change the Service name to watson-studio-tutorial
4. Select a resource group and click Create



3. Click on the Launch in IBM Cloud Pak for Data button.
4. Create a project by clicking on New project under the Projects tile and then in the subsequent page click Create an empty project.
5. Provide iris_project as the project name and Leave the Restrict who can be a collaborator checkbox unchecked as there's no confidential data.
6. Under Storage, choose an existing Object Storage service.
7. Click Create.
Query reference 

The full list of Discovery query parameters, operators, and aggregations. You can refer to this information for help when you write queries with the Discovery Query Language.



* For more information, see the Discovery [API reference](https://cloud.ibm.com/apidocs/discovery-dataquery).
* For an overview of query concepts, see the [Query overview](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-concepts).



In the Discovery user interface, you can write and test [natural language queries](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersnlq) on the Improve and customize page.



 Parameters descriptions 

Use query parameters to search your collection, identify a result set, and analyze result sets.



Search parameters

 Parameter Description Example 

 [aggregation](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-referenceaggregations) A statistical query of the results set aggregation=term(enriched_text.entities.type) 
 [filter](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersfilter) An unranked query language search for matching documents. filter=bees 
 [natural_language_query](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersnlq) A ranked natural language search for matching documents natural_language_query="How do bees fly" 
 [query](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parametersquery) A ranked query language search for matching documents. query=bees 





Structure parameters

 Parameter Description Example 

 [count](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameterscount) The number of result documents to return. count=15
* If you use custom query expansion, [add your query expansions](https://cloud.ibm.com/apidocs/discoverycreate-or-update-expansion-list) for each collection, as well.
* If you use any custom entity models from IBM Watson™ Knowledge Studio for enrichment, [reimport that model](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicecustom-entity-model) into your Discovery instance.



After you set up your collections as it was before, begin ingesting your source documents. Depending upon how you ingested your documents previously, you can do so by using your own solution or one of the following methods:



* The [API](https://cloud.ibm.com/apidocs/discoveryadd-a-document)
* A [connector](https://cloud.ibm.com/docs/discovery?topic=discovery-sources)





 Restoring training data 

After you restore the collections, you can begin the process of [recreating your relevancy training models](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-api).

Retrieve the training data you backed up, then:



1. [Upload your queries](https://cloud.ibm.com/apidocs/discoveryadd-query-to-training-data).
2. [Upload the example documents](https://cloud.ibm.com/apidocs/discoveryadd-example-to-training-data-query) for each query.
3. After training data is uploaded, review this [blog post](https://developer.ibm.com/dwblog/2017/get-relevancy-training/) to ensure you meet the previous accuracy numbers. Keep in mind that old document ids must be transferred to the new training data, or updated to reflect the new document ids in your new collection.







 Restoring connections to external data sources 

In case of an unanticipated loss of data, you might lose your scheduled crawls of external data sources.
* [Extracting Text Patterns with User Highlights with Pattern Induction](https://towardsdatascience.com/pattern-induction-what-is-a-pattern-part-1-79ee1bd5adc6)
* [Pattern Induction: Best Practices for Extracting Text Patterns](https://maeda-han.medium.com/pattern-induction-best-practices-for-extracting-text-patterns-part-3-2c0ee6481a3c)



To define a pattern, complete the following steps:



1. From the Teach domain concepts section of the Improvement tools panel, choose Patterns.
2. Click New.
3. Pick how you want to choose documents.



* Allow Discovery to choose 10 random documents for you.
* Choose the documents yourself (up to 20 can be selected).

Each document must be under 5,000 characters in length. Documents that exceed the limit are truncated to 5,000 characters.



4. Click Next.
5. Start selecting example words or phrases that fit the pattern you want to define.

For example, if you have a collection of articles that discuss ISO standards, you might start highlighting the numbers of the standards in each document.

If you annotate something, and then change your mind, hover over the selection, and then click x to delete it.
6. Continue selecting examples.

After you identify enough examples, Discovery shows a list of suggested examples for you to review and determine to be valid or not valid examples. Suggested examples are taken from the field that is configured to be used in search results. If the source of result content is configured to be passages, the text field is used. For more information, see [Changing the result content](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-resultsquery-results-content).
7. Choose Yes or No for each suggestion.

Click the Preview document icon if you want to see the example in context before you make a choice.
8. Continue highlighting examples and validating suggestions until a message is displayed to inform you that you identified enough examples.
9.
Splitting the PDF creates two smaller files that can be enriched faster in Discovery.







 Step 2: Create a Document Retrieval project 

Now that you have the latest copy of the product documentation, add it to a Discovery project as your data source.

In Discovery, you will create a Document Retrieval project type. Documents that you add to a project of this type are automatically enriched in the following ways:



* Entities, such as proper nouns, are identified and tagged.
* Parts of speech are identified and tagged.



This tagged information is used later when a natural language phrase is submitted as a search query to return an accurate response.



1. Open a new web browser page.
2. From the Discovery Plus plan service page in IBM Cloud, click Launch Discovery.
3. From the My Projects page, click New Project.
4. Name your project Discovery documentation, and then click the Document Retrieval tile.

Zoom

![Shows the project type options](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-new-project.png)

Figure 1. Project type options
5. Click Next.



You'll configure the data source for the project in the next step.





 Step 3: Upload data to the project 

Add the documentation PDFs to your Discovery project.



1. From the Select data source page, click the Upload data tile, and then click Next.

Zoom

![Shows that the Upload data option is chosen from the data sources page](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-data-source.png)

Figure 2. Creating a collection from uploaded data
2. Name the collection Discovery docs part 1, and then click Next.
3.
For more information about which ratios to apply, see [Document set management](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-improve-mlwks_mamanagedata).
6. Click Train to train the model, or click Train & Evaluate to train the model, evaluate annotations added by the machine learning model, and analyze the performance statistics.

Training a machine learning model can take several minutes or several hours, depending on the number of human annotations that exist and the total number of words across all documents.
7. Select the document sets that you want to use for training the model.

The document sets must contain at least 10 annotated documents.
8. After the model is created, select one of the following actions:

Table 1. Document options



 Option Description 

 Log View the log file to see whether any problems occurred. 
 Details View the annotation performance statistics, change the document sets that you want to use for training and testing the model, and create snapshot versions of the model artifacts. 
 Export If you have a Standard plan or a Premium plan, you can export a ZIP file to your local system that contains the components that are required for the model to run in a machine learning runtime environment, such as Watson Explorer. 











 Evaluating annotations added by the model 

You can compare the ground truth view for annotations added by human annotators to the annotations added by the model.



 Procedures 

To evaluate the annotations added by the model:



1. Select Machine Learning Model > Performance > Train and evaluate. The Training/Test/Blind Sets page is displayed.
2. Click View Ground Truth for the training set or test set to see the annotations that were added through pre-annotation and by human annotators. The ground truth editor opens. Click to open individual documents and see how the mentions, relations, and coreferenced mentions were annotated.
3. On the Performance page, click View Decoding Results to see the annotations that the machine learning model added to documents in the test set. This button is available only after you evaluate the model.
* Complete step 1 of [Creating an advanced rules model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-advanced-rules-modelcreate-advanced-rules-model-procedure)



2. Right click somewhere in the "Projects" tab and click Import Project
3. Input a project name, check "Include Documents" if you need to, and select a ZIP file exported in Export advanced rules project above







 Export model file from project 



1. Complete step 6 of [Creating an advanced rule model](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-create-advanced-rules-modelcreate-advanced-rules-model-procedure)







 Deploy and call model file to Natural Language Understanding 



1. Refer to the Natural Language Understanding documentation for [Analyzing text with advanced rules (Beta)](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizingadvanced-rules)
* [Discovery entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-export)



2. From the Teach domain concepts section of the Improvement tools panel, and then click Import machine learning models.
3. Specify a name for the model, and then choose the language that was used to define the model.
4. Click Upload to browse for the file that you exported earlier.
5. Click Create.
6. Choose the collection and field where you want to apply the enrichments from the model, and then click Apply.



If the model is too large to upload from the product user interface, you can use the [Create an enrichment](https://cloud.ibm.com/apidocs/discovery-datacreateenrichment) method of the API to import the file.



 Rule-based model example 

For example, when a machine learning model is applied as an enrichment to a field, it extracts all entity types in that field that were specified in a Knowledge Studio rule-based model. If the model recognizes entity types such as person, surname, and job title they are recognized in your documents and tagged.

In the output, the information that is extracted by the Machine Learning enrichment in the enriched_{field_name} array, within the entities array. In this example, the field that is selected for enrichment is text.

{
"enriched_text": [
{
"entities":
{
"path": ".wksrule.entities.PERSON",
"text": "George Washington",
"type": "PERSON"
},
{
"path": ".wksrule.entities.GIVENNAME",
"text": "George",
"type": "GIVENNAME"
},
{
"path": ".wksrule.entities.SURNAME",
"text": "Washington",
"type": "SURNAME"
},
{
"path": ".wksrule.entities.POSITION",
For more information about how to get the most from enrichments, read the [Enriching your documents can make search more effective](https://community.ibm.com/community/user/watsonai/blogs/bill-murdock1/2022/01/14/enriching-your-documents-can-make-search-more-effe) blog post.



 Choosing the right enrichment type 

The following diagram helps you to choose the right enrichment for your use case.

Zoom

![If you want to tag significant information in your data, find the right enrichment to use by answering these questions: Do you want to tag terms, passages, or documents? If passages or documents, create a classifier enrichment. If terms, are the terms expressed in a finite list? If yes, create a dictionary enrichment. If not, does the term syntax follow a pattern? If so, do all of the variations of the term fit a single pattern? If so, create a regular expression enrichment. If not, create a patterns enrichment that uses term examples that you provide to find patterns in term variations. If no set of patterns can capture the terms, create an entity extractor to identify terms based on the context in which they're used.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/enrichment-flowchart-plus-text-revision.drawio.png)

Figure 1. Flow diagram for choosing the right enrichment





 Using enrichments together 

You can use many enrichments together to tackle various challenges that you might encounter as you develop a search application.

Many teams start by creating a dictionary enrichment. Dictionaries are a great tool for identifying important terms and tagging them so they can be retrieved later. Let's say you're building a search application that needs to extract ingredients from recipes. A dictionary enrichment can recognize mentions of most ingredients. However, the dictionary enrichment might partially match against two-word terms.
* The field with the longest string value and highest number of distinct values is chosen.
* If more than one field meets the previous condition, one of the fields is chosen at random.



* If you want to apply an enrichment to a nested field, you must create a Content Mining project, and then apply the enrichment to the field. If you want to use a project type other than Content Mining, you can reuse the collection that you created with the Content Mining project type elsewhere. For more information, see [Applying enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-database-cp4dconnector-database-cp4d-enrich-db).



You can specify the normalizations and conversions objects in the [Update a collection](https://cloud.ibm.com/apidocs/discovery-dataupdatecollection) method of the API to move or merge JSON fields.







 How passages are derived 

Discovery uses sophisticated algorithms to determine the best passages of text from all of the documents that are returned by a query. Passages are returned per document by default. They are displayed as a section within each document query result and are ordered by passage relevance.

Discovery uses sentence boundary detection to pick a passage that includes a full sentence. It searches for passages that have an approximate length of 200 characters, then looks at chunks of content that are twice that length to find passages that contain full sentences. Sentence boundary detection works for all supported languages and uses language-specific logic.

For all project types except Conversational Search, you can change how the passages are displayed in the search results from the Customize display > Search results page. For example, you can configure the number of passages that are shown per document and the maximum character size per passage.