vania-janet/multiturn-rag-retrieval-data huggingface dataset:
See [Document set management](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-improve-mlwks_mamanagedata) for help determining which ratios to apply.
6. Click Train to train the model, or click Train & Evaluate to train the model, evaluate annotations added by the machine learning model, and analyze the performance statistics.

> Important: Training a machine learning model can take several minutes or several hours, depending on the number of human annotations that exist and the total number of words across all documents.
7. Select the document sets that you want to use for training the model.

> Note: The document sets must contain at least 10 annotated documents.
8. After the model is created, select one of the following actions:





Document options
Each row in this table describes one option for a choice.

 Option Description 

 Log View the log file to see whether any problems occurred. 
 Details View the annotation performance statistics, change the document sets that you want to use for training and testing the model, and create snapshot versions of the model artifacts. 
 Export If you have a Standard plan or a Premium plan, you can export a ZIP file to your local system that contains the components that are required for the model to run in a machine learning runtime environment, such as Watson Explorer. 









 Evaluating annotations added by the model 

You can compare the ground truth view for annotations added by human annotators to the annotations added by the model.



 Procedure 

To evaluate the annotations added by the model:



1. Select Machine Learning Model > Performance > Train and evaluate. The Training/Test/Blind Sets page is displayed.
2. Click View Ground Truth for the training set or test set to see the annotations that were added through pre-annotation and by human annotators. The ground truth editor opens. Click to open individual documents and see how the mentions, relations, and coreferenced mentions were annotated.
3. On the Performance page, click View Decoding Results to see the annotations that the machine learning model added to documents in the test set.
Testing and sharing your project 

As you improve your project, periodically test how enrichments and search setting changes impact the query results.

For all project types except Conversational Search, you can see the fields that are associated with an indexed document by looking at the JSON view of a document that is returned by a query. Checking the JSON structure of a document can be useful if you want to check whether certain types of information are being captured.

After you enrich your collection, you can use the JSON view of a query result to check whether your enrichments are being applied and retrieved properly. For example, you can check the JSON to confirm that a synonym that you defined in a dictionary is being tagged as an occurrence of the corresponding dictionary term.

To test your project, complete the following steps:



1. From the navigation panel, open the Improve and customize page.
2. Retrieve query results by doing one of the following things:



* Content Mining project: Choose or add a facet to apply to the documents, and then click View filtered documents. To analyze your data in more depth, open the Content Mining application in a new window or tab by clicking Launch application.
* Other project types: Enter a test query to submit or leave the field empty and press Enter to submit an empty query.



3. From the query result list, click the link to view the document.

A representation of the original document is displayed.
4.
IBM Cloud

Click Open advanced view to see useful summary information, such as the number of occurrences of any enrichments that are detected in the document.

Optional: Select an enrichment to highlight every occurrence of the element within the document text.

For a Document Retrieval for Contracts project, the Contract Data page is displayed. For more information about Contract filter options, see [Understanding contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-contracts-schemacontracts-elements).
5. You can learn more about information that is identified by the enrichments that are applied to your documents by reviewing the JSON representation of a document that is returned in a search result.

IBM Cloud



1.
Find out how Discovery is transforming data into artist insights at the 2023 GRAMMYs®. Read the [IBM Business Operations blog post](https://www.ibm.com/blogs/internet-of-things/watson-discovery-grammys) to learn more.



 Overview video 

Watch a video about how Discovery uses AI-powered search, retrieval, and content mining. This overview covers the key basics of projects, collections, fields, and enrichments. It explains how to upload your data and query for answers, find insights, and spot trends.



* Video transcript

Get started with Watson Discovery presented by David Williams - (Music intro) Welcome to Watson Discovery with AI.

In this video, we'll walk through some key concepts and show you how to get started.

Watson Discovery is made up of four main concepts, projects, collections, fields, and enrichments.

A project is a space where can import different types of data from a variety of sources, and query for insights or answers.

A collection is a set of documents that you upload or crawl from a connected data source.

As documents are crawled, unstructured text is organized into fields such as author, file type, text, and more.

And enrichments are AI capabilities that you can apply to fields to identify and extract relevant information from your documents. This helps you find answers or insights from your data.

Let's dive in to the different project types.

A document retrieval project is used to build an AI-powered search function that finds answers in your business data.

A conversational project is used to enhance your chatbot's question and answer ability.

A content mining project helps you spot trends across large volumes of text-heavy business data.

Watson Discovery supports a wide selection of data sources you can crawl, like webpages, Cloud Object Storage, Microsoft SharePoint, and more. You can even upload your own data from any data source.

After connecting and processing your data, you can apply enrichments to bring your data to life. Some commonly used enrichments are entities, contracts, and table understanding. Entities enrichment can be used to recognize people, organizations, and more. Contracts enrichment can be used to decompose contracts to fields, clauses, and relationships.
* The rating that you apply to the result that indicates whether the result is relevant or not relevant



To apply relevancy training to a project, complete the following steps:



1. Go to the Improve and customize page. On the Improvement tools panel, select Improve relevance, then Relevancy training.
2. Enter a natural language query in the Enter a question to train field.

Do not include a question mark in your query. Use the same wording as your users. For example, IBM Watson in healthcare. Write queries that include some of the terms that are mentioned in the target answer. Term overlap improves the initial results when the natural language query is evaluated.
3. Click Add+.
4. Click Rate results.
5. After the results are displayed, assess each result, and then select Relevant or Not relevant, whichever option applies given the quality of the result.

When you select Relevant, you apply a score of 10 to the result. Not relevant applies a score of 0. You can use a different scoring scale if you use the API to rate results, but you can't mix scoring scales within the same project.

If the result shows the message, “No content preview available for this document”, it means that the document that was returned does not contain a text field or that its text field is empty. If none of the documents in your collection have a text field, use the API to train the project instead of training it from the product user interface.
6. When you are finished, click Back to queries.
7. Continue adding queries and rating them.

As you rate results, your progress is shown. Check your progress to see when enough rating information is available to meet the training threshold needs. Your progress is broken into the following tasks:



* Add more queries
* Rate more results
* Add more variety to your ratings



You must evaluate at least 50 unique queries, maybe more, depending on the complexity of your data. You cannot add more than 10,000 training queries.
8. You can continue adding queries and rating results after you reach the threshold.
For example, @city checks whether any of the city names that are defined for the @city entity were detected in the user input. If so, the node or response is processed.

Consider creating a peer node to handle the case where none of the entity's values or synonyms are recognized.

For more information about entities, see [Defining entities](https://cloud.ibm.com/docs/assistant?topic=assistant-entities).
* Entity value: The node is used if the entity value is detected in the user input. Use the syntax, @entity_name:value and specify a defined value for the entity, not a synonym. For example: @city:Boston checks whether the specific city name, Boston, was detected in the user input.

If you check for the presence of the entity, without specifying a particular value for it, in a peer node, be sure to position this node (which checks for a particular entity value) before the peer node that checks only for the presence of the entity. Otherwise, this node will never be evaluated.

If the entity is a pattern entity with capture groups, then you can check for a certain group value match. For example, you can use the syntax: @us_phone.groups[1] == '617' See [Storing and recognizing pattern entity groups in input](https://cloud.ibm.com/docs/assistant?topic=assistant-dialog-tipsdialog-tips-get-pattern-groups) for more information.
* Intent: The simplest condition is a single intent. The node is used if, after your assistant's natural language processing evaluates the user's input, it determines that the purpose of the user's input maps to the pre-defined intent. Use the syntax, intent_name. For example, weather checks if the user input is asking for a weather forecast. If so, the node with the weather intent condition is processed.

For more information about intents, see [Defining intents](https://cloud.ibm.com/docs/assistant?topic=assistant-intents).
* Special condition: Conditions that are provided with the product that you can use to perform common dialog functions.
Use an abstract, summary, or highlight field from the collection as the search result body. The field text is used by default.
* URL: This field can be populated with any footer content that you want to include at the end of the search result. When you use a web crawl data collection, metadata.source.url is specified by default.

For example, you might want to include a hypertext link to the original data object in its native data source. Most online data sources provide self-referencing public URLs for objects in the store to support direct access.



You must choose a value for at least one of the search result sections.

1.5.0: If your project contains more than one data collection, choose fields that are common to all or as many of the data collections as possible.

See [Tips for collection field selection](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addskill-search-add-field-tips) for help.

If no options are available from the drop-down fields, give Discovery more time to finish creating the collection. After waiting, if the collection is not created, then your collection might not contain any documents or might have ingestion errors that you need to address first.

To continue the [example of the uploaded JSON file](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-skill-search-addskill-search-add-json-collection-example), a good mapping is to use the Title, Shortdesc, and url fields.

![Shows that the Title, Shortdesc, and url fields are selected and the preview search card is populated with information from those fields](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/search-skill-configure-fields.png)

As you add field mappings, a preview of the search result is displayed with information from the corresponding fields of your data collection.
For more information, read the [Structured Information Extraction from Tables in PDF Documents with Pandas and IBM Watson](https://medium.com/ibm-data-ai/structured-information-extraction-from-tables-in-pdf-documents-with-pandas-and-ibm-watson-fac302fd25bd) blog post on Medium.com.







 Output schema 

The output schema from the Table Understanding enrichment is as follows.

{
"tables": [
{
"location" : {
"begin" : int,
"end" : int
},
"text": string,
"section_title": {
"text": string,
"location": {
"begin" : int,
"end" : int
}
},
"title": {
"location": {
"begin": int,
"end": int,
},
"text": string
},
"table_headers" :
{
"cell_id" : string,
"location" : {
"begin" : int,
"end" : int
},
"text" : string,
"row_index_begin" : int,
"row_index_end" : int,
"column_index_begin" : int,
"column_index_end" : int
},
...
],
"column_headers" :
{
"cell_id" : string,
"location" : {
"begin" : int,
"end" : int
},
"text" : string,
"text_normalized" : string,
"row_index_begin" : int,
"row_index_end" : int,
"column_index_begin" : int,
"column_index_end" : int
},
...
],
"row_headers" :
{
"cell_id" : string,
[Search icon in the Intents page header](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/search_icon.png).
2. Enter a search term or phrase.

The first time you search, an index is created. You might be asked to wait for the text in your dialog nodes to be indexed and then resubmit your request.



Dialog nodes that contain your search term, with corresponding examples, are shown. Select a result to open it for editing.

![Intent search return](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/search_dialog.png)





 Copying a dialog node 

You can duplicate a node to create an exact copy of it as a peer node directly after it in the dialog tree. The copied node itself is given the same name as the original node, but with -copy n appended to it, where n is a number that starts with 1. If you duplicate the same node more than once, then the n in the name increments by one for each copy to help you distinguish the copies from one another. If the node has no name, it is given the name copy n.

When you duplicate a node that has child nodes, the child nodes are duplicated also. The copied child nodes have the exact same names as the original child nodes. The only way to distinguish a copied child node from an original child node is the copy reference in the parent node name.



1. On the node you want to copy, click the More![More icon](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/kabob.png) icon, and then select Duplicate.
logging.info("Completed Generating Schematic Plan")





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 1052
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "POST /v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.0f4b9089/plan HTTP/1.1" 202 49
{
"activityid": "1f4655d2f8ec00798f8a562c8e59aba7"
}
INFO:root:Completed Generating Schematic Plan







 Next steps 

After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-applying-plan&interface=api) to deploy your IBM Cloud resources to build a Slurm cluster instance.
Add meaningful names to the dialog nodes in your dialog.

When a conversation is transferred to an agent, the name of the most-recently processed dialog node is sent with it. To ensure that a useful summary is provided to service desk agents when a conversation is transferred to them, add short dialog node names that reflect the node's purpose. For example, Find a store.

Some older skills specify the purpose of the node in the external node name field instead.

![Screen capture of the field in the node edit view where you add the node purpose summary.](https://cloud.ibm.com/docs-content/v1/content/a2b18c153837bce00a5b4de496e4b4ca64bbf56e/assistant-data/images/disambig-node-purpose.png)

Every dialog branch can be processed by the assistant while it chats with a customer, including branches with root nodes in folders.
4. If a child node in any dialog branch conditions on a request or question that you do not want the assistant to handle, add a Connect to human agent response type to the child node.

At run time, if the conversation reaches this child node, the dialog is passed to a human agent at that point.



Your dialog is now ready to support transfers from your assistant to a service desk agent.





 Measuring containment 

From the Analytics page, you can measure conversation containment. Containment is the rate at which your assistant is able to satisfy a customer's request without human intervention per conversation.

To measure containment accurately, the metric must be able to identify when a human intervention occurs. The metric primarily uses the Connect to human agent response type as an indicator. If a user conversation log includes a call to a Connect to human agent response type, then the conversation is considered to be not contained.

However, not all human interventions are transacted by using a Connect to human agent response type. If you use an alternate method to deliver additional support, you must take additional steps to register the fact that the customer's need was not fulfilled by the assistant. For example, you might direct customers to your call center phone number or to an online support ticket form URL.
! for tips.

For each node that you want to make available from the disambiguation options list, complete the following steps:

<-- <ol> -->

1. Click to open the node in edit view.
2. In the external node name field, describe the user task that this dialog node is designed to handle. For example, Cancel an account.

!]
To prevent a node from being shown as a disambiguation option, remove the description from the external node name field. If you cannot remove text from the field permanently, then switch the disambiguation setting off on the Options page, and then try again to remove the text from the field.

<-- </ol> -->

<-- </ol> --><-- </section "id="section-dialog-runtime-disambig-enable" "> --><-- <section "id="section-dialog-runtime-choose-nodes" "> --> Choosing nodes Choose nodes that serve as the root of a distinct branch of the dialog to be disambiguation choices. These can include nodes that are children of other nodes. The key is for the node to condition on some distinct value or values that distinguish it from everything else.Watson Assistant can recognize intent conflicts, which occur when two or more intents have user examples that overlap. Resolve any such conflicts]] ! !! first to ensure that the intents themselves are as unique as possible, which helps your assistant attain better intent confidence scores.Keep in mind:<-- <ul> --> * For nodes that condition on intents, if your assistant is confident that the node's intent condition matches the user's intent, then the node might be included as a disambiguation option. * For nodes with boolean conditions (conditions that evaluate to either true or false), the node might be included as a disambiguation option if the condition evaluates to true. For example, when the node conditions on an entity type, if the entity is mentioned in the input that triggers disambiguation, then the node might be included.
The creation of a model is an iterative multiple-step process that involves several stages: knowledge curation, ground truth generation, model development, model evaluation, and runtime deployment.



 End-to-end domain adaptation 

The following diagram summarizes the interactions between these five stages of model development and the typical activities that occur at each stage.

![A summary of the five stages of model development and the activities that occur at each stage.](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/wks_flow.svg) Figure 2. A summary of the five stages of model development and the activities that occur at each stage.





 Knowledge curation 

This stage, which is external to Knowledge Studio, refers to the process of selecting, collecting, preserving, and maintaining content relevant to a specific domain. Curation adds value to data; it transforms data into trusted information and knowledge.





 Ground truth generation 

This stage refers to the use of Knowledge Studio tools and best practices to produce a collection of vetted data that can be used to adapt a Watson solution to a particular domain. The accuracy of this vetted data, called ground truth or gold standard documents, is critical because inaccuracies in the ground truth will correlate to inaccuracies in the applications that rely on it.

An essential part of teaching Watson about a new domain involves providing it with knowledge about entities of interest in your domain content, the relationships between them, and how the entities co-reference each other. Collecting this knowledge includes the following activities:



* Involving domain subject matter experts to create the following resources, or to identify existing resources that can be re-used or modified for your domain:



* Annotation guidelines and examples to help human annotators learn how words and passages in your domain content are to be annotated.
* Type systems that define the domain-specific types (objects) and features (data classifications) that can be discovered in your domain content through text analysis. The type system controls the types of annotations that a human annotator can add to documents.
{
"name": "resource_group_name",
"metadata": {}
},
{
"name": "resource_group_id",
"metadata": {}
}
],
"last_job": {},
"updated_at": "0001-01-01T00:00:00Z"
},
{
"command_object_name": "basic-cos-storage",
"source": {
"source_type": "git_hub",
"git": {
"git_repo_url": "https://github.com/Cloud-Schematics/blueprint-example-modules/tree/main/IBM-Storage",
"git_branch": "main"
},
"catalog": {},
"cos_bucket": {}
},
"inputs":
{
"name": "cos_instance_name",
"value": "$blueprint.cos_instance_name",
"metadata": {}
},
{
"name": "cos_storage_plan",
"value": "$blueprint.cos_storage_plan",
"metadata": {}
},
{
"name": "cos_single_site_loc",
"value": "ams03",
"metadata": {}
},
{
"name": "resource_group_id",
"value": "$module.basic-resource-group.outputs.resource_group_id",
"metadata": {}
}
],
"outputs":
{
"name": "cos_id",
"metadata": {}
},
{
"name": "cos_crn",
"metadata": {}
}
],