vania-janet/multiturn-rag-retrieval-data huggingface dataset:
After you specify a small number of examples, Discovery will suggest additional rules that you verify to complete the pattern. You can use pattern induction as an enrichment or to create a facet. For more information, see [Patterns](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-pattern) and [Creating a facet by identifying a pattern](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facetsfacetpattern). For a statement explaining beta features, see [Beta features](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-release-notesbeta-features).

Change to Document Retrieval projects
: In new Document Retrieval projects, the suggested refinements query setting is now set to false by default. It was previously set to true.





 14 September 2020 

New pre-trained model for SDU
: A new pre-trained model is available in Smart Document Understanding for Document Retrieval projects. This model is ideal if you need to extract data from documents that include a large number of tables. For more information, see [Identifying fields](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldsidentify-fields).





 30 August 2020 

Update to API version
: The current API version (v2) is now 2020-08-30. The following change was made with this version:

Change to 'options' object
: The List enrichments method no longer returns the options object per enrichment. Use the Get enrichment method to return the options object for a single enrichment.





 16 July 2020 

New release for Premium instances
: This release is available for Premium instances of Discovery on IBM Cloud created after 16 July 2020. For Premium instances created before that date and for all Lite and Advanced plans, see [Getting started with Discovery](https://cloud.ibm.com/docs/discovery?topic=discovery-getting-started).

Change to IBM Cloud Premium
: The Premium plan is now generally available.

New Project-based interface
logging.info("Completed Generating Schematic Plan")
Show more





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 1052
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): us.schematics.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://us.schematics.cloud.ibm.com:443 "POST /v1/workspaces/us-south.workspace.Sample-Schematic-API-workspace.0f4b9089/plan HTTP/1.1" 202 49
{
"activityid": "1f4655d2f8ec00798f8a562c8e59aba7"
}
INFO:root:Completed Generating Schematic Plan







 Next steps 

After you've successfully generated a plan, you can begin [Applying a plan](https://cloud.ibm.com/docs/hpc-slurm?topic=hpc-slurm-applying-plan&interface=api) to deploy your IBM Cloud resources to build a Slurm cluster instance.
You can get some of the text that was stored in the index when the original document was ingested by using the Query API. For more information, see [Recovering documents](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-migrate-to-v2migrate-to-v2-advanced-transfer).





 Training data 

Refer to this task to back up your training data queries and examples for a trained project. Training data is used for explicit training of your projects and is stored on a per project basis. To extract the training data, use the API to download the queries and the ratings from Discovery. To back up training data queries and examples, complete the following steps:



1. Download your training data by using the [list training queries](https://cloud.ibm.com/apidocs/discovery-datalisttrainingqueries) API.
2. Save your training queries and examples locally.



The document IDs that you use in your training data point to the documents in your current project. Use the same IDs in your new projects to ensure that the correct documents are referenced. If the IDs do not match, your restored relevancy training will not work.





 Expansion lists 

If you are using synonyms (query expansions) for query modification, back up your .json expansion list, and store it locally. For more information, see [Implementing synonyms](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settingsquery-expansion).





 Stopwords 

In the case of stopwords, back up the text file. For more information about stopwords, see [Defining stopwords](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settingsstopwords).





 Collection information 

This is not required, but it is a best practice to [retrieve the status](https://cloud.ibm.com/apidocs/discovery-datagetcollection) for each collection regularly and store the information locally. By retaining these statistics, you can later verify that your restoration processes were successful if needed.
You know the name of a collection in your project where the Keywords enrichment is configured to be applied to documents in the collection. You can use the API to [list your collections](https://cloud.ibm.com/apidocs/discovery-datalistcollections) to find the ID associated with the collection that you look for by name.

After you get the collection ID, include it in the POST request that you submit to apply the configuration settings from the collection to your JSON file. For example, the following request submits the JSON snippet in a file that is named favorites2.json for keyword analysis.

curl --location --request POST 'https://my-cloud-pak-for-data-cluster/discovery/zen-wd/instances/{instance-id}/api/v2/ projects/{project-id}/collections/{collection-id}/analyze?version=2020-08-30' --header 'Authorization: Bearer ...' --form 'file=@"/quotations/favorites2.json"'

The result contains a list of keywords that were recognized in the quotation.

{
"result": {
"enriched_Quote": [
{
"keywords":
{
"text": "day",
"mentions":
{
"text": "day",
"location": {
"begin": 10,
"end": 13
}
}
],
"relevance": 0.673739
},
{
"text": "stranger",
"mentions":
{
"text": "stranger",
"location": {
"begin": 28,
"end": 36
}
}
],
"relevance": 0.596757
},
{
"text": "parents",
"mentions":
{
"text": "parents",
"location": {
"begin": 52,
"end": 59
}
Indicate the collection to use, by doing one of the following things:



* Choose an existing project.

You can click the Open Discovery icon to review the configuration of a project before you decide which one to use.

Go to [Configure the search](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-search-addskill-search-add-configure) and follow the remaining steps.
* If you do not have a project or do not want to use any of the projects that are listed, click Create a new project to add one. Follow the steps in [Create a project](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-search-addskill-search-add-create-discovery-collection).

The Create a new project button is not displayed if you have reached the limits based on your Discovery service plan. See [Discovery pricing plans](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-pricing-plans) for plan limit details.









 Create a project 



1. On the OK, where is your data? page, select your data source, then click Next. Example choices include Salesforce, SharePoint, Box, IBM Cloud Object Storage, web crawl, and upload data.
2. On the Let's create a collection for your data pages, enter the information on how to connect to your data and configure your collection. The information you need to enter is different depending on the data source. For example, you need to enter your authentication credentials for services such as Salesforce, Sharepoint, and Box. For web crawl, you specify the website where your existing information resides.
3. Click Finish. Give Discovery a few minutes to start creating documents. You can use the Manage collections page within the project to see the progress.

After the digestion process is completed, a summary page is displayed in Discovery, which is hosted in a separate web browser tab.
4. Wait for the collection to be fully ingested, then click Back to Watson Assistant.







 Configure the search 



1.
If you are not getting any results, you might need to filter on your namespace for the blockchain nodes to be visible in the Kubernetes dashboard. To find the namespace, open any Certificate Authority tile in your blockchain console and click the Info and usage tab. View the value of the API URL. For example: https://nf85a2a-znorg10524.ibpv2-cluster.us-south.containers.appdomain.cloud:7054.

The namespace is the first part of the URL beginning with the letter n and followed by a random string of six alphanumeric characters. So in the example above the value of the namespace is nf85a2a. Then run the kubectl command:

kubectl get po -n <NAMESPACE>

Replace <NAMESPACE> with the value of your namespace. For example:

kubectl get po -n nf85a2a

The output looks similar to: NAME READY STATUS RESTARTS AGE ibp-operator-6558fdc54f-c5zmn 1/1 Running 0 65m org1ca-74b8f49fb6-z8gvx 1/1 Running 0 60m peerorg1-5765d88c85-jglk2 4/4 Running 0 25m

In the IBM Log Analysis search box, prepend the name of the pod with the text pod: and the value of the namespace. The resulting search string in the tracking dashboard would then be:

pod:nf85a2apeerorg1-5765d88c85-jglk2

In this case, nf85a2apeerorg1-5765d88c85-jglk2 represents the name of the peer pod.





 View smart contract logs 

The process to view the logs for a smart contract is similar to how you view your node logs but varies depending on whether your peer is running a Fabric v1.4 or v2.x image. The Fabric version that the peer is running is visible when you click on a peer node in the console:

Zoom

!
You can then show the details of an element in the array by including the full content of the document this time.



1. Request the first document that is retrieved from the database.

result_collection = Result(myDatabaseDemo.all_docs, include_docs=True)
print("Retrieved minimal document:n{0}n".format(result_collection[0]))
2. See the result, which is similar to the following example:

[
{
"value": {
"rev": "1-b2c48b89f48f1dc172d4db3f17ff6b9a"
},
"id": "14746fe384c7e2f06f7295403df89187",
"key": "14746fe384c7e2f06f7295403df89187",
"doc": {
"temperatureField": 10,
"descriptionField": "cold",
"numberField": 6,
"nameField": "six",
"_id": "14746fe384c7e2f06f7295403df89187",
"_rev": "1-b2c48b89f48f1dc172d4db3f17ff6b9a"
}
}
]
Show more









 Step 5: Calling an IBM Cloudant API endpoint directly 

You can work with the IBM Cloudant API endpoints directly, from within a Python application.

In this example code, you again request a list of all the documents, including their content. However, this time you do so by invoking the IBM Cloudant [/_all_docs](https://cloud.ibm.com/apidocs/cloudantpostalldocs) endpoint.



1. Identify the endpoint to contact and any parameters to supply with it.
"document_id": "bdcd6a9cc1438a3faa8c925f6a8d9429"
}
]
},
"event_type": "query",
"session_token": "1_LKczxWGEWx5TJAi1_Qs35yOoa7",
"created_timestamp": "2018-09-12T05:20:07.469Z"
}
Show more

With query logs you can investigate the type of results returned to your end users and investigate ways to improve result quality using the approaches available in Discovery. For example:



* relevancy training, see [Improving result relevance with the API](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-api) and [Improving result relevance with the tooling](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-tooling)
* [query operators](https://cloud.ibm.com/docs/discovery?topic=discovery-query-operators)
* [query expansion](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquery-expansion)
* custom configurations, see the [Configuration reference](https://cloud.ibm.com/docs/discovery?topic=discovery-configref)







 Creating event logs 

Event logs are used to track the interactions of users within your application. This can help you understand how well your application is performing, as well as identify areas that you might need to focus on to improve results. Discovery provides an API that can be embedded in your application to track events. Calling this API with the appropriate information when a user performs an action sends a signal back to Discovery, which can then be viewed in the logs.

Events can help gather information on computing metrics (like clickthrough rate) to measure how effective the application is at helping end users find relevant information, or can be used to help seed training by reviewing the queries and clicks to start to build ground truth.
* Documents in the collection must be in one language only, the language that you specify for the collection.



To create a collection, complete the following steps:



1. Open a project, go to the Manage collections page, and then click New collection.



* The Conversational Search, Document Retrieval, and Custom project types can contain up to 5 collections.
* A Content Mining project can contain only 1 collection.



2. Choose how you want to add data to your collection.



* [Uploading data](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-upload-data)
* [Reusing data from a collection](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-manage-collectionsmanage-collections-reuse)
* Crawling an external data source.

For supported data sources, see the appropriate topic for your deployment type:



IBM Cloud Pak for Data

[IBM Cloud Pak for Data data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collection-types)
IBM Cloud

[IBM Cloud data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sources)



These topics also describe how to connect to data sources that are not supported by default per deployment type.





For information about how to troubleshoot issues that you might encounter when you add documents to a collection, see [Troubleshooting ingestion](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-troubleshoot-ingestion).

For more information about how to create a collection programmatically, see the [API reference documentation](https://cloud.ibm.com/apidocs/discovery-datacreatecollection).





 Optical character recognition 

One of the optional features that you can apply to a collection when you create it is optical character recognition. The optical character recognition (OCR) feature extracts text from images. This capability is useful for preserving information that is depicted in diagrams or graphs, or in text that is embedded in files such as scanned PDFs.
To restore your external data, reestablish your connections to these data sources, and then recrawl them.

To find the data source credentials that you stored, follow the instructions for your chosen data source in [Configuring IBM Cloud data sources](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sources). These instructions explain how you can reconnect to your data sources and get the data imported into Discovery.





 Restoring Smart Document Understanding models 

To import a previously exported Smart Document Understanding (SDU) model, see [Importing and exporting models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldsimport). SDU models have the file extension of .sdumodel.

When importing an SDU existing model into a new collection, it is a best practice to create the new collection and add one document, then import the model and upload the remainder of your documents.





 Restoring dictionary enrichments 



1. Open your project.
2. On the Improvement tools panel of the Improve and customize page, click Teach domain concepts, Dictionaries, and then Upload.
3. In the Apply dictionary dialog box, enter a name for your .csv file, select a language, specify the facet path, click Upload, and select your dictionary .csv file.
4. Click Create.



After you upload dictionary .csv files for recovery, you cannot use the dictionary editor to further edit the terms. If you want to use the dictionary editor, create a dictionary, and manually add the dictionary terms.

For information about uploading a dictionary enrichment .csv file, see [Dictionary](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-dictionary).





 Restoring pattern enrichments 

You can restore pattern enrichment .zip files as advanced rules models .zip files by completing the following steps:



1. Open your project.
2. On the Improvement tools panel of the Improve and customize page, click Teach domain concepts, Advanced rules models, and then Upload.
3.
For more information about the Entities and Part of Speech enrichments, see [Applying prebuilt enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-nlu). For more information about the enrichments that are applied to projects by default, see [Default project settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-project-defaults).



 Creating facets 

For more information about how to add facets to a Content Mining project, see [Adding facets](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-add-facets).

For other project types, you can create facets in the following ways:



* [From existing fields in a collection](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facetsfacetexist)
* [By adding a dictionary entry](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facetsfacetdict)
IBM Cloud

[By identifying a pattern](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-facetsfacetpattern)





 Creating a facet from existing fields in a collection 

As you apply enrichments to a collection, new fields are added to the index. Information that is recognized by the enrichment is stored in these new fields. You can use the enrichment fields as the source for facets.

For example, if you apply the Keywords prebuilt enrichment, you can create a facet based on the keywords that are found in your collection. As it processes your document, the Keywords enrichment recognizes any keyword mentions that occur and stores information about them in a set of fields that begin with the prefix enriched_{field_name}.keywords. To create a keyword facet, add a facet that takes its categories from the field where keyword mentions are stored, the enriched_{field_name}.keywords.mentions.text field.
You can use the [documents-ml.csv](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-ml.csv) file.

For more information about adding documents to a workspace, see [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotation).
3. Create an annotation set that uses the documents-ml.csv file as the base set, and assign it to yourself, the administrator.

After you complete the following steps to pre-annotate the new documents, you can view the annotation set to see how the machine learning model annotated the documents. Typically, you assign annotation sets to one or more human annotators. For more information about creating and assigning annotation sets, see [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-documents-for-annotation).
4. To pre-annotate the new documents:



* On the Machine Learning Model > Pre-annotation page click Run Pre-annotators.
* Select Machine Learning Model, then click Next.
* Select the document set that you added to the corpus, documents-ml.csv, and click Run.



5. After the pre-annotation is complete, create a human annotation task that includes the annotation set you created.

For more information about creating an annotation task, see [Annotation setup](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-annotate-documents).
6. To view the annotations that were applied by the machine learning model to the new documents, open the annotation task.

Because the new documents were pre-annotated with the machine learning model, human annotation requires less time.
{
"name": "resource_group_id",
"metadata": {}
},
{
"name": "resource_group_name",
"metadata": {}
}
],
"last_job": {
"command_object_id": "us-south.workspace.basic-resource-group.483b1ea2",
"command_name": "workspace_apply",
"job_status": "job_finished"
}
},
{
"module_id": "us-south.workspace.basic-cos-storage.535f45d6",
"module_type": "terraform",
"name": "basic-cos-storage",
"source": {
"source_type": "git_hub",
"git": {
"git_repo_url": "https://github.com/Cloud-Schematics/blueprint-example-modules/tree/main/IBM-Storage",
"git_branch": "main"
},
"catalog": {},
"cos_bucket": {}
},
"created_at": "0001-01-01T00:00:00Z",
"updated_at": "0001-01-01T00:00:00Z",
"inputs":
{
"name": "cos_instance_name",
"value": "$blueprint.cos_instance_name",
"metadata": {}
},
{
"name": "cos_single_site_loc",
"value": "ams03",
"metadata": {}
},
{
"name": "cos_storage_plan",
"value": "$blueprint.cos_storage_plan",
"metadata": {}
globalSearchService.search(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err)
});
2. Extract the value of the CRN field from the response.

{
"items": [
{
"crn": "crn:v1:bluemix:public:cf:au-syd:a/000af2ea12345aabb1234567801fffab::cf-organization:aaaf4100-0011-2233-1111-11aaffee0011"
}
],
"limit": 1,
"search_cursor": "e34535305339jg0rntt39nt3nu5tt9yn3u5ntt329nutt92u4ntt92u4t9u5gt2u5gt92u4n5g982458hg2t45hg9u42rg9t2u49gh285ght28h5t2835th85ht318h4tg9138h4tg91u3hgt931hg45918h34tg18h43hgt31hgt3rng0fnefvodnfvpojdpbojarfdbpojeafrbeafbjeqpnrqngrpqgrhbHNlIiwiY2FuVGFnIjoiZmFsc2UiLCJsdfshfksdhfkshfdkshfdkhkhewrkfehrkhkwhfkwrhkfhrgk3h5k3h45k3hk45hgk3hgk3hfk3h4k3hfkrfh3rkgh3krghk3rgh3kghk3hgk3hgk3rhgdWJsaWM6Y2Y6YXUtwriretoeiroteito4i5ot4i5oyti45ito4tio45ito45io5ogno5iogn5oin5oingoi5o5ngogo4ngo4ngro3jrong3gor3g3gno3jrgo3jrngo3ngro3g32ODQiXX0t"
}
Show more
3. Call the following command:



* Curl
* Java
* Python
* Go
* Node



curl -X POST -H "Authorization: {iam_token}" -H "Accept: application/json" -H "Content-Type: application/json" -d '{ "resources": [{ "resource_id": "crn:v1:bluemix:public:cloud-object-storage:global:a/59bcbfa6ea2f006b4ed7094c1a08dcdd:1a0ec336-f391-4091-a6fb-5e084a4c56f4::" }], "tag_names": ["project:myproject"] }' "https://tags.global-search-tagging.cloud.ibm.com/v3/tags/detach?tag_type=access"

Resource resourceModel = new Resource.Builder().resourceId(resourceCRN).build();
DetachTagOptions detachTagOptions = new DetachTagOptions.Builder()
.addResources(resourceModel)
.addTagNames("project:myproject")