vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Back up your machine learning model .zip or .pear files, and store them locally. For more information, see [Machine learning enrichments and Watson Explorer Content Analytics Studio models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-ml).





 Pattern enrichments 



1. Open your project.
2. On the Improvement tools panel of the Improve and customize page, click Teach domain concepts and then Patterns (Beta).
3. Click the download icon next to your pattern. Your pattern model then downloads as a .zip file.







 Advanced rules models enrichment 

Back up your model files as .zip files, and store them locally. For more information, see [Advanced rules models](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-mladvanced-rules).





 Classifier enrichments 

Back up your classifier .csv files, and store them locally. For more information, see [Classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifier).





 Entity extractor enrichments 

To back up entity extractor models, download the models and store them locally. A model must be fully trained before it can be downloaded. For more information, see [Exporting the entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractorentity-extractor-export).







 Content Mining application resources 

You cannot back up certain data types and must manually re-create them. There are several Content Mining custom user resources that the application does not automatically back up. If data loss occurs, you must either manually re-create the following custom user resources in the Content Mining application or upload a locally saved file that contains the resource:



* Custom map
* Searched document export: You can export a searched document in the Documents view in the Content Mining application, but you cannot reupload it in the application.
Facets give you a way to slice your data and visualize a subset of information so it is easier to comprehend.

From the data analysis page for your collection, you can choose for the data to be shown in one of the following views:

Facets
: Shows facets that are derived from from annotations that are added to your documents by enrichments that are applied to your documents. Enrichments can include built-in Natural Language Processing enrichments, such as Part of Speech or Entities. They can also include custom enrichments that you add, such as dictionaries, regular expression patterns, and machine learning models.

Metadata facets
: Shows facets that are derived from your data. When you add files to a collection, Discovery analyzes and indexes the data. Annotations are added to identify content types and are shown as metadata facets. The best metadata facets result when you ingest structured data, such as records from a CSV file.

Custom
: Shows only the facets that you choose to add to the view. You can add a mix of enrichment-derived and content-derived facets to your custom view.

When you create a Content Mining project type, the Part of Speech facet is applied to your data automatically. This facet is a great place to start because it is valid for all data, no matter the subject. The output gives you a quick look at the terminology that is most common in the data.

Zoom

![Watson Discovery content mining launch page](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/miningloginapp.png)

Figure 1. Watson Discovery Content Mining application home page

From this starting point, you can determine other ways to filter the data that might be useful.

If your data consists of traffic reports, for example, the Part of Speech facet might show that high frequency keywords include terms such as engine, brake, fire, smoke, and spark. Given this common terminology, you can create dictionaries to help you categorize and filter the data.
* [Step one: Create the peer organization's CA and metadata in cluster one](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mribp-console-hadr-peerCA)
* [Step two: Export the metadata and identities from cluster one](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mribp-console-hadr-export-meta1)
* [Step three: Import the metadata and identities in to cluster two and three](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mribp-console-hadr-import-meta23)
* [Step four: Create new peers in cluster two and three and join a channel](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mribp-console-hadr-create-new-peers)





[Setting up multiregion High Availability (HA) deployments for the ordering service](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mr-osibp-console-hadr-mr-os)



* [Before you begin](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mr-osibp-console-hadr-mr-os-prereq)
* [Deploy the ordering service in Region 1](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mr-osibp-console-hadr-mr-os-region1)



* [Create the ordering service CA](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-hadr-mr-osibp-console-hadr-mr-os-region1-ca)
Totals include enrichments that you create in this Content Mining project and in other projects in the same service instance.







 Machine learning configuration 

You can import an existing machine learning model.

To use Discovery to create a model, see [Entity extractor](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-entity-extractor).

To import a model, complete the following steps:



1. Click Select file, and then browse for the machine learning model file.
2. In the Facet path field, specify the root facet name to use for the model.

The facet name that you specify for the annotator is the facet name that is displayed from the collection search view.
3. Click Save.





 Machine learning model limits 



ML model plan limits

 Plan ML models per service instance 

 Cloud Pak for Data Unlimited 
 Premium 10 
 Enterprise 10 



Totals include enrichments that you create in this Content Mining project and in other projects in the same service instance.







 Regular expressions configuration 

You can import existing patterns by uploading them in a JSON file or you can add patterns.

To add patterns, complete the following steps:



1. Add the regular expression pattern to the New pattern field, and then click Add.
2. Specify a name for the pattern, and then identify the facet name to use for this pattern.

The facet name that you specify for the annotator is the facet name that is displayed from the collection search view.
3. Optional: Specify a facet value. You can specify a value from the options that are described in the table.



Regular expression facet value options

 Facet value Description 

 $0 Displays the matched text as-is. 
 $n If your regular expression pattern contains groups, you can specify a group number to return the matched text from the pattern group only. For example, if your regular expression consists of 3 groups that define a US phone number pattern, such as (d{3})-(d{3})-(d{4}), and you want to return only the area code portion of the phone number, you can specify $1.
Before we add anything to our new assistant, let's check on the status of our data.





 Step 5: Prepare your data for retrieval 

To improve the retrievability of the information in your PDF files, you will split the PDF files into many smaller documents. To do so, you will first teach Discovery about the structure of your PDF files, so it understands how subsections are formatted and can split the document by subsection.



1. Return to the web browser tab where your Discovery project is displayed.

The Improve and customize page for the last PDF file that you uploaded is displayed.
2. From the Improvement tools panel, expand Define structure, and then click New fields.

Zoom

![Shows the chat bot preview in a fake web page](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-new-fields.png)

Figure 5. Opening the tool for defining fields
3. Choose the Discovery docs part 1 collection.

The Identify fields tab is displayed, where you can choose the type of Smart Document Understanding model that you want to use.
4. Click User-trained models, and then click Submit.

Zoom

![Shows the chat bot preview in a fake web page](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-user-trained.png)

Figure 6. Creating a user-trained model
5. Click Apply changes and reprocess.

After some processing occurs, a representation of the document is displayed in the Smart Document Understanding tool. The tool shows you a view of the original document along with a representation of the document, where the text is replaced by blocks. The blocks represent field types.
* If you are importing a node that is deployed on a network deployed locally, you must ensure that the gRPC web proxy port used by the component is externally exposed to the console. For more information, see [Importing nodes from a locally deployed network](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-import-nodesibp-console-import-icp).
* When you open the tile of an imported node, the Fabric version is not visible and the Usage and info tab is not available, even if you have also imported the admin identity.







 Exporting and importing in bulk 

In cases where users want to export and import all of the peers, CAs, ordering services, MSPs, and identities at once, the IBM Blockchain Platform console now allows for the bulk management of data as a ZIP file that contains the JSON representing the various nodes, MSPs, and identities. To export or import data in bulk, navigate to the Settings tab in the left hand navigation. You will see a section called Bulk data management with two buttons below it. The Export button will open a panel on the right, where you have several options of what to export. The Import button also opens a panel where you can select the ZIP from your local file system.

While exporting and importing components in bulk is highly convenient for some use cases, there are important considerations to keep in mind:



* While you have the option to check or deselect the boxes representing peers, CAs, ordering services, MSPs, and identities, you cannot, for example, choose to export only some peers and not others. If the peer box is checked, every peer will be included in the ZIP, and likewise for other components.
* If a console has already imported some of the information contained in a bulk transfer, for example a few of the peers, duplicate representations of these components will appear in the console after performing the import. These duplicates do no harm, but it is a best practice to only have one representation of a component at a time. The same duplication will not occur when importing ordering nodes.
{"type":"record","name":"Citizen","fields":[{"name": "firstName","type":"string"},{"name":"lastName","type":"string"},{"name":"age","type":"int"},{"name":"phoneNumber","type":"string"}]}

Getting the latest version of a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Reader role access to the schema resource that matches the schema that is retrieved.







 Getting a specific version of a schema 

To retrieve a specific version of a schema, make a GET request to the /artifacts/{schema-id}/versions/{version} endpoint (where {schema-id} is the ID of the schema, and {version} is the version number of the specific version you need to retrieve). If successful, the specified version of the schema is returned in the payload of the response.

Example curl request

curl -u token:$APIKEY $URL/artifacts/my-schema/versions/3

Example response:

{"type":"record","name":"Citizen","fields":[{"name": "firstName","type":"string"},{"name":"lastName","type":"string"},{"name":"age","type":"int"},{"name":"phoneNumber","type":"string"}]}

Getting the latest version of a schema requires at least both:



* Reader role access to the Event Streams cluster resource type.
* Reader role access to the schema resource that matches the schema that is retrieved.







 Listing all of the versions of a schema 

To list all versions of a schema currently stored in the registry, make a GET request to the /artifacts/{schema-id}/versions endpoint (where {schema-id} is the ID of the schema).
"metadata": {}
},
{
"name": "resource_group_id",
"value": "$module.basic-resource-group.outputs.resource_group_id",
"metadata": {}
}
],
"outputs":
{
"name": "cos_crn",
"metadata": {}
},
{
"name": "cos_id",
"metadata": {}
}
],
"last_job": {
"command_object_id": "us-south.workspace.basic-cos-storage.535f45d6",
"command_name": "workspace_destroy",
"job_status": "job_finished"
},
"updated_at": "0001-01-01T00:00:00Z"
}
],
"updated_at": "0001-01-01T00:00:00Z"
}
},
"bastion": {},
"log_summary": {
"log_start_at": "0001-01-01T00:00:00Z",
"log_analyzed_till": "0001-01-01T00:00:00Z",
"repo_download_job": {},
"workspace_job": {},
"flow_job": {},
"action_job": {
"recap": {}
},
"system_job": {}
},
"updated_at": "0001-01-01T00:00:00Z"
}

For more information, see [troubleshooting section](https://cloud.ibm.com/docs/schematics?topic=schematics-bp-apply-fails).







 Next steps 



* Looking for additional blueprint samples to work with?
Indicate the collection to use, by doing one of the following things:



* Choose an existing project.

You can click the Open Discovery icon to review the configuration of a project before you decide which one to use.

Go to [Configure the search](https://cloud.ibm.com/docs/services/assistant?topic=assistant-skill-search-addskill-search-add-configure) and follow the remaining steps.
* If you do not have a project or do not want to use any of the projects that are listed, click Create a new project to add one. Follow the steps in [Create a project](https://cloud.ibm.com/docs/services/assistant?topic=assistant-skill-search-addskill-search-add-create-discovery-collection).

The Create a new project button is not displayed if you have reached the limits based on your Discovery service plan. See [Discovery pricing plans](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-pricing-plans) for plan limit details.









 Create a project 



1. On the OK, where is your data? page, select your data source, then click Next. Example choices include Salesforce, SharePoint, Box, IBM Cloud Object Storage, web crawl, and upload data.
2. On the Let's create a collection for your data pages, enter the information on how to connect to your data and configure your collection. The information you need to enter is different depending on the data source. For example, you need to enter your authentication credentials for services such as Salesforce, Sharepoint, and Box. For web crawl, you specify the website where your existing information resides.
3. Click Finish. Give Discovery a few minutes to start creating documents. You can use the Manage collections page within the project to see the progress.

After the digestion process is completed, a summary page is displayed in Discovery, which is hosted in a separate web browser tab.
4. Wait for the collection to be fully ingested, then click Back to Watson Assistant.







 Configure the search 



1.
* Saved analysis
* Report
* Dashboard







 Restoring your data to a new Watson Discovery instance 

Consider using your backups to restore to a new Discovery instance in a different data center, also known as a region or location.

To begin restoration, first start by reviewing your list of collections and associated data sources, as well as your file backups.



* Create your projects and collections. Use the Discovery tooling, or the API. See [Create a project](https://cloud.ibm.com/apidocs/discovery-datacreateproject) and [Create a collection](https://cloud.ibm.com/apidocs/discovery-datacreatecollection).
* Add back stopwords into the collections. See [Defining stopwords](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-stopwords).
* If you use custom query expansion, add your query expansions. See [Implementing synonyms](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settings).
* If you use any custom entity models from IBM Watson® Knowledge Studio for enrichment, reimport that model into your Discovery instance. For details, see [Managing enrichments](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-managing-enrichments).



After you set up your projects and collections as they were before, begin ingesting your source documents. Depending upon how you ingested your documents previously, you can do so by using your own solution or one of the following methods:



* The [API](https://cloud.ibm.com/apidocs/discovery-dataadddocument)
* A [connector](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sources)





 Restoring training data 

After you restore your projects, you can begin the process of re-creating your relevancy training models.
Discovery News, a public data set that is pre-enriched with cognitive insights, is also included with Discovery. You can use it to query for insights; for example: news alerts, event detecting, and trending topics in the news; that you can integrate into your applications. See [Watson Discovery News](https://cloud.ibm.com/docs/discovery?topic=discovery-watson-discovery-news) for more information. You cannot adjust the Discovery News configuration or add documents to this collection.



To create an environment and private data collection with the Discovery tooling do the following:



1. On the Manage Data screen, click the ![Environment](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/icon_settings.png) icon on the upper right and choose Create environment. The environment is created based on the IBM Cloud plan you selected earlier. The status of your environment is always available from this drop-down.
2. Once your environment is ready, click the Upload your own data button, then you can Name your new collection.

You can select the language of the documents you want to add to this collection: English, German, Spanish, Arabic, Japanese, French, Italian, Korean, or Brazilian Portuguese. There must only be one language in each of your collections. After you click Create, your data collection appears as a tile.



Your environment and data collection are ready. You can start [Adding content](https://cloud.ibm.com/docs/discovery?topic=discovery-addcontent) immediately.

You can use the Discovery tooling or API to crawl various data sources, or do a web crawl. See [Connecting to data sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for more information.





 Adding enrichments 

Discovery enriches (adds cognitive metadata to) the text field of your ingested documents with semantic information collected by these four Watson functions - Entity Extraction, Sentiment Analysis, Category Classification, and Concept Tagging.
* <SMART_CONTRACT_POD> with the name of the pod where the smart contract is running.
* <NAMESPACE> with the name of your cluster namespace.



For example:

kubectl logs -f chaincode-execution-0a8fb504-78e2-4d50-a614-e95fb7e7c8f4 -n nf85a2a





 Hyperledger Fabric v1.4 peer image 

![version 1.4](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/1-4_Pill.png) Use these instructions when the peer is based on a Fabric v1.4 image.



 Viewing smart contract logs on an IBM Cloud Kubernetes Service cluster 



1. Open your Kubernetes dashboard, filter on your [namespace](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-console-manage-consoleibp-console-manage-console-node-logs), and click the peer pod where the smart contract is running.
2. Click the Logs link from your dashboard. By default it points to peer container.
3. Switch to the dind container by selecting it from the drop-down list.



All of your smart contract logs are visible in this window and can be downloaded using the download icon on the panel.





 Viewing smart contract logs on a Red Hat OpenShift 4.x cluster 

See the Red Hat OpenShift [documentation](https://docs.openshift.com/container-platform/4.3/logging/cluster-logging-viewing.htmlcluster-logging-viewing-logs-console_cluster-logging-viewing). Access the logs for the peer pod where the smart contract is running and select the dind container.









 Using IBM Log Analysis to view the node logs 

By default, the logs of your nodes are collected locally within your cluster.
[Backing up and restoring components and networks](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore)



* [ Overview](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-overview)



* [Backup considerations for each node type](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-node-considerations)
* [Scheduling snapshots](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-schedule-snapshot)



* [Taking snapshots](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-take-snapshot)



* [Node snapshots](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-peer-snapshot)



* [Restoring nodes](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore)



* [Sequencing restorations](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-sequence)
* [Restoring a peer](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-peer)
* [Restoring an ordering node](https://cloud.ibm.com/docs/blockchain?topic=blockchain-backup-restorebackup-restore-restore-orderer)









 Building a network with APIs 

[Building a network with APIs](https://cloud.ibm.com/docs/blockchain?topic=blockchain-ibp-v2-apisibp-v2-apis)
: You can view a list of collections that are connected to a particular gateway. For more information, see [Viewing collections connected to a gateway](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sourcesgateway-connection).





 17 December 2020 

Improved date and time display on Activity tab
: Each collection now displays the Next sync scheduled for date and time on the Activity tab of the Manage collections page.

New beta FAQ extraction
: Released the beta feature FAQ extraction. FAQ extraction automatically extracts question-and-answer pairs from FAQ (frequently asked questions) documents and web pages so that your application returns more precise answers. For more information, see [FAQ extraction](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sourcesfaq-extraction). For a statement explaining beta features, see [Beta features](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-release-notesbeta-features).





 3 December 2020 

New Content Intelligence
: You can now apply the Contracts enrichment to a Document Retrieval project when you create it. The Contracts enrichment can be used to classify contract terms, parties, effective dates and more within your documents. For more information, see [Document Retrieval for Contracts](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projectsdoc-retrieval-contracts).





 10 November 2020 

New Box connector
: Crawl Box systems. For more information, see [Box](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-sourcesconnectboxpublic).

New SharePoint 2016 On-Premises connector
: Crawl SharePoint 2016 On-Premises systems. For more information, see [SharePoint 2016 On-Premises](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-connector-sharepoint-onprem-cloud).

The Box connector does not run on Safari