vania-janet/multiturn-rag-retrieval-data huggingface dataset:
There a few things to consider as you decide how to break up your source content into collections.



* Getting content from different data sources

If you store similar content in more than one type of data source (a website and Salesforce, for example), you can create one project with two separate collections. Each collection adds documents from a single data source. When they are built together into a single project, a user can search across both sources at the same time.
* Applying enrichments

Creating a collection is a good way to group documents that you want to enrich in a similar way. For example, maybe a subset of your documents contains industry jargon and you want to add a dictionary that recognizes the terms. You can create a separate collection and apply the Parts of Speech enrichment so you can use the term suggestions feature to speed up the process of creating the dictionary.
* Creating separate Smart Document Understanding (SDU) models

You can use the Smart Document Understanding tool to identify content based on the structure of a document. If you have 20 PDF files that were created by your Sales department and use one template and 20 PDF files that were created by your Research department and use a different template, group each set into its own collection. You can then use the SDU tool to build a model for each structure separately, a model that understands the unique structure. You can also use the tool to define custom fields that are unique to the source documents.







 Creating a collection 

Before you can create a collection, you must create a project. For more information, see [Creating projects](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-projects).

Things to keep in mind:



* A collection can support only one external data source.
* Documents in the collection must be in one language only, the language that you specify for the collection.



To create a collection, complete the following steps:



1. Open a project, go to the Manage collections page, and then click New collection.



* The Conversational Search, Document Retrieval, and Custom project types can contain up to 5 collections.
* A Content Mining project can contain only 1 collection.



2. Choose how you want to add data to your collection.
Apply the Part of Speech enrichment to your data ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Use the Entity Extraction, Document and Phrase Sentiment Analysis, and Keyword Extraction enrichments ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Use the Category classification, Concept tagging, Relation Extraction, Emotion Analysis, and Semantic Role Extraction, Sentiment of Keywords and Entities enrichments, which are available with the [Natural Language Understanding](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-about) service ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Build a custom entity type system ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Apply Watson Knowledge Studio NLP models to your data !
* [Table understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-understanding_tables)[1]
* [Text classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifier)



For the complete list of the enrichments that are supported in each language, see [Language support](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support).

For more information, see the Discovery [API reference](https://cloud.ibm.com/apidocs/discovery-dataanalyzedocument).



 Analysis example 

The data that you submit for analysis must be in JSON format. The text must be specified as a string; it cannot be specified as an array. For example, the following JSON file contains a quotation in the Quote field that you want to analyze to find any keyword mentions in the text.

{
"Author": "Jane Austen",
"Book": "Pride and Prejudice",
"Quote": "From this day you must be a stranger to one of your parents. Your mother will never see you again if you do not marry Mr. Collins, and I will never see you again if you do.",
"Year": "1813/01/01",
"Subject":"Parental love",
"Speaker": "Mr. Bennett",
"url": "https://www.gutenberg.org/files/1342/1342-h/1342-h.htmlink2HCH0020"
}

You know the name of a collection in your project where the Keywords enrichment is configured to be applied to documents in the collection. You can use the API to [list your collections](https://cloud.ibm.com/apidocs/discovery-datalistcollections) to find the ID associated with the collection that you look for by name.

After you get the collection ID, include it in the POST request that you submit to apply the configuration settings from the collection to your JSON file.
* Any incident issue that is found during the scan or updated is marked with a due date.
* Generate a summary.json file and store in IBM Cloud Object Storage at the end of every run that summarizes the details of the scan.



Let's get started with the creation and exploration of the CC toolchain.





 Step 1: Start the CC toolchain setup 

Start the CC toolchain configuration by using one of the following options:



* Click Create toolchain.

[![Create toolchain](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/create_toolchain_button.png)](https://cloud.ibm.com/devops/setup/deploy?repository=https%3A%2F%2Fus-south.git.cloud.ibm.com%2Fopen-toolchain%2Fcompliance-cc-toolchain&env_id=ibm:yp:us-south)
* From the IBM Cloud console, click Menu![Menu icon](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/icons/icon_hamburger.svg) and select DevOps. On the Toolchains page, click Create toolchain. On the Create a Toolchain page, click CC - Keep your app continuously Compliant with DevSecOps practices.







 Step 2: Set up the CC toolchain name and region 

The setup starts with a Welcome page that explains the toolchain, prerequisites needed to set this up, useful links like documentation or tutorial, and a framework diagram that explains the correlation between different integrated tools.

Zoom

!
* [Passing context from node to node](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-node-to-node)
* [Defining a context variable](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-var-define)
* [Common context variable tasks](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-common-tasks)
* [Deleting a context variable](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-delete)
* [Updating a context variable](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-update)
* [How context variables are processed](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-processing)
* [Order of operation](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-order-of-ops)
* [Adding context variables to a node with slots](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-runtime-contextdialog-runtime-context-var-slots)





 Passing context from the application 

Pass information from the application to the dialog by setting a context variable and passing the context variable to the dialog.

For example, your application can set a $time_of_day context variable, and pass it to the dialog which can use the information to tailor the greeting it displays to the user.

!
Visualizations are a great way to get a useful overview of data, but you also need a way to navigate into the event records that populate the charts.



1. Hover over the arrow icon in the bottom-left corner of any visualization. A small arrow appears.
2. Click the arrow to see a table of the data that is used in that visualization.
3. Click the View Events label to navigate into the individual event details for up to 100 records.

![statuscodetable.png](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/statuscodetable.png)



You can edit, move, and delete visualizations on your dashboard.





 Activity 3: Building new dashboards 

Now, let's create a new dashboard that will provide view of API traffic patterns. These are all available using built-in visualizations.



1. Click the new dashboard icon and click the Choose from existing visualizations link.

![newdashboard.png](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/newdashboard.png) A list of available visualizations is displayed.
2. Select a few visualizations to add to your dashboard. For example:



* Subscribed Apps
* Apps per Plan
* Success Rate
* API Calls per Day



Tip When you select each visualization, the selection tab blocks your dashboard view, so you may not realize that the visualization was added to the dashboard. Select one visualization at a time and close the selection tab each time to see the changes to your dashboard.
3. Click Save and give your dashboard a name: Subscriber Dashboard.
[Rich document preview](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/search-doc-preview-example.png)

Figure 4. Rich document view

[Try it](https://watson-developer-cloud.github.io/discovery-components/storybook/?path=/story/documentpreview--default)





 Deploying a project 

To deploy your project, complete the following steps:



1. To use the API, you need to know the project ID for your project. Go to the Integrate and Deploy > API Information page.
2. From the Integrate and Deploy > UI Components page, find links to resources that a developer can use to get started.



* [GitHub](https://github.com/watson-developer-cloud/discovery-componentsusing-discovery-components)
* [Storybook](https://watson-developer-cloud.github.io/discovery-components/storybook/)









 Getting started with the GitHub sample app 

From resources available in GitHub, you can run a script to start a sample app with prebuilt UI components. In fact, the sample app looks a lot like the Improve and customize page of the product because the product itself uses these UI components.

The script requires some prerequisite software to function. After you start the script, it checks whether you have the necessary software installed on your system. If not, it lets you know what software you need to install. Install the following packages if they are not installed already:



* [Node.js](https://nodejs.org/en/)
* [Yarn](https://yarnpkg.com/getting-started/install)



The script needs information about your service instance and project to use the data and search settings that you configured for your project and apply them to the sample app.
Upload more documents to the workspace. You can use the [documents-ml.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-ml.csv) file.

For more information about adding documents to a workspace, see [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotation).
3. Create an annotation set that uses the documents-ml.csv file as the base set, and assign it to yourself, the administrator.

After you complete the following steps to pre-annotate the new documents, you can view the annotation set to see how the machine learning model annotated the documents. Typically, you assign annotation sets to one or more human annotators. For more information about creating and assigning annotation sets, see [Adding documents for annotation](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-documents-for-annotation).
4. To pre-annotate the new documents, click Machine Learning Model > Versions, and then click Run this model.
5. Select the document set that you added to the corpus, documents-ml.csv, and click Run.
6. After the pre-annotation is complete, create a human annotation task that includes the annotation set you created.

For more information about creating an annotation task, see [Annotation setup](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-annotate-documents).
7. To view the annotations that were applied by the machine learning model to the new documents, open the annotation task.
Exploring bucket objects 

To explore the objects in a bucket, use one of the following methods:



*  Exploring bucket objects in list view





1.  Click the name of bucket that you want to explore. The bucket information page opens.
2.  Click Objects.





*  Exploring bucket objects in topology view





1.  Click the bucket that you want to explore. The bucket information page opens.
2.  Click Objects.
Connecting to data sources 

You can use Discovery to connect to and crawl documents from remote sources.

You can connect to a data source and pull documents on a schedule into Discovery by configuring a collection to associate with that source. You can configure each collection with one data source. Discovery pulls documents from the data source using a process called crawling. Crawling is the process of systematically browsing and retrieving documents from the specified start location. Discovery only crawls items that you explicitly specify. The first time that a crawler crawls a data source is a full crawl, and every time that the crawler runs after the full crawl, per the crawl schedule, is a refresh, also called a recrawl.

This documentation applies to Discovery service instances that you create with a Lite or an Advanced plan, or that you created with a Premium plan before 16 July 2020. For more information about features in Premium plan instances created on or after 16 July 2020, and in Plus (including Plus Trial) plan instances, see [these docs](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collections).



 General source requirements 

The following general requirements apply to all data sources:



* The individual document file size limit for Box, Salesforce, SharePoint Online, SharePoint OnPrem, IBM Cloud® Object Storage, and Web Crawl is 10MB.
* You must have the credentials, file locations, or URLs for each data source. A developer or system administrator typically provides the credentials, file locations, and URLs of the data source.
* You must know which resources of the data source to crawl, which the source administrator can provide. If you crawl Box or Salesforce, a list of available resources is presented when you configure a source, using the Discovery tooling.
* If you are using the Discovery tooling, you can configure a collection with a single data source.
* Crawling a data source uses resources, namely API calls, of the data source. The number of API calls depends on the number of documents that need to be crawled. You must obtain an appropriate level of service license, for example Enterprise, for the data source.
[Filter the list of subnets](https://cloud.ibm.com/docs/subnets?topic=subnets-view-all-subnetsfilter-details) by location, such as Dallas 5.
3. In the resulting filtered list, click the name of the VLAN in the VLAN column. To identify a specific POD within a data center, check the VLAN name for a string like, bcr02 or fcr04. Bcr02 = backend customer router, POD 2. Fcr04 = frontend customer router, POD 4.
4. On the Instance detail page, locate the Devices section that identifies virtual server instances and bare metal servers that are provisioned in the VLAN.
5. To identify all devices in the data center, complete these steps for each VLAN in the data center.



2. Order replacement servers in new data centers. For more information, see the following documentation:



* [Planning for VPC instances](https://cloud.ibm.com/docs/vpc?topic=vpc-vsi_best_practices)
* [Getting started with virtual servers](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-getting-started-tutorial)
* [Getting started with bare metal servers](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-getting-started)
* [Comparing IBM Cloud Classic and VPC infrastructure environments](https://cloud.ibm.com/docs/cloud-infrastructure?topic=cloud-infrastructure-compare-infrastructure)



3. Copy your data to a new server by using the following methods. You must establish connectivity between the old and new servers and have sufficient storage in the new server.



* Use [Secure Copy Protocol (SCP)](https://www.ibm.com/support/knowledgecenter/ST5Q4U_1.5.2/com.ibm.storwize.v7000.unified.152.doc/usgr_usng_scp.html) to securely copy a file from source to destination.
Type '3' and press enter.

===============================================================================
Select a Starter Kit:

APPSERVICE
-------------------------------------------------------------------------------

1. Node-RED - A starter to run the Node-RED open-source project on
IBM Cloud.

2. Node.js + Cloudant - A web application with Node.js and Cloudant

3. Node.js Express App - Start building your next Node.js Express
app on IBM Cloud.

WATSON
-------------------------------------------------------------------------------

4. Natural Language Understanding Node.js App - Use Watson Natural
Language Understanding to analyze text to help you understand its
concepts, entities, keywords, sentiment, and more.

5. Speech to Text Node.js App - React app using the Watson Speech to
Text service to transform voice audio into written text.

6. Text to Speech Node.js App - React app using the Watson Text to
Speech service to transform text into audio.

7. Visual Recognition Node.js App - React app using the Watson
Visual Recognition service to analyze images for scenes, objects, text,
and other subjects.
For more information, see [Managing projects and extractors > Running an extractor](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractorsrunning-an-extractor).
6. Export your model



1. Click Save to save your model.
2. In the Extractors pane, right-click an extractor category, or right-click a single view under an extractor category. Click Export.
3. If you want the model to be able to analyze HTML content and output spans over the HTML tags in Natural Language Understanding, select Enable Detagging.
4. Enter the Name of file, then click OK.



Exporting a model from a Category, if multiple concepts internally use the same names for one or more output views, is expected to fail when deploying to Discovery or Natural Language Understanding.





 Uploading an advanced rules model to other services 

After you export an advanced rules model, you can upload it to Natural Language Understanding. See the following page for more details.



* Natural Language Understanding: [Customizing > Analyzing text with advanced rules (Beta)](https://cloud.ibm.com/docs/natural-language-understanding?topic=natural-language-understanding-customizingadvanced-rules)
* Discovery for Cloud Pak for Data: [Creating enrichments > Advanced rules models enrichment (Beta)](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domainadvanced-rules)
Gathering Ingress logs 

Run the following commands to gather the required logs for debugging Ingress.



1.  List ALBs.

ibmcloud ks ingress alb ls -c CLUSTERID
2.  Get the Ingress status.

ibmcloud ks ingress status-report get -c CLUSTERID
3.  Get nodes and node labels.

kubectl get nodes --show-labels
4.  Get endpoints.

kubectl get endpoints -o wide
5.  Describe your Ingress resource.

kubectl describe ing <ingress name> -n <namespace>
6.  Get the logs for the nginx-ingress container in your Ingress pods.

kubectl logs <ingress pods> -n kube-system -c nginx-ingress
7.  List host names for network load balancer (NLB).

ibmcloud ks nlb-dns ls -c CLUSTERID
8.  List the alb pods in the kube-system namespace.

kubectl get pods -n kube-system | grep alb
9.  Run an nslookup on your Ingress subdomain.

nslookup ingresssubdomain.com



Review the output for error messages, then review the list of [Ingress and routers troubleshooting topics](https://cloud.ibm.com/docs/openshift?topic=openshift-sitemapsitemap_ingress_and_routers) for help resolving common Ingress issues.
[Shows that the Upload data option is chosen from the data sources page](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-data-source.png)

Figure 2. Creating a collection from uploaded data
2. Name the collection Discovery docs part 1, and then click Next.
3. Click Drag and drop files here or upload, and then browse to add the first PDF file that you created earlier.
4. Click Finish.

Your file is processed as it is added to the collection.
5. From the navigation panel, click Manage collections, and then click New collection.

Zoom

![Shows the Manage collections page where you can add another collection to the project.](https://cloud.ibm.com/docs-content/v1/content/ca9232b3702768f24bb897f48f10778fcda28718/watson-assistant/images/tut-neural-second-collection.png)

Figure 3. Adding a second collection
6. Repeat the previous steps to add the second PDF file as a collection named Discovery docs part 2.



After the data is uploaded, it is processed and indexed by Discovery. While the data is being processed, let's create our virtual assistant.





 Step 4: Create an assistant 

For this tutorial, you will create an assistant with a single action. First, you must create a Watson Assistant service instance.

Both Lite and Trial plan Watson Assistant service instances are available at no cost. You will create a Trial plan.



1. From a new web browser tab, return to the IBM Cloud catalog.

Keep the Discovery page open in a separate tab so you can switch easiily between the two applications.
2.
For more information about the binary option, see [RTF 4522](https://datatracker.ietf.org/doc/html/rfc4522).

In the Binary attributes field, specify the names of the binary attributes that you want to index.



9. If you want the crawler to extract text from images on the site, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
10. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.