vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Use NeuralSeek to return polished answers from existing help content 

In this tutorial, you will use the Watson Discovery, Watson Assistant, and NeuralSeek services that are available from the IBM Cloud catalog to create a virtual assistant that can answer questions about Watson Discovery. The assistant will generate answers by using the existing Watson Discovery product documentation as its knowledge base.

This tutorial shows the steps for creating a managed deployment of Discovery. However, you can create a Discovery service instance that is either hosted by IBM Cloud or installed in IBM Cloud Pak for Data and connect it to a NeuralSeek service instance.



 Learning objectives 

By the time you finish the tutorial, you will understand how to:



* Create a Document Retrieval project in Discovery.
* Upload PDF documents to your project, and apply a user-trained Smart Document Understanding model to your PDFs.
* Connect your Discovery project to a NeuralSeek service instance. NeuralSeek is an AI-powered answer generation engine.
* Create an assistant in Watson Assistant and apply a NeuralSeek integration to it.
* Add an action to your Watson Assistant that connects to NeuralSeek for answers.
* Use your assistant to answer questions about Discovery.



NeuralSeek is a third-party product that is provided by a vendor outside of IBM and is subject to a separate agreement between you and the third party, if you accept their terms. IBM is not responsible for the product and makes no privacy, security, performance, support, or other commitments regarding the product.



 Duration 

This tutorial will take approximately 4 to 5 hours to complete.





 Prerequisite 



1. Before you begin, you must set up a paid account with IBM Cloud.

You can complete this tutorial at no cost by using a Plus plan, which offers a 30-day trial at no cost. However, to create a Plus plan instance of the service, you must have a paid account (where you provide credit card details). For more information about creating a paid account, see [Upgrading your account](https://cloud.ibm.com/docs/account?topic=account-upgrading-account).
2. Create a Plus plan Discovery service instance.
* You can use any of the fields that are indexed by default. To see your choices, check the Fields to index list. Fields that have a Type value are stored in the index.
* The number of segments per document is limited to 1,000. After segment number 999 is created, any remaining document content is stored within segment 1,000.
* Metadata from PDF and Microsoft Word documents and any custom metadata is extracted and included in the index with each segment.



Be careful with documents that contain repeating sections, such as a catalog that has a description and specifications section for each product entry. If you split the document at too granular a level, the subsections, such as a section with specification details, can be disassociated from the product to which it belongs.

To split the documents in a collection, complete the following steps:



1. Click Manage collections from the navigation panel, and then click to open a collection.
2. Open the Manage fields page.

A list of the identified fields is displayed.
3. From the Improve query results by splitting your documents section, click Split document.
4. Choose the field that you want to use as your page break marker from the Select field dropdown.

The list that you can choose from includes a subset of all the identified fields.
5. Click Apply changes and reprocess.



You can check the status of the splitting process from the Activity page.

The metadata field includes the parent document ID. Each resulting segment of the original document can contain different information. For example, if you split the document based on the subtitle field, the first segment might contain only a title field. The next segment might contain a subtitle and a text field. The third might contain a subtitle field, a text field, and a footer field.



 Updating documents that were split 

If a document that was split changes and you want to upload the document again, work with a developer to replace the document by using the API. A developer can use the Update a document method to replace the original parent document.
iamUrl (Optional) The IAM token service URL that is used when iamApiKey is specified. The default value is https://iam.cloud.ibm.com/identity/token. 
 maxTriggers (Optional) Stop firing triggers when this limit is reached. Defaults to infinite. 
 filter (Optional) The filter function that is defined in a design document. 
 query_params (Optional) Any additional query parameters that might be needed for the filter function. 
 includeDoc (Deprecated) The includeDoc parameter is no longer supported for use with /whisk.system/cloudant/changes feed. 





1. Create a trigger named cloudantTrigger with the changes feed in the package binding that you created previously. Including the filter and query_params functions to fire the trigger when a document is added (or modified) when the status is new.

Replace /_/myCloudant of your package. This example uses a database that is called test.

ibmcloud fn trigger create cloudantTrigger --feed /_/myCloudant/changes --param dbname test

Example output

ok: created trigger feed cloudantTrigger
2. Save the following JavaScript code as cloudantChange.js.

function main(data) {
console.log(data);
}
3. Create an action called cloudantChange that you can use to observe the changes feed. Replace <file_path> with the file path to your cloudantChange.js file on your computer.

ibmcloud fn action create cloudantChange <file_path>/cloudantChange.js
4. Create a rule named cloudantRule to connect the cloudantChange action to the cloudantTrigger that you created earlier.

ibmcloud fn rule create cloudantRule cloudantTrigger cloudantChange
5. In another window, start polling so that you can see when activations occur.

ibmcloud fn activation poll
6. From the IBM Cloudant dashboard, either modify an existing document or create one.
7.
In general, try to design and implement your applications to do the minimum necessary amount of deletion.

See an example filter to exclude deleted documents during a replication:

{
"_id": "_design/filters",
"filters": {
"deleted_filter": "function(doc, req) { return !doc._deleted; };"
}
}





 Advanced removal of tombstone documents 

The [simple removal technique](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-tombstone-docssimple-removal-of-tombstone-documents) works well, if documents aren't being updated in the source database while the replication takes place.

If updates are made during replication, it's possible that a complete document is replicated to the target database as normal, and is also deleted from the source database, leaving a tombstone. The problem is that the tombstone is not replicated across to the target database because the tombstone is excluded by the filter. Therefore, the document that was deleted from the source database is not deleted from the target database, causing an inconsistency.

A solution is to do more advanced removal of tombstones by using a [validate_doc_update function](https://docs.couchdb.org/en/3.2.2-docs/ddocs/ddocs.htmlvalidate-document-update-functions).

A validate_doc_update function is stored in a design document. The function is run every time that a document is updated in the database. The function can be used to prevent invalid or unauthorized document updates.

The function works by using the following parameters:



* The new version of the document.
* The current version of the document in the database.
* A user context, which provides details about the user that supplied the updated document.



The function inspects the request to determine whether the update is allowed to proceed. If the update is acceptable, the function returns. If the update is not acceptable, a suitable error object is returned. In particular, if the user is not authorized to make the update, an unauthorized error object is returned, along with an explanatory error message.
For more information, see [Classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-classifier).

New Red Hat OpenShift support
: Added support for installing IBM Cloud Pak® for Data on Red Hat OpenShift.

Fixed the following defects in Discovery for Cloud Pak for Data offerings purchased on or after August 30, 2019
: During an active web crawl, if you add an enrichment, then click the Recrawl collection button on the Activity page, the collection will stop processing. If the collection does not return to a Syncing state on its own, clicking the Recrawl collection button an additional time might be required.
: While training a collection in the tooling , if you rate the relevancy of a result (for example, asRelevant), then switch to the opposite rating (Not relevant), the page may go blank. To restore the page, refresh the browser. Your updated rating will be retained.
: Chinese, Japanese, and Korean language Microsoft Word, Excel, and PowerPoint documents will not display correctly in the index or the Smart Document Understanding editor.
: If you upload a zip, gzip, or tar file to your collection, and that file contains multiple files/file types supported by Smart Document Understanding (PDF, Word, Excel, PowerPoint, PNG, TIFF, JPEG), only one of the files in that zip, gzip, or tar file will be available for training in the SDU editor (unless the SDU document limit has already been met). All of the documents will be available in the index. Unzip the file before uploading to avoid this issue.
: Query expansion and autocomplete return the wrong error code when the collection_id is invalid. Query expansion will return a 500 error code instead of a 404. Autocomplete will return a 400 when the collection_id is invalid and the prefix parameter isn’t set. It should also return a 404.
: When crawling Microsoft SharePoint 2019 collections, only HTML documents will be crawled and indexed. This is a SharePoint issue with how it processes mime-types.
If we want to fetch a single document back from the database, then documents sit one level beneath the database in the hierarchy of the URL.

So acurl $URL/books/id means "get document ID from the database books from the IBM Cloudant service at the supplied URL".

Notice the hierarchy: service, database, and document.

So far we only used the GET HTTP method, which is the default one for curl and the one used when you enter a URL into your web browser.

IBM Cloudant's API often uses the HTTP method as a verb to describe the action that is asked of the database: GET for fetching data.

With curl, we can specify the method that we want to use with the -X command-line option.

So to write a new document to our books database that uses the API, we're going to use the POST method, massing a document as the body of the HTTP request.

acurl -X POST specifies we're using the POST HTTP method. -d specifies the document that we want to write, which is sent as the body of the request, and finally, the URL we are writing to which is $URL/books - the books database.

Alternatively, we can use the PUT method, if we are supplying the ID of the document that is being written. The URL becomes $URL/books/ followed by the ID we want to write.

Both write methods yield identical responses. OK: True to show that the write was successful. ID being the document ID written, and rev being the revision token that was generated by the database.

To modify a document, we can use the PUT method to write the new body to the URL that points to the document ID we want to overwrite. -d supplies the new document body, and the URL not only contains the database and ID of the document, but critically the rev - the revision of the document we intend to mutate.

If we forget and omit the rev parameter, we get an error response.

HTTP response codes show whether a request succeeds or not. Responses in the 200 range are successful. Responses in the 400 range are user errors (for example, invalid parameters), and responses in the 500 range are server-side errors.
* ENV NAME - Define environment variables.
* CMD - Define commands that run when the container launches.



Images are typically stored in a registry that can either be accessible by the public (public registry) or set up with limited access for a small group of users (private registry). Public registries, such as Docker Hub, can be used to get started with Docker and Kubernetes to create your first containerized app in a cluster. But when it comes to enterprise apps, use a private registry, like the one provided in IBM Cloud Container Registry to protect your images from being used and changed by unauthorized users.

To containerize an app and store it in IBM Cloud Container Registry:



1. You would need to create a Dockerfile, below is an example of a Dockerfile.

 Build JPetStore war
FROM openjdk:8 as builder
COPY . /src
WORKDIR /src
RUN ./build.sh all

 Use WebSphere Liberty base image from the Docker Store
FROM websphere-liberty:latest

 Copy war from build stage and server.xml into image
COPY --from=builder /src/dist/jpetstore.war /opt/ibm/wlp/usr/servers/defaultServer/apps/
COPY --from=builder /src/server.xml /opt/ibm/wlp/usr/servers/defaultServer/
RUN mkdir -p /config/lib/global
COPY lib/mysql-connector-java-3.0.17-ga-bin.jar /config/lib/global
2. Once a Dockerfile is created, next you would need to build the container image and push it to IBM Cloud Container Registry. You can build a container using a command like:

docker build . -t <image_name>
docker push <image_name>







 Deploy your app to a Kubernetes cluster 

After a container image is built and pushed to the cloud, next you need to deploy to your Kubernetes cluster. To do that you would need to create a deployment.yaml file.
You can complete this tutorial at no cost by using a Plus plan, which offers a 30-day trial at no cost. However, to create a Plus plan instance of the service, you must have a paid account (where you provide credit card details). For more information about creating a paid account, see [Upgrading your account](https://cloud.ibm.com/docs/account?topic=account-upgrading-account).
2. Create a Plus plan Discovery service instance.

Go to the [Discovery](https://cloud.ibm.com/catalog/services/watson-discovery) resource page in the IBM Cloud catalog and create a Plus plan service instance.

Specify Dallas as the location.

As part of this tutorial, you will provision other services also. The services must be hosted in the same data location so that they can connect to one another. Because the NeuralSeek service is available only from Dallas, you will create all of the service instances in Dallas.



If you decide to stop using the Plus plan and don't want to pay for it, delete the Plus plan service instance before the 30-day trial period ends.







 Step 1: Get the product documentation 

To use the Discovery product documentation as our knowledge base, we will download the product documentation as a PDF file.



1. From a web browser, go to the product documentation site.

https://cloud.ibm.com/docs/discovery-data
2. From the table of contents panel, click the overflow menu icon in the Product guide section, and then choose View as PDF.
3. Save the PDF file to your system by clicking the Save icon from the page header.
4. Use a PDF file editor to split the PDF document into two separate PDF files of similar size.

Splitting the PDF creates two smaller files that can be enriched faster in Discovery.







 Step 2: Create a Document Retrieval project 

Now that you have the latest copy of the product documentation, add it to a Discovery project as your data source.

In Discovery, you will create a Document Retrieval project type. Documents that you add to a project of this type are automatically enriched in the following ways:



* Entities, such as proper nouns, are identified and tagged.
Client-side bottlenecks 

Often, poor performance is investigated and there is no indication of any lag or bottlenecks on the server side. These issues are resolved by making improvements to other aspects of the application architecture.



  Application design 

Reading and writing data to an object store can use significant resources in order to facilitate integrity checks, especially for applications that are transferring large objects. It is important to ensure that the application design takes into account any potential bottlenecks on CPU, disk IO, memory, and network use in order to minimize any performance impact.





  Compute resource power 

Containers, virtual machines, or bare metal must have sufficient power for the application to run efficiently. No amount of careful software design and engineering can overcome an underpowered computing environment, and optimal performance is dependent on adequate resources.





  Network 

NIC speeds can determine the limits of data throughput. It is possible to configure a virtual 10GB NIC, but there are limitations on how many transactions and threads a virtualized NIC can effectively perform and manage. Due to the countless potential configurations and types of virtualized NICs, it isn't possible to provide precise guidance. It is recommended to experiment with a rational amount of transactions to COS on a single virtualized NIC, perhaps 25-50 at first. Then test again, moving up or down in the number to try and find the proper balance and to provide the best performance on a consistent basis.

If encountering network errors, it is useful to provide the specific endpoint where requests are being sent when opening a support ticket. This allows the support team to efficiently investigate the networking to see if there are errors on IBM's network hops.
Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Event Streams. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

Event Streams documentation on service security and deletion of data covers methods to manage cardholder data within the environment in accordance with PCI requirements. It is the customer’s responsibility to familiarize themselves with these processes and to manage data retention and removal from the service according to the customer’s policies. To facilitate this process, no cardholder data can be used in an Event Streams document ID. If PAN data is to be stored in Event Streams, they must be rendered unreadable (in accordance with PCI requirement 3.4) before transmission to the Event Streams service.

A full list of PCI DSS-ready IBM Cloud Platform services, and options to request a PCI DSS AOC and SRM guide, can be found at the [IBM Cloud compliance page](https://www.ibm.com/cloud/compliance).

The Enterprise and Standard plans are PCI DSS certified.
For more information, see [How fields are handled](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overviewfield-name-limits).
* If a root-level field is an array but contains no items, the field is omitted from the index.
* If a root-level field is an array and contains only one item, the array is indexed as the data type of the one item. For example, a string array with one string is indexed as a string.
* If a nested field contains an array, even if the array has only one value, it is indexed as an array.
* If a root-level field is an array and contains more than one item, the data is indexed as an array.
* If you copy JSON that is generated by Discovery and then upload it as a JSON file, remove these system-generated fields from the file first: document_id, parent_document_id, filename, and title.
* You cannot enable the Optical Character Recognition (OCR) feature for JSON files.
* If your source document has a field with the name document_id, the field is skipped and not added to the index in the collection.

How the document_id field in a JSON file is handled changed with the 2023-03-31 version update of the API. Before the update, when you uploaded a JSON file from the product user interface or used the API to add it with the Add document method, the value in the document_id field from the file was shown as the document_id value in query results. However, a different document ID was assigned to it and stored in the parent_document_id field. The assigned document ID is what was returned when you called the List documents method and is what had to be used as the document_id in the endpoint URL for a Delete document method request. When you used the Update document method to assign a new document_id, the original ID continued to be returned in query results. However, the assigned ID had to be used to delete the document.
The next decision is whether to use the databases in an [active-active](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-configuring-ibm-cloudant-for-cross-region-disaster-recoveryactive-active) or [active-passive](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-configuring-ibm-cloudant-for-cross-region-disaster-recoveryactive-passive) manner.



 Active-active 

In an active-active configuration, different application instances can write to different databases.

For example, application "A" might write to database myaccount-dc1.cloudant.com/mydb, while application "B" might write to database myaccount-dc2.cloudant.com/mydb.

This configuration offers several benefits:



* Load can be spread over several accounts.
* You can configure applications to access an account with reduced latency (not always the geographically closest).



An application can be set up to communicate with the "nearest" IBM Cloudant account. For applications hosted in DC1, it's appropriate to set their IBM Cloudant URL to "https://myaccount-dc1.cloudant.com/mydb". Similarly, for applications that are hosted in DC2, you would set their IBM Cloudant URL to "https://myaccount-dc2.cloudant.com/mydb".





 Active-passive 

In an active-passive configuration, all instances of an application are configured to use a primary database. However, the application can fail over to the other backup database, if circumstances make it necessary. The failover might be implemented within the application logic itself, or by using a load balancer, or by using some other means.

A simple test of whether a failover is required would be to use the main database endpoint as a "heartbeat". For example, a simple GET request that is sent to the main database endpoint normally returns [details about the database](https://cloud.ibm.com/apidocs/cloudantgetdatabaseinformation).