vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Sample data for CSV files

 Claim_id Date Product_line Product Client_segments Client_location Client age Feedback Label 

 0 2016/1/1 tea lemon tea Not Member Manhattan 20 The straw was peeled off from the juice pack. package_container 
 1 2016/1/2 ice cream vanilla ice cream Silver Member Queens 20 I got some ice cream for my children, but there was something like a piece of thread inside the cup. contamination_tampering 



Note that the two required fields are present in the sample. The required fields have the following names:



* Feedback: Natural language text to label.
* Label: Label to apply to the feedback.









 Opening the Content Mining application 

If you didn't do so, create the project and add a collection to it. If you already created the project and collection, you can skip this procedure and [create the document classifier](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifiercm-doc-classifier-task).



1. In Discovery, create a Content Mining project.
2. Choose to upload data to create the collection. Name your collection, and click Next.
3. Upload the CSV file that contains your training data.

The training data file must contain the following information at a minimum:



* A column that contains sample text that you want to classify. For example, the sample text might be a product review.
* A column that contains a class or category label that is assigned to the sample text.



4. After collection processing is complete, click Launch application to open the Content Mining application.

The facet details are displayed for the collection.







 Creating a document classifier 

To create a document classifier, complete the following steps:



1. From the Content Mining application, click the Collections link in the breadcrumb to open the Create a collection page.

The status of index creation is displayed. Wait for the collection to be fully indexed before you continue with this procedure.
2. To create a classifier, click collection, and then choose classifier from the list.

Zoom

!
SELECT COUNT() AS logLines FROM cos://eu-de/results-at/jobid=f178778e-7707-46a9-982d-1e89261b63a5 STORED AS PARQUET
INTO cos://eu-de/results-marisa STORED AS CSV





 Step 3.6. Run a query to get a custom view of a subset of the event fields 

To see information about each event, run the following query:

SELECT <FIELDS> FROM <PARQUET_FILE> STORED AS PARQUET
INTO <RESULTS_BUCKET> STORED AS CSV

Where



* FIELDS is the list of fields that you want to get information on for the different records. For example, you can enter _source.eventTime AS EVENTTIME, _source.action AS ACTION, _source.severity AS SEVERITY, _source.outcome AS OUTCOME, _source.o_initiator.id AS INITIATOR_ID, _source.o_initiator.name AS INITIATOR_NAME
* PARQUET_FILE is the Result location URL that you get when you transform the archive file from JSON to PARQUET
* RESULTS_BUCKET is the SQL URL of the custom COS bucket that you plan to use to upload the query results



For example, to get the list of actions, you can run the following query:

SELECT DISTINCT _source.action
FROM cos://eu-gb/sql-results/jobid=17cee056-4da1-4429-8aca-3a7eb320ee27 STORED AS PARQUET
INTO cos://eu-gb/sql-results STORED AS CSV

For example, to get the list of actions for a user, you can run the following query:

SELECT _source.eventTime, _source.action, _source.o_target.name
FROM cos://eu-gb/sql-results/jobid=3aa9e732-ba88-4ffe-b9fc-b8a265876467 STORED AS PARQUET
WHERE _source.o_initiator.name = "xxx@ibm.com"
3. Click EXPLORE.







 Downloading a capture 

Complete the following steps to download a capture file:



1. Navigate to the Captures section in the web UI. [Learn more](https://cloud.ibm.com/docs/monitoring?topic=monitoring-launchlaunch).
2. Identify and select the capture file that contains the data that you want to download.
3. Click DOWNLOAD.







 Chisels 

A chisel is a script that is written in Lua, a scripting language. You can use it to analyze data in a capture file.

For more information, see [Chisels User Guide](https://github.com/draios/sysdig/wiki/Chisels-User-Guide).





 Csysdig 

Csysdig is an open source tool for Linux that is designed for monitoring and debugging. You can use it to analyze capture files.

For more information, see the following resources:



* [Csysdig blog](https://sysdig.com/blog/csysdig-explained-visually/)
* [Csysdig video](https://www.youtube.com/watch?v=UJ4wVrbP-Q8)
To query JSON data where no schema can be inferred, for example, in logs, use the option to read the data AS TEXT. The input is then read without inferring any schema and each line is put into a single column named value.

To see how the data is structured, execute a basic select query.

SELECT 
FROM cos://us-geo/sql/LogDNA/_year=2019/_dayofyear=226/_hour=14 STORED AS TEXT
LIMIT 5

You can further extract fields by using the options get_json_object or regexp_extract. You can then filter data on the extracted columns.

WITH logs (
SELECT get_json_object(value, "$._source._host") as host,
from_unixtime(get_json_object(value, "$._source._ts") / 1000, 'yyyy-MM-dd HH:mm:ss') as timestamp,
get_json_object(value, "$._source._file") as file,
get_json_object(value, "$._source.request_method") as request_method,
get_json_object(value, "$._source.request_uri") as request_uri
FROM cos://us-geo/sql/LogDNA/_year=2019/_dayofyear=226/_hour=14 STORED AS TEXT
)
SELECT 
FROM logs
WHERE request_uri LIKE "/iae_instances/%"
ORDER BY timestamp

To get the logs for a specific timeframe, use the following query.

WITH logs (
SELECT get_json_object(value, "$._source._host") as host,
get_json_object(value, "$._source._ts") / 1000 as unix_timestamp,
from_unixtime(get_json_object(value, "$._source._ts") / 1000, 'yyyy-MM-dd HH:mm:ss') as timestamp,
* [Using private modules with templates](https://cloud.ibm.com/docs/schematics?topic=schematics-download-modules-pvt-gitnetrc-example)



[Managing IBM Cloud resources with workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecyclemanage-lifecycle)



* [Deploying your resources](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycledeploy-resources)
* [Updating your resources](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycleupdate-resources)
* [Managing drift between your cloud environment and your Terraform configuration](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycledrift-report)
* [Removing your resources](https://cloud.ibm.com/docs/schematics?topic=schematics-manage-lifecycledestroy-resources)



[Detecting drift in workspaces](https://cloud.ibm.com/docs/schematics?topic=schematics-drift-notedrift-note)



* [Example drift scenario](https://cloud.ibm.com/docs/schematics?topic=schematics-drift-notedrift-scenario)
* [Drift detection in IBM Cloud](https://cloud.ibm.com/docs/schematics?topic=schematics-drift-notedrift-in-ibm)
* [Drift detection using the UI](https://cloud.ibm.com/docs/schematics?topic=schematics-drift-notedrift-ui)



* [Viewing detect drift logs using the UI](https://cloud.ibm.com/docs/schematics?topic=schematics-drift-notedrift-view-ui)
1. Open the Actions page.
2. If you have no actions, choose Create a new action. If you already have some actions, choose New action.
3. On Create an action, choose Quick start with templates.

Quick start with templates is available in English-language assistants only.
4. On Quick start with templates, add the NeuralSeek search starter kit.







 Edit system actions 



1. Click Set by assistant and open the No action matches action.
2. Delete the two default steps.
3. Add a step. Set And then to Go to a subaction and choose the NeuralSeek search action.
4. If you aren't connecting your customers to a live agent, you might want to edit the Fallback action in the same way as No action matches.







 Using your NeuralSeek extension 

Issue a query to your assistant. If no action that matches that query, then it uses NeuralSeek to produce search results.

For many use cases, NeuralSeek alone is enough to deploy an assistant. If you are happy with your assistant, you might want to deploy it for real-world use right away. Use the Analyze page in Watson Assistant or the Curate page in NeuralSeek to see what kinds of questions users are asking and actions for the common user requests. The Curate page can automate the creation of new actions and generate new example utterances that trigger those actions. It can also merge any existing actions with the actions that it generates so you can update an existing assistant. For more information, see the [NeuralSeek documentation](https://neuralseek.com/documentation)
if newDocument.exists():
print("Document '{0}' successfully created.".format(number))







 Step 4: Retrieving data 

To perform a minimal retrieval, you first request a list of all documents within the database. This list is returned as an array. You can then show the content of an element in the array.



1. Retrieve a minimal amount of data.

result_collection = Result (myDatabaseDemo.all_docs)
print("Retrieved minimal document:n{0}n".format(result_collection[0]))
2. See a result similar to the following example.

[{
"id": '60e19edf809418e407fb6791a1d8fec4',
"key": '60e19edf809418e407fb6791a1d8fec4',
"value": {
"rev": '2-3d6dc27627114431c049ddecae9796e0'
}
}]



In a relational database, the first document that is stored in a database is always the first document that is returned in a list of results. This notion doesn't necessarily apply to NoSQL databases, such as IBM Cloudant.



 Full retrieval of a document 

Additionally, to perform a full retrieval, you request a list of all documents within the database, and specify that the document content must also be returned. You run a full retrieval by using the include_docs option. As before, the results are returned as an array. You can then show the details of an element in the array by including the full content of the document this time.



1. Request the first document that is retrieved from the database.

result_collection = Result(myDatabaseDemo.all_docs, include_docs=True)
print("Retrieved minimal document:n{0}n".format(result_collection[0]))
2. See the result, which is similar to the following example:

[
Find the Welcome node in the dialog tree.
3. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon on the Welcome node, and then select Add node below.
4. Start to type General_About_You into the If assistant recognizes field of this node. Then select the General_About_You option.
5. Add the following message in the response text field:

I am a virtual assistant that is designed to answer your questions
about the Truck Stop Gourmand restaurant. What should I call you?

![Shows the #General_About_You node being added.](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/gs-ass-add-about-you-node.png)
6. Click ![Close](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/close.png) to close the edit view.
7. Click the More![More options](https://cloud.ibm.com/docs-content/v1/content/417db917d18067d9dcc4cf2612564a9a87c3ddc8/assistant/images/kebab.png) icon on the General_About_You node, and then select Add child node.
8. In the If assistant recognizes field of this node, enter true.
9. Add the following message in the response text field:

Hello, <? input.text ?>!
Most of the content that is returned in search results that you submit from the Improve and customize page originates from this one field. How to parse and return only relevant chunks of information from this field is determined by the query result configuration that is used by the project. For more information, see [Previewing the default query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-results).

More processing adds more fields. And more processing is applied automatically depending on the project type. When processes run on documents in a collection, extra fields are added to store information that is associated with the process. For example, when the built-in Entities enrichment is applied to a collection, it starts a process that adds fields with names that begin with enriched_{field_name}.entities to the documents in the collection.



* For more information about the enrichments that are applied by default, see [Default project settings](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-project-defaults).





 How fields are handled 

For most unstructured file types, the bulk of the content from the file is added to a field named text. For file types that have an inherent data structure, such JSON files, names from the source file are used to name the fields in which the content is stored. When you upload files of this type, be aware of some naming limitations that exist for fields.

The following field names have special meaning. If possible, do not use these names in your structured source files.



* document_id
* highlight
* html
* metadata
* parent_document_id
* result_metadata
* score
* spans



Avoid field names that meet the following conditions. Field names with these restricted characters are not queried.



* Start with the characters _, +, and -. For example, +extracted-content.
* Contain the characters ., ,, , ?, (, ), or : or spaces. For example, extracted content or new:extracted-content.
* End with numbers, for example, extracted-content2.





 HTML fields
See [Query expansion](https://cloud.ibm.com/docs/discovery?topic=discovery-query-conceptsquery-expansion) for more information.





 17 April 2019 

Deprecated Data Crawler : The Data Crawler is no longer available for download and is no longer supported. See [Connecting to Data Sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for other connectivity options.





 2 April 2019 

Update to scoring algorithms : Beginning April 9, 2019, there is an upgrade to IBM Watson™ Discovery. This upgrade includes changes to the scoring algorithms used in Discovery for ranking documents and passages. This means score and confidence results might change, following the upgrade. If your applications make use of the score or confidence fields directly, be prepared to update the application as necessary. See [Upgrading the infrastructure of the IBM Watson™ Discovery](https://www.ibm.com/blogs/cloud-archive/2019/03/announcement-for-the-ibm-watson-discovery-community/) for details.





 25 March 2019 

Update to version string : The version string for all API calls changed to 2019-03-25 from 2019-01-01

Improved extraction of title field from HTML documents : The title field is now extracted from HTML documents as a top-level field during conversion. This title field is included in each segment of any documents split, using document segmentation. For an example, see [Splitting documents with document segmentation](https://cloud.ibm.com/docs/discovery?topic=discovery-configservicedoc-segmentation). This might improve the confidence score for query results, and might change the passages returned because the title might be returned, as part of a passage. Existing collections must be reindexed to extract the title field. : The Discovery tooling does not yet use the current API version: 2019-03-25 (it currently uses 2018-12-03), so the title is not extracted when ingesting HTML documents by uploading or using the Web Crawl connector in the Discovery tooling.

New enrichment for arrays : Arrays can now be enriched (previously, arrays could not be enriched).
For more information, see [Identifying the text field](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-edit-collectiontext-field).)
3. Click Apply changes and reprocess.











 Results of classification 

After the enrichment is applied to a collection, a facet is generated that you can use to find the predicted classes. In this example, the predicted field is named label_answer_predicted.

Zoom

![Shows the facet that is generated after the doc classifier enrichment is applied to a collection](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/cm-doc-class-predicted.png)

Figure 2. A Label_answer_predicted facet is generated

Use the generated facet to filter documents by classification and analyze subsets of documents. Doing so helps you to find patterns and discover other insights. You can export these target documents to share with team members or to analyze further. For more information, see [Exporting data](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-export).

When the document classifier classifies a document, it stores the classification in the document_level_enrichment.classes.class_name field.

For example, the following JSON excerpt shows a document that was classified with the package_container class.

Zoom

![Shows the JSON source for a document that has been classified with the package_container class.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/json-classifier-found.png)

Figure 2. Document classifier enrichment syntax



 Document classifier limits
[Evidence collection](https://cloud.ibm.com/docs-content/v1/content/5eb88a2575e30e316dbb83829c7f4c8b1ddbc67c/devsecops/images/CI-CD-evidence.png)

Figure 3. Evidence collection



 Code stage 

In the Code stage, evidence is collected for the following steps:



* Detect secrets
* Unit test results
* Code Vulnerability scan, CIS check, and Bill of Material check by way of Code Risk Analyzer







 Build stage 

In the Build stage, evidence is collected for the following steps:



* Vulnerability Advisor scan
* Image signing







 Deploy stage 

In the Deploy stage, evidence is collected for the following steps:



* Create change request
* Approval of change request
* Service specific tests
* Acceptance tests
* Closing the change request









 v1 evidence format 

The Evidence type represents the schema of the evidence. Although the schema uses typescript syntax, you can convert it to use JSON schema.

interface Evidence {
evidence_type_id: string; // name of the evidence
evidence_type_version: string; // version of the evidence schema
date: string; // ISO-8601, UTC
toolchain_crn: string;
pipeline_id: string;
pipeline_run_id: string;
result: 'success' | 'failure';
issues: string[]; // list of issues that were opened to track 'failed' state
log: Artifact[]; // logs related to the evidence
subject: string; // asset url for the evidence
subjects: string[]; // a piece of evidence can be relevant for multiple asset in v2
artifacts: Artifact[]; // additional artifacts related to the evidence
tool: string; | undefined // the tool related to the evidence (optional)

[origin]: {
url: string; // origin of the evidence, e.g. path github where the task resides;
Continue annotating documents until Watson can correctly and consistently map different types of content to the appropriate fields for you.
8. After you teach Watson to identify fields, click Apply changes and reprocess.



Custom fields that you define by using the SDU tool are indexed as root-level fields.



 What to do next 

When you build a user-trained model, you change where information is stored in your documents. Next, change how the search results are configured. By default, search results are retrieved from passages or the text field. You might have a better field to use as the source of the result body. For more information, see [Changing the result content](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-resultsquery-results-content).

If your project is being used by a virtual assistant, update the search skill configuration to pull the answer body from a different field. For more information, see [Configure the search](https://cloud.ibm.com/docs/assistant?topic=assistant-skill-search-addskill-search-add-configure).

You can apply enrichments, either custom or prebuilt enrichments, to the new root fields that are generated by the SDU model.

If you want to return shorter text snippet with a search result, you can split your documents based on one of the new fields that you defined, such as chapter or section.





 Available fields 

The following fields are available for you to apply to documents by using the Smart Document Understanding tool.

The fields are arbitrary. You can apply the image field to every title in the document if you want. Although, it might be difficult to know which field to search later for information that you need if the field names don't match the content. The default set are representative field types that are meant to help you get started. Only the text and table fields have special significance. Do not use them to identify anything other than text and tables.



Default field labels

 Field Definition 

 answer In a question-and-answer pair (often in an FAQ), the answer to the question. 
 author Name of author or authors.
If you want the crawler to extract text from images in documents, expand More processing settings, and set Apply optical character recognition (OCR) to On.

When OCR is enabled and your documents contain images, processing takes longer. For more information, see [Optical character recognition](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-collectionsocr).
11. Click Finish.



The collection is created quickly. It takes more time for the data to be processed as it is added to the collection.

If you want to check the progress, go to the Activity page. From the navigation pane, click Manage collections, and then click to open the collection.



 Using Windows Authentication on Linux 

The JDBC driver from Microsoft does not support Windows Authentication on Linux. If you want to use Microsoft Windows authentication to access your SQL Server on Linux, you can use a third-party JDBC driver called jTDS from [Sourceforge](http://jtds.sourceforge.net/). Specify the following values during the configuration:



* Database URL: jdbc:jtds:sqlserver://<host>:<port>;databaseName=<database>;domain=<domain>;useNTLMv2=true;
* JDBC driver type: OTHER
* JDBC driver class name: net.sourceforge.jtds.jdbc.Driver









 Applying enrichments to content from a database 

If you use a database as your data source and want to apply enrichments to the nested fields that are indexed from the database, you must use a Content Mining project type.

If your goal is to create a search application by using a Document Retrieval project type, create a Content Mining project type first. From the Content Mining project, you can connect to the database and enrich the data. Then, you can reuse the enriched collection from a Document Retrieval project.

To enrich database content for use in a Document Retrieval project, complete the following steps:



1. Create a Content Mining project.