vania-janet/multiturn-rag-retrieval-data huggingface dataset:
On the Watson Assistant search integration page, verify that the Discovery instance and project you want to use is selected, then click Next.
2. In the Configure result content section, review the Discovery fields and examples that are used in the search results shown to your customers. You can accept the defaults, or customize them as you want.

The appropriate collection fields to extract data from vary depending on your collection's data source and how the data source was enriched. After you choose a data collection type, the collection field values are prepopulated with source fields that are considered most likely to contain useful information given the collection's data source type. However, you know your data better than anyone. You can change the source fields to ones that contain the best information to meet your needs.

To learn more about the structure of the documents in your collection, including the names of fields that contain information you might want to extract, open the collection in Discovery, and then use the Identity fields and Manage fields tabs.

Each search result can consist of the following sections:



* Title: Search result title. Use the title, name, or similar type of field from the collection as the search result title.

You must select something for the title or no search result response is displayed in the Facebook and Slack integrations.
* Body: Search result description. Use an abstract, summary, or highlight field from the collection as the search result body.

You must select something for the body or no search result response is displayed in the Facebook and Slack integrations.
* URL: This field can be populated with any footer content that you want to include at the end of the search result.

For example, you might want to include a hypertext link to the original data object in its data source. Most online data sources provide self-referencing public URLs for objects in the store to support direct access. If you add a URL, it must be valid and reachable. If it is not, the Slack integration doesn't include the URL in its response and the Facebook integration doesn't return any response.

The Facebook and Slack integrations can successfully display the search result response when the URL field is empty.



You must use a field for at least one of the search results.
* [Stack Overflow ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](http://stackoverflow.com/search?q=watson-iot+ibm-bluemix) - tag your posts with "ibm-cloud" and "watson-iot".





* [IBM developerWorks dW Answers ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://developer.ibm.com/answers/topics/watson-iot/?smartspace=bluemix) - tag your posts with "ibm-cloud" and "watson-iot".



See [Getting help ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://cloud.ibm.com/docs/get-support?topic=get-support-getting-customer-supportasking-a-question) for more details about using the forums.





 Slack 



* [Slack ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://ibm-developers.slack.com/) - join the #internet-of-things channel.







 Support tickets 



* [Contacting support ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://cloud.ibm.com/docs/get-support?topic=get-support-getting-customer-supportusing-avatar)







 Ideas portal 



* [IBM Cloud – Structured Ideas ![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://ibmcloud.ideas.aha.io)
* Oracle Database: 12c, 18c, 19c
* PostgreSQL: 9.6, 10, 11



Support for Data Virtualization was added with IBM Cloud Pak for Data 4.5.x releases
* You must obtain any required service licenses for the data source that you want to connect to. For more information about licenses, contact the system administrator of the data source.







 Prerequisite step 



* Decide which database tables you want to crawl. You can crawl multiple tables in a collection, and you can specify tables that have different schemas, or sets of columns. You must know the following information:



* Schema names
* Table names



For Data Virtualization on IBM Cloud Pak for Data, you can get these details from the IBM Cloud Pak for Data web client. Click the main menu icon, expand Data, and then select Data virtualization. At the start of the page, choose to show Virtualized data.

Zoom

![Shows the Data virtualization view from Cloud Pak for Data](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/connector-db-dv.png)

Figure 1. Virtualized data view in Cloud Pak for Data
* Be careful if you plan to crawl multiple tables that have columns with the same name but different data types. In Content Mining projects, columns with the same name but different data types are assigned to fields that have a data type suffix in the name, such as DATA_string. In all other project types, the data in one of the tables is excluded from the index. For example, if you have two tables that have columns that are called DATA and the DATA column in one table is populated with dates and the column in the other table is populated with strings, the data in one of the tables is excluded from the index.
* Get the user credentials for a user who has permission to access the tables that you want to crawl.
* Before you can connect to a database, you must get the JDBC driver library for the database.
Creating APIs 

You can create APIs in the Lite, Enterprise, and Hybrid plans by downloading the developer toolkit and by using the command line interface (CLI) or API Designer.



 The developer toolkit CLI 

The developer toolkit provides a command line interface that you can use to publish APIs to IBM® API Connect. You publish APIs by including them in a Product and then publishing the Product. You define your APIs and Products by creating and validating YAML definition files in your local file system. You can then interact with IBM® API Connect by using the toolkit commands.





 API Designer 

The API Designer is an offline graphical user interface within the developer toolkit and provides functions for the creation and configuration of APIs. The API Designer is run by using the edit command from the command line interface. When the edit command is used, the API Designer opens in your default browser or can be accessed through the local host port that is displayed when you run the command.





 Installing the developer toolkit 



 Prerequisites 

Before you begin, You must install [Node.js](https://nodejs.org/en/) on the machine on which you want to use the toolkit, and ensure that node is in your PATH.

The following items are installed when you install the developer toolkit.



* API Connect command line tool
* API Connect graphical editor
* Local API Connect Micro-gateway





1. To install the toolkit, open your command prompt as administrator and enter the following command.

npm install -g apiconnect

Note: During installation, you might see errors from the node-gyp module. These errors are from an optional dependency and the installation will complete successfully.
2. To view summary usage help for all the toolkit commands, enter the following command:

apic
3. To view usage help for any command, use the --help option. For example:

apic validate --help









 Installing LoopBack connectors 

Before you can use a LoopBack data source to access data in a backend system such as a database, you must install the data source connector.
* [AbortMultipartUploadRequest](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/AbortMultipartUploadRequest.html)
* [CompleteMultipartUploadRequest](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/CompleteMultipartUploadRequest.html)
* [InitiateMultipartUploadRequest](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/InitiateMultipartUploadRequest.html)
* [InitiateMultipartUploadResult](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/InitiateMultipartUploadResult.html)
* [SdkClientException](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/SdkClientException.html)
* [UploadPartRequest](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/UploadPartRequest.html)
* [UploadPartResult](https://ibm.github.io/ibm-cos-sdk-java/com/ibm/cloud/objectstorage/services/s3/model/UploadPartResult.html)



* Methods
Crawling your data repository 

The Data Crawler is no longer supported or available for download beginning [17 April 2019](https://cloud.ibm.com/docs/discovery?topic=discovery-release-notes17apr19). This content is provided for existing installations only. See [Connecting to Data Sources](https://cloud.ibm.com/docs/discovery?topic=discovery-sources) for other available connectivity options.

After you configure the crawler options, you can run a crawl against your data repository.

Never run the crawler as root, unless you need access to files only root can read.

Run the following command: crawler

The crawler prompts you with documentation that explains what to do. You can run a test crawl, or run a crawl, in addition to other crawl options.



 Running a test crawl 

Run the following command: crawler testit

This runs a test crawl, which crawls only the seed URL and displays any enqueued URLs. If the seed URL results in indexable content (for example, it is a document), then that content is sent to the output adapter, and the content is printed to the screen. If the seed URL retrieval causes URLs to be enqueued, those URLs are displayed, and no content is sent to the output adapter. By default, five enqueued URLs are displayed.

You can also specify a custom configuration file as an option for the crawl command, for example: crawler testit --config [config/myconfigfile.conf]

The path to the configuration file passed in the --config option must be a qualified path. That is, it must be in relative formats, such as config/myconfigfile.conf or ./myconfigfile.conf, or in an absolute path such as /path/to/config/myconfigfile.conf.

Additionally, you can set the limit for the number of enqueued URLs that are displayed as an option for the testit command, for example: crawler testit --limit [number]





 Running a crawl
1. Download the [dictionary-items-month.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/dictionary-items-month.csv) file to your computer. This file contains dictionary terms in CSV format, suitable for uploading into a Knowledge Studio dictionary.
2. Click Assets > Dictionaries.
3. Click the Create Dictionary button to add a dictionary.
4. In the Name field, type Month dictionary and click Save to create the dictionary. The new dictionary is created and automatically opened for editing.
5. In the dictionary pane, click Upload.
6. Select the dictionary-items-month.csv file from your computer and click Upload.

The terms from the file are imported into the dictionary.









 Lesson 2: Adding sample documents 

In this lesson, you will learn how to add documents with linguistic patterns that illustrate the types of rules you want to define.



 About this task 

For more information about adding documents, see [Adding documents for defining rules](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-wks_rule_anno_add).





 Procedure 



1. Download the [documents-new.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-new.csv) file to your computer. This file contains example documents suitable for uploading.
2. Click Rule-based Model > Rules.
3. Click the Add a document icon, which is next to the Documents page heading.
4. Click the Upload CSV file tab.
5.
print('usage: %s <endpoint-url> <apikey> <instance-guid>' % sys.argv[0])
print('endpoint-url: The endpoint URL that should point to IBM Cloud')
print('apikey: IBM Cloud IAM apikey that will be used to retrieve an access token')
print('instance-guid: GUID of an IBM Cloud Monitoring with monitoring instance')
sys.exit(1)

if len(sys.argv) != 4:
usage()

URL = sys.argv[1]
APIKEY = sys.argv[2]
GUID = sys.argv[3]

DASHBOARD_NAME = 'My New Dashboard from Python'
PANEL_NAME = 'CPU Over Time'

 Instantiate the client
ibm_headers = IbmAuthHelper.get_headers(URL, APIKEY, GUID)
sdclient = SdMonitorClient(sdc_url=URL, custom_headers=ibm_headers)

 Create an empty dashboard
ok, res = sdclient.create_dashboard(DASHBOARD_NAME)

 Check the result
dashboard_name = None
if ok:
print('Dashboard %d created successfully' % res['dashboard'])
dashboard_name = res['dashboard']
else:
print(res)
sys.exit(1)

 Add a time series panel
panel_type = 'timeSeries'
metrics = [
{'id': 'proc.name'},
{'id': 'cpu.used.percent', 'aggregations': {'time': 'avg', 'group': 'avg'}}
]
ok, res = sdclient.add_dashboard_panel(
dashboard_name, PANEL_NAME, panel_type, metrics)

 Check the result
if ok:
print('Panel added successfully')
dashboard_name = res['dashboard']
else:
The App Configuration SDK handles the automatic delivery of the appropriate flag state or property value into your application. It connects to the endpoints provided by the App Configuration API, fetches collections, and evaluates segment and targeting rules. Server-side SDKs connect to the App Configuration service through a web socket for real-time updates. Client-side SDKs pull values from the App Configuration service upon a lifecycle change such being opened or brought to the foreground. SDKs are available in various client-side and serve-side languages, and more languages are being added all the time.
You must upload the type system that was downloaded from the other workspace in JSON format before you can upload the corresponding dictionary file. You can edit and add entries to a dictionary that you reuse from another Knowledge Studio workspace. See [Uploading resources from another workspace](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-exportimport) for more details.

Uploading a CSV file is also supported, but uploading it directly as a dictionary creates a preview-only dictionary that you cannot edit or download. To upload a CSV file that you can edit and download, click Create Dictionary to first create an empty dictionary, and then upload the CSV content as entries to that newly created dictionary.



* Click the Create Dictionary button to create an empty dictionary to which you can subsequently add dictionary entries. Specify a descriptive name for the dictionary, and then click Save.



3. To add entries to the dictionary, perform one of the following tasks:



* Click Add Entry to add a dictionary entry. Specify the lemma (the most representative word form for the term).
* Click Upload to upload a CSV file that contains dictionary entries, and then browse to select the file. The CSV file must be smaller than 1MB.
* [Add suggested entries](https://cloud.ibm.com/docs/watson-knowledge-studio?topic=watson-knowledge-studio-dictionariesadding-dictionary-suggestions) from the list provided in the Entry Suggestions pane. Knowledge Studio uses the documents in your workspace and the existing entries in your dictionary to suggest new entries.



4. After uploading or adding entries, you can edit the entries.

Open an entry to specify equivalent terms, called surface forms. Each surface form must be 256 or fewer characters in length. You can change which of the surface forms is used as the lemma. For example, the lemma IBM might have surface forms like IBM Corp. and International Business Machines, Inc..
5. Select the appropriate part of speech for each lemma and surface form in the dictionary.
Resolve those errors and run the script until you are able to get a valid response and generate a plan.
8. If you want to check the logs of the action, see [Retrieving action logs with Schematics API](https://cloud.ibm.com/docs/hpc-spectrum-symphony?topic=hpc-spectrum-symphony-retrieve-action-logs) to retrieve the logs. The response contains the status of the action, and it appears in either a COMPLETED or FAILED state.





 Example Python request 

 Generate a plan using Schematics Python API

import json, logging

from ibm_cloud_sdk_core.authenticators import IAMAuthenticator
from ibm_schematics.schematics_v1 import SchematicsV1

logging.basicConfig()
logging.root.setLevel(logging.NOTSET)
logging.basicConfig(level=logging.NOTSET)

authenticator = IAMAuthenticator('<ibmcloud-api-key>')

schematics_service = SchematicsV1(authenticator = authenticator)

schematics_service.set_service_url('https://us.schematics.cloud.ibm.com')

logging.info("Started Generating Schematic Plan")

workspace_activity_plan_result = schematics_service.plan_workspace_command(
    w_id='<workspace id>',
    refresh_token='<refresh-token>'
).get_result()

print(json.dumps(workspace_activity_plan_result, indent=2))

logging.info("Completed Generating Schematic Plan")





 Example Python response 

INFO:root:Started Generating Schematic Plan
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): iam.cloud.ibm.com:443
DEBUG:urllib3.connectionpool:https://iam.cloud.ibm.com:443 "POST /identity/token HTTP/1.1" 200 985
[List of repositories](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution24-github-traffic-analytics/RepositoryList.png)

List of repositories
5. For testing, click on Administration, then Collect statistics. It retrieves the traffic data on demand. Thereafter, click on Repositories and Daily Traffic. It should display collected data.

Zoom

![Traffic data](https://cloud.ibm.com/docs-content/v1/content/f84e2238288b7f0355715a4c355609ce7ac48fb0/solution-tutorials/images/solution24-github-traffic-analytics/RepositoryTraffic.png)

Traffic data







 Step 6: Set up daily data retrieval (shell) 

With the app in place and configured, the last part is to initiate daily retrieval of GitHub traffic data. You are going to [create a cron subscription](https://cloud.ibm.com/docs/codeengine?topic=codeengine-clicli-subscription-cron-create). Similar to a [cron job](https://en.wikipedia.org/wiki/Cron), the app subscribes to events on the specified schedule (eventing).



1. Create the cron subscription ghstats-daily with a daily schedule at 6 am UTC with a POST event at the path /collectStats. Replace SECRET_TOKEN_AS_IDENTIFIER with your chosen secret value. It is used to identify the event giver to the app.

ibmcloud ce subscription cron create --name ghstats-daily --destination ghstats-app --path /collectStats --schedule '0 6 * * ' --data '{"token":"SECRET_TOKEN_AS_IDENTIFIER"}' --content-type application/json
import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

service.postSearch({
db: '<db-name>',
ddoc: '<ddoc>',
index: '<index-name>',
query: ':',
includeDocs: true,
limit: 1
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.post_search(
db='<db-name>',
ddoc='<ddoc>',
index='<index-name>',
query=':',
include_docs=True,
limit=1
).get_result()

print(response)

postSearchOptions := service.NewPostSearchOptions(
"<db-name>",
"<ddoc>",
"<index-name>",
":",
)
postSearchOptions.SetIncludeDocs(true)
postSearchOptions.SetLimit(1)

searchResult, _, err := service.PostSearch(postSearchOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(searchResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)



 Query Parameters
.getResult();

System.out.println(response);

import { CloudantV1 } from '@ibm-cloud/cloudant';

const service = CloudantV1.newInstance({});

const selector: CloudantV1.Selector = {
ts: {'$gte': '20181213'}
}
service.postPartitionFind({
db: 'readings',
partitionKey: 'bridge-9876',
selector: selector
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.post_partition_find(
db='readings',
partition_key='bridge-9876',
selector={'ts': {'$gte': '20181213'}}
).get_result()

print(response)

selector := map[string]interface{}{
"ts": map[string]string{
"$gte": "20181213",
},
}

postPartitionFindOptions := service.NewPostPartitionFindOptions(
"readings",
"bridge-9876",
selector,
)

findResult, response, err := service.PostPartitionFind(postPartitionFindOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(findResult, "", " ")
fmt.Println(string(b))

The previous Go example requires the following import block:

import (
"encoding/json"
"fmt"
"github.com/IBM/cloudant-go-sdk/cloudantv1"
)
done < <(list_repos)

Outputs the following lines to the console:

Repository saved as 'my-frontend' is at: 'github.com/my-team/frontend'
Repository saved as 'my-backend' is at: 'github.com/my-team/backend'





 save_result 

 <stage>: Stage name e.g. test, detect-secrets, static-scan
 <path>: Path where will be stored the file, string
save_result <stage> <path>

Saves an arbitrary test, scan result file for a stage. Later this file can be retrieved with load_result. By default, data is saved with the workspace-relative path as key.

Using the PIPELINECTL_USE_PATH_AS_KEY feature flag, data is saved with the provided path as the key.

Example:


 save the contents of the file ./results/mocha_results.json
 as an entry named "mocha_results.json" for the "test" stage

save_result test ./results/mocha_results.json


 save the contents of the file ../data/coverage.xml
 as an entry named "coverage.xml" for the "test" stage

save_result test ../data/coverage.xml


 Using the PIPELINECTL_USE_PATH_AS_KEY environment variable
 save the contents of the file ../data/coverage.xml
 as an entry named "../data/coverage.xml" for the "test" stage

PIPELINECTL_USE_PATH_AS_KEY=1 save_result test ../data/coverage.xml
Show more





 list_results 

 <stage>: Stage name
list_results <stage>

Lists the saved file names for a stage.

Example:

list_results test
 mocha_results.json
 coverage.xml





 load_result 

 <stage>: Stage name e.g. test, detect-secrets, static-scan
 <file>: File name e.g. mocha_results.json
load_result <stage> <file>
end = 0

 Sampling time:
 - for time series: sampling is equal to the "width" of each data point (expressed in seconds)
 - for aggregated data (similar to bar charts, pie charts, tables, etc.): sampling is equal to 0
sampling = 60

 Load data
ok, res = sdclient.get_data(metrics, start, end, sampling, filter=filter)
if ok:
print(res)
Show more

You must include the following information: <MONITORING-ENDPOINT>, <IAM_APIKEY>, and <GUID> These data is required to authenticate the request with the monitoring instance. To get the monitoring instance information, see [Authenticate your user or service ID by using IAM](https://cloud.ibm.com/docs/monitoring?topic=monitoring-python-clientpython-client-iam-auth).





 Samples 

For Python examples, see any of the following examples:



* [Example 1](https://github.com/draios/python-sdc-client/blob/master/examples/get_data_simple.py): This sample script shows how to get data by creating a request that has no filter and no segmentation.
* [Example 2](https://github.com/draios/python-sdc-client/blob/master/examples/get_data_advanced.py): This sample script shows how to get data by creating a request that has a filter and segmentation.
* [Example 3](https://github.com/draios/python-sdc-client/blob/master/examples/get_data_datasource.py): This sample script shows how to get data by creating a request where you specify the datasource.







 Metrics dictionary 

To see the pre-defined metrics, see [Metrics dictionary](https://docs.sysdig.com/en/metrics-dictionary.html).