vania-janet/multiturn-rag-retrieval-data huggingface dataset:
--type or -t Optional The type of resource that you want to retrieve. Supported values are vsi. 
 --query-file or -f Optional The path to the JSON file where you specified the details of your resource query. To find a list of supported queries, see [Supported resource queries](https://cloud.ibm.com/docs/schematics?topic=schematics-inventories-setupsupported-queries). 
 --file or -f Optional The path to the JSON file that specifies the details of the resource query that you want to create. 
 --output or -o Optional Returns the command-line output in JSON format. Currently only JSON file format is supported. 
 --no-prompt Optional Set this flag to create the resource query without an interactive command-line session. 





 Using the payload file 

You can provide a payload file to specify certain parameters for the resource_query create command. Then, you pass the file name to the command by using the --file command option. For a list of supported resource queries, see [Supported resource queries](https://cloud.ibm.com/docs/schematics?topic=schematics-inventories-setupsupported-queries).

You need to replace the <...> placeholders with the actual values. For example, "<WORKSPACE_ID" as us-east.workspace.ID1231.

Syntax

[{
"query_type": "workspaces",
"query_condition":
{
"name": "workspace-id",
"value": "<WORKSPACE_ID>",
"description": "string"
},
{
"name": "resource-name",
"value": "<RESOURCE_NAME>",
"description": "string"
}
]
}]

Example

[{
"query_type": "workspaces",
"query_condition":
{
"name": "workspace-id",
This stricter query matches only those documents in which there is an entities object with text equal to Gilroy and type equal to Location.

As another example, if you want to match documents that contain an entities object with text equal to Gilroy but typenot equal to Location, you can use the not equal operator in the query, for example:

{
"filter": "enriched_text.entities:(text::Gilroy,type::!Location)"
}

You can also use aggregations to do more sophistocated filtering of the results. For more information about the available aggregation types, see [Query aggregations](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-aggregations).

For more information about the Discovery Query Language, see the following topics:



* [Query parameters](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-parameters)
* [Query operators](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-operators)
You can use the [expansions.json](https://watson-developer-cloud.github.io/doc-tutorial-downloads/discovery/expansions.json) file as a starting point when you build a query expansion list.
2. From the navigation pane, open the Improve and customize page.
3. Expand Improve relevance from the Improvement tools pane.
4. Click Synonyms, and then click Upload synonyms for the collection.

Do not upload a synonyms file while documents are being added to your collection. The ingestion processing that occurs when documents are added can cause the index to be unavailable.

Only one synonyms list can be uploaded per collection. If a second expansion list is uploaded, the second list replaces the first.
5. Run a test query to verify that the query expansion is working as expected.

Query expansions are applied at query time, not during indexing, so you can add synonyms without reprocessing your collection.



To disable query expansion, delete the synonyms file. However, do not delete a synonyms file while new documents are being processed.
Other information, such as the file name, is stored in nested fields with names like extracted_metadata.filename. You can determine whether a field is a nested field by its name. If the field name includes a period, it is a nested field. For more information about how different file types are handled, see [How your data source is processed](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-index-overview).





 Which type of query should I use in my custom app? 

When you submit a query, you can choose to submit a natural language query or use the Discovery Query Language to customize the search to target specific fields in the index, for example. For more information about the different types of queries and how to decide which one to use, see [Choosing the right query type](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-query-conceptsquery-choose).
For example, given the key/field pair {"borrower": "GOVERNMENT OF TIMOR"}, query-borrower:"GOVERNMENT OF TIMOR" returns results, but query-borrower:"GOVERNMENT OF TIOR" does not. Using a wildcard is not applicable within phrases because all of the characters within the quotation marks (") of a phrase are escaped.





 24 March 2017 

New feature for My data insights : Added filtering to the "My data insights" screen in the Discovery tooling.





 15 March 2017 

Known issues : All fields that are ingested from HTML, PDF, and Word documents are typed as string. JSON fields and calculated fields, such as sentiment score, are typed as defined. [Update](https://cloud.ibm.com/docs/discovery?topic=discovery-addcontentadding-content-with-the-api-or-tooling) : The preview operation does not currently check for nested JSON arrays within a submitted JSON document. The service does not currently support nested JSON arrays, so a document with nested arrays can successfully pass the preview operation but fail upon an ingestion attempt. See [Can I upload JSON arrays?](https://cloud.ibm.com/docs/discovery?topic=discovery-troubleshootfaq-array) : If you encounter ingestion errors with the message unsupported text language, update your configuration with the "language": "english" enrichment option to force all text to be interpreted as English, as shown in the following example.


"enrichments": [
{
"enrichment": "alchemy_language",
"source_field": "author.label",
"options": {
"extract": "taxonomy,entity,relation,doc-emotion,doc-sentiment,concept,keyword",
"sentiment": true,
"quotations": true,
"language": "english"
}
}
]


Fixed known issue : Improved performance and stability of the service.





 8 March 2017 

Improved back end performance : Optimized the back end, including the addition of new timeouts, to improve overall performance.
[Shows that multiple responses are returned for the query.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-rule-response.png)

Figure 26. Multiple responses are returned for the query

Our updates only improved the quality of the accurate responses that were returned before.
3. Now, let's ask a question that returned poor results previously. Enter What are PTFs? as the search query.

The same response that was returned as the only response last time is returned again. However, this time we get more than one response. And we can see that the second response that is returned defines the acronym for us.

(“principal trading firms” or “PTFs”)

Zoom

![Shows responses that are returned to answer the question about PTFs.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-ptf-responses.png)

Figure 27. Responses that answer the question about PTFs
4. Let's try the other problematic search query. Enter Where do muni bond trades get reported to? as the search query.

This time it's the third response that provides an answer to the question. You must view the full passage to see the entire definition.

Zoom

![Shows responses that are returned to answer the question about muni bonds.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-muni-responses.png)

Figure 28.
Consider a naming convention similar to your Cloud Identity and Access Management access groups for ease of management.
9. Specify the Access Scope.

The access scope is defined as a search query. The query uses the following format.





Table 2. Example access scope search queries

 Example Query Behavior Example Matches 

 level:error Case-insensitive prefix match Error, error, errors 
 level:=error Case-sensitive prefix match error, errors 
 level:==error Case-insensitive exact term match error, Error 
 level:===error Case-sensitive exact term match error 
 level:[warning,error] Case-insensitive list of prefixes warning, Warning, Warnings, error, ERROR, errors 
 level:===[warning,error] Case-sensitive list of prefixes warning, error 
 level:* Matches if the field exists All lines containing the field level 



For example, if you have two apps (myapp and myapp1), then a service group with an access scope of app:myapp will allow access to data from both apps. If the access scope is app:===myapp, then users in the group will only be able to access data from the myapp app.

If you want to create a group of administrators with access to all data, specify host:* for the Access Scope.





 Editing or deleting service groups 

Complete the following steps to edit or delete a service group:



1. Log in to your IBM Cloud account.

Click [Log in to IBM Cloud](https://cloud.ibm.com/login) to sign in to the IBM Cloud.

After you log in with your user ID and password, the IBM Cloud console opens.
2. Select your account.
3. Click the Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis//images/icon_hamburger.svg) > Observability.
4. Click Logging.
5.
* nested(enriched_text.entities).filter(enriched_text.entities.disambiguation.subtype::City).term(enriched_text.entities.text,count:3) aggregates the top 3 entities of subtype City
* filter(enriched_text.entities.disambiguation.subtype::City).term(enriched_text.entities.text,count:3) returns the top 3 entities where the result contains at least one entity of subtype City.











 Querying Watson Discovery News 

Discovery News, a public data set that is pre-enriched with cognitive insights, is also included with Discovery. See [Watson Discovery News](https://cloud.ibm.com/docs/discovery?topic=discovery-watson-discovery-news) for more information about this collection.

You can query this collection using natural language queries, for example "IBM Watson partnerships", or the Discovery Query Language. To learn more about natural language queries, see the [Natural language query](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parametersnlq).

You cannot adjust the Discovery News configuration, train, or add documents to Discovery News collection.

The English, Spanish, German, Korean, French, and Japanese language Watson Discovery News collections are available from both the Discovery tooling and the API.

The default language of Watson Discovery News in the tooling is English. To switch the language, you must first click the ![Manage Data](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery//images/icon_yourData.png) icon, then choose the appropriate language from the drop-down.

For information about querying a collection via the API, see [API Reference](https://cloud.ibm.com/apidocs/discoveryquery-your-collection). The collection_id of the English language version of Watson Discovery News is news-en. Formerly, the collection_id was news.
You can define query parameters to refine the events that you want to export.

The following table lists the query parameters that you can set:



Query parameters

 Query parameter Type Status Description 

 from int32 Required Start time. Set as UNIX timestamp in seconds or milliseconds. 
 to int32 Required End time. Set as UNIX timestamp in seconds or milliseconds. 
 size string Optional Number of event lines to include in the export. 
 hosts string Optional Comma-separated list of services.<br><br>To get the value that you can set for a specific service, you can check the label Source in an event that is generated by the service. 
 apps string Optional Comma-separated list of service instances represented as CRNs. <br>To get the value that you can set for a specific service, you can check the label App in an event that is generated by the service. 
 levels string Optional Comma-separated list of severities. <br>Use this field to set the severity of the events that you want to export. 
 query string Optional [Search query](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-viewsviews_step2). <br>You can test your query in a view in the UI. Then, when you copy the query to use for the export, replace each blank space in the query with %20. 
 prefer string Optional Defines the log lines that you want to export. Valid values are head, first log lines, and tail, last log lines. If not specified, defaults to tail. 
 pagination_id string Optional Indicates which page of results is retrieved from an export. For the initial export request, this parameter should be omitted. Subsequent requests for pagination should provide the token sent in the response to this parameter. 



For example, you can define a set of parameters to include information:

ENDPOINT/v2/export?to=START_TIME&from=END_TIME&hosts=LIST_OF_HOSTS<=vels=LIST_OF_LEVELS&size=N&query=(SEARCH_QUERY)" -u $TOKEN:





 Exporting events
curl -u "4ba1624f-48b6-484a-8e32-18d1c205c1fa":"qUy3B0CbGf9G" "{url}/v1/environments/a56dd9b4-040b-4ea3-a736-c1e7a467e191/collections/99040100-fe6a-4782-a4f5-28f9eee30850/training_data?version=2019-04-30"

The JSON that is returned is in the following format:

{
"query_id": "484baad4-d440-44b7-a0dd-70bb2ac77baa",
"natural_language_query": "why is the sky blue",
"filter": "text:meteorology",
"examples": [
{
"document_id": "9afcf06e-2ef7-438e-b2a5-e4d9432dc877",
"relevance": 0
},
{
"document_id": "54f95ac0-3e4f-4756-bea6-7a67b2713c81",
"relevance": 1
},
{
"document_id": "01bcca32-7300-4c9f-8d32-33ed7ea643da",
"cross_reference": "my_id_field:1463",
"relevance": 4
},
{
"document_id": "46b52fdc-3a72-4700-9849-ac217532b373",
"relevance": 3
}
]
}
Show more
4. Check the status of the training by using the [List collection details](https://cloud.ibm.com/apidocs/discoveryget-collection-details) method and looking at the value of the "training"/"available" field.
Query performance 

Query performance in Discovery is dependent on a number of factors.



 Evaluating performance 

If you expect high simultaneous load on your application, it is important to evaluate the performance of your Discovery application. There are several dimensions commonly used to measure performance:



1. Response latency - The time it takes for Discovery to return the results of your query.



* Latency can vary depending on a number of factors, see [Factors that impact performance](https://cloud.ibm.com/docs/discovery?topic=discovery-qpperffactors) for more information.
* It is important to run tests in your production environment using a representative set of queries to determine your average latency.



2. Query rate - The number of requests that can be processed in a given time, usually measured in Queries per Second (QPS). This measure is most important when you expect high simultaneous load on your application.



* The rate that can be achieved varies based on many of the same factors as latency.
* Evaluate the query rate by using representative queries at the loads you expect to see. Remember to account for peak loads during testing.









 Factors that impact performance 

Query performance in Discovery varies, based on many factors, including plan size, query complexity, features used, and collection size and complexity.



 Plan size 

Discovery pricing plans limit the number of available documents and also provide differing ability to handle query load. The larger the plan size, the more resources that are available to handle queries. The average rate limits also vary by plan size. Upgrading your plan can improve performance. See [Discovery pricing plans](https://cloud.ibm.com/docs/discovery?topic=discovery-discovery-pricing-plans) for plans and limits.





 Query complexity 

There are many ways to run queries against Discovery and the different operations used can impact performance. These query characteristics can impact performance:



1. Length - Longer queries most likely have poorer performance.
2. Aggregations - Aggregations are a more complex query type, with nested aggregations having the highest impact on performance.
3. Operators - Wldcarding and fuzzy matching can impact performance.
4.
"minimum_queries_added": true
}
}

To satisfy the requirements for training, the following must all be true:



* minimum_queries_added
* minimum_examples_added
* sufficient_label_diversity



In the response:



* successfully_trained is the last time training was successfully completed. 
* available: true indicates that training occurred and a model is available.
* processing: true indicates that training is in progress.
* If the data_updated date is later than the successfully_trained date, then a new model has not been trained since a recent data upload.







 How do I check errors and warnings? 

You can use the [Query notices API](https://cloud.ibm.com/apidocs/discoveryquery-system-notices) to view errors or warnings.

curl -u "apikey":"{apikey_value}" {url}/v1/environments/{environment_id}/collections/{collection_id}/notices?version=2019-04-30"

Replace {apikey} and {url} with your API key and URL.

Other API operations are listed in [Performing other training-data query operations](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-apitraining-data-operations).





 How do I interpret the confidence score that appears in natural language query results after training? 

See [Confidence scores](https://cloud.ibm.com/docs/discovery?topic=discovery-improving-result-relevance-with-the-toolingconfidence) for more information.







 Interpreting Errors and Warnings 

Explanations of common error and warning messages.



 Warning: Invalid training data found: The document was not returned in the top 100 search results for the given query, and will not be used for training 

This warning is caused by the document_ids in your training data not matching the document_ids in a search performed against the collection. Check your queries, and make sure that the document_id of the document you are rating is returned in the top 100 results for that query.
You can adjust the fields in the documents over which passage retrieval searches with the [passages.fields](https://cloud.ibm.com/docs/discovery?topic=discovery-query-parameterspassages_fields) parameter.

The passages parameter returns matching passages (passage_text), as well as the score, document_id, the name of the field the passage was extracted from (field), and the starting and ending characters of the passage text within the field (start_offset and end_offset), as shown in the following example. The query is shown at the top of the example.

curl -u "apikey":"{apikey_value}" "{url}/v1/environments/{environment_id}/collections/{collection_id}/query?version=2019-04-30&natural_language_query='Hybrid%20cloud%20companies'&passages=true"

Replace {apikey} and {url} with your API key and URL.

The returned JSON is in the following format:

{
"matching_results":2,
"passages":[ ! ! ! ! ! ! ! !
{
"document_id":"ab7be56bcc9476493516b511169739f0",
"passage_score":15.230205287402338,
"passage_text":"a privately held company that provides hybrid cloud recovery, cloud migration and business continuity software for enterprise data centers and cloud infrastructure."
"start_offset":120
"end_offset":300
"field":"text"
},
{
"passage_text":"Disaster Recovery Services for Hybrid Cloud</title></head>n<body>nnn<p>Published: Thu, 27 Oct 2016 07:01:21 GMT</p>n"
"passage_score":10.153470191601558,
"document_id":"fbb5dcb4d8a6a29f572ebdeb6fbed20e",
Local name of the <from list item>, that is scoped within the select statement or the extract statement. A local name can be a simple identifier or a double-quoted identifier. Local names that contain spaces, punctuation characters, or AQL keywords must be enclosed in double quotation marks.







 Examples 

Example 1: A from list with a view and a nested statement

This example shows a from list that references a view and a nested extract statement. The example assigns the result of the statement to the local name FN. The example also assigns the outputs of the LastName view to the local name Last Name.

create dictionary LastNamesDict as
(
'Anthony', 'Lehman', 'Radcliff'
);

create view LastName as
extract dictionary 'LastNamesDict'
on D.text as lastname
from Document D;

create view FromList as
select 
from
(extract dictionary 'first.dict' on D.text
as firstname from Document D) FN,
LastName "Last Name"
where Follows(FN.firstname,
"Last Name".lastname, 0, 1);
Show more

The following names are contained in the external dictionary first.dict:

Dictionary for given names
Aaron
Candra
Freeman
Mathew
Matthew
Zoraida







 The where clause 

The optional where clause defines a predicate to be applied on each tuple generated from the Cartesian product of all tuples in the relations in the from clause.



 Syntax 

select <select list>
from <from list>
[where <where clause>]





 Description 



* <where clause>

Specifies one or more predicates. A join occurs when any predicate in a where clause involves fields from more than one view that belongs to the from list. This predicate must be a conjunction of a set of built-in predicate functions or other user-defined functions that return the data type Boolean:

function1() and function2()
and ... and functionn()
Command options



Schematics resource query get flags

 Flag Required / Optional Description 

 --id or -i Required The ID of the resource query that you want to retrieve. 
 --profile or -p Optional The depth of information that you want to retrieve. Supported values are detailed and summary. The default value is summary. 
 --output or -o Optional Specify the output format. Only JSON format is supported. 
 --no-prompt Optional Set this flag to retrieve a resource query without an interactive command-line session. 



Example

ibmcloud schematics resource-query get --id us-east.INVENTORY.inventoryid12342





 ibmcloud schematics resource query list 

Retrieve a list of all Schematics resource queries in your account.

Syntax

ibmcloud schematics resource-query list [--limit LIMIT] [--offset OFFSET] [--output OUTPUT]

Command options



Schematics resource query list flags

 Flag Required / Optional Description 

 --limit or -l Optional The maximum number of resource queries that you want to list. The number must be a positive integer between 1 and 200. The default value is -1. 
 --offset or -m Optional The position of the resource query in the list of resource queries. For example, if you have three resource queries in your account, the command returns these resource queries as a list with three elements. To see a specific resource query in this list, you must enter the position number that the resource query has in the list. To list the first resource query in the list, enter 0. To list the second resource query, enter 1 and so forth. Negative numbers are not supported and are ignored. The default value is -1. 
 --output or -o Optional Specify the output format. Only JSON format is supported. 



Example

ibmcloud schematics resource-query list --output listoutput.json





 ibmcloud schematics resource query update 

Update or replace a resource query creates a copy of an resource query and relaunches an existing resource query by updating the information of an existing IBM Cloud Schematics resource query.

Syntax