vania-janet/multiturn-rag-retrieval-data huggingface dataset:
The query first selects the documents that have at least one entity of type Location and whose text is Gilroy. This action returns 3 documents. From the returned documents, the aggregation then counts the number of documents in which each entity type appears.

{
"matching_results": 3,
"retrieval_details": {
"document_retrieval_strategy": "untrained"
},
"aggregations": [
{
"type": "term",
"field": "enriched_text.entities.type",
"results":
{
"key": "Location",
"matching_results": 3
},
{
"key": "Person",
"matching_results": 3
},
{
"key": "Company",
"matching_results": 2
},
{
"key": "GeographicFeature",
"matching_results": 2
},
{
"key": "Organization",
"matching_results": 2
},
{
"key": "Quantity",
"matching_results": 2
},
{
"key": "Facility",
"matching_results": 1
},
{
"key": "PrintMedia",
"matching_results": 1
}
]
}
]
}
Show more

The 3 matching documents all have a Location and a Person entity type ("matching_results": 3). However, only 2 of the matching documents have a Company entity type.

By default, the top 10 matches are returned, sorted by relevance. You can change the number of results by adding the count parameter to the aggregation.

{
"filter": "enriched_text.entities:(text::Gilroy,type::Location)",
"aggregation": "term(enriched_text.entities.type,count:20)"
}



 Add a filter
For example, a dictionary used to extract information about people with a military rank such as Officer, Warrant Officer, or Chief Warrant Officer might result in three overlapping matches for every occurrence of Chief Warrant Officer. See [Eliminate duplicate and overlapping results](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractorseliminate-duplicate-and-overlapping-results) for information needed to address this issue.



 Defining a list 

When using standardized terms such as names of departments in your organization, sales priorities, congressional committees, or military ranks to extract values, you can identify the needed terms in a list.



1. Click ![New Dictionary](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/new_extractor_dictionary.png) on the canvas toolbar.
2. Specify a name for the dictionary. You can extend the name with a description. Lengthy text might be truncated in the display but you can hover over the dictionary to read the full description.
3. On the Settings tab under Extractor Properties, enter each term manually or load a list from a file. The maximum length for each entry is 1024 characters.



* To specify the terms manually, click ![Add Term](https://cloud.ibm.com/docs-content/v1/content/148d3bd95f946aa1bb53ea1540475f522e6b61c9/watson-knowledge-studio-data/images/add_term.jpg), enter the term, and press Enter. Enter one term per row.
* To load terms from a UTF-8 text file with each term on a separate line, !
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(searchInfoResult, "", " ")
fmt.Println(string(b))

The JSON structure includes the following individual fields:

name
: Name or ID of the Search within the design document.

search_index
: The Search Index

pending_seq
: The sequence number of changes in the database that reached the Lucene index, both in memory and on disk.

doc_del_count
: Number of deleted documents in the index.

doc_count
: Number of documents in the index.

disk_size
: The size of the index on disk, in bytes.

committed_seq
: The sequence number of changes in the database that were committed to the Lucene index on disk.

See the following example response in JSON format:

{
"name":"_design/app/description",
"search_index":{
"pending_seq":63,
"doc_del_count":3,
"doc_count":10,
"disk_size":9244,
"committed_seq":63
}
}
"bookmark":"g...",
"rows":[...],
"ranges": {
"price": {
"expensive": 278682,
"cheap": 257023
}
}
}







 Geographical searches 

In addition to searching by the content of textual fields, you can also sort your results by their distance from a geographic coordinate.

To sort your results in this way, you must index two numeric fields that represent the longitude and latitude.

You can then query by using the special <distance...> sort field, which takes five parameters:



* Longitude field name - The name of your longitude field (mylon in the example).
* Latitude field name - The name of your latitude field (mylat in the example).
* Longitude of origin - The longitude of the place you want to sort by distance from.
* Latitude of origin - The latitude of the place you want to sort by distance from.
* Units - The units to use include, km for kilometers or mi for miles. The distance is returned in the order field.



You can combine sorting by distance with any other search query, such as range searches on the latitude and longitude, or queries that involve nongeographical information.

That way, you can search in a bounding box, and narrow down the search with extra criteria.

See the following example geographical data:

{
"name":"Aberdeen, Scotland",
"lat":57.15,
"lon":-2.15,
"type":"city"
}

See the following example of a design document that includes a search index for the geographic data:

function(doc) {
if (doc.type && doc.type == 'city') {
index('city', doc.name, {'store': true});
index('lat', doc.lat, {'store': true});
index('lon', doc.lon, {'store': true});
}
}
! for more information.<-- </section "id="section-dialog-runtime-disable-digressions" "> --><-- <section "id="section-dialog-runtime-digression-tutorial" "> --> Digression tutorial Follow the tutorial]] ! ! ! ! ! ! ! ! !
You can use many enrichments together to tackle various challenges that you might encounter as you develop a search application.

Many teams start by creating a dictionary enrichment. Dictionaries are a great tool for identifying important terms and tagging them so they can be retrieved later. Let's say you're building a search application that needs to extract ingredients from recipes. A dictionary enrichment can recognize mentions of most ingredients. However, the dictionary enrichment might partially match against two-word terms. For terms such as olive oil or mustard greens, it might incorrectly recognize only olive and mustard. To improve the accuracy of the search, you can augment the dictionary enrichment with a pattern enrichment that can recognize two-word ingredient mentions. Maybe a few recipes mention food coloring codes in European format (E104). You can add a regular expression enrichment to recognize occurrences of codes with the syntax E1nn. Finally, to catch terms that no other enrichment can recognize, you can use a machine learning enrichment. The enrichment can be one that you build in an external tool and import to Discovery or one that you build in Discovery by creating an entity extractor enrichment.

The entity extractor enrichment is more sophisticated than the other enrichments. For example, a dictionary enrichment recognizes only exact matches of dictionary terms and synonyms that occur in your documents. A regular expression enrichment recognizes only specific patterns. In contrast, occurrences of an entity are recognized based on the context in which an entity example is mentioned in a sentence.

For example, maybe you want to recognize locations and the document you want to process contains the following types of sentences:



* I live in Massachusetts.
* We're traveling from New York City to Paris next week.



To use a dictionary enrichment to recognize location names successfully, the dictionary must list every possible location. However, if you use an entity extractor enrichment, you can identify when a location is mentioned based on how the location is referenced in a sentence. With phrases such as, “I live in x” or “I'm from x” or “I'm traveling to x” in its training data, the entity extractor can learn that x is a reference to a location.

When you need to choose between using a dictionary or an entity extractor enrichment, follow these guidelines:
If documents that were used previously to train the model are removed from a collection, you must remove any references to them from the training data before you start to retrain the model. The model expects both the documents and queries from training data pairs to continue to exist. To remove these references, delete the training queries that returned the deleted documents. If the queries continue to be relevant, you can add them back to the training data and pair them with other documents.

For more information about the relevancy training API, see the [API reference documentation](https://cloud.ibm.com/apidocs/discovery-datacreatetrainingquery).



 When to use relevancy training 

Relevancy training is optional. Test the quality of your search results. If the results of your queries meet your needs, no further training is necessary.

The training improves the relevancy of the documents that are returned in query responses. It does not improve the passages or answers that are returned per document. If you're using passage retrieval and your test results are returning good documents, but the wrong passages from the documents, relevancy training will not help.

For more information about when to use relevancy training, read the [Relevancy training for time-sensitive users](https://medium.com/ibm-data-ai/ibm-watson-discovery-relevancy-training-time-sensitive-18a190b0874a) blog post on Medium.





 How fields are handled 

When you train a project from the product user interface, the results are always taken from the text field of the documents. If your documents don't have a text field, use the API to train your project instead. Your documents might not have a text field if you uploaded a CSV file that doesn't have a column named text, or uploaded a JSON file that doesn't have an object named text, or if you used the Smart Document Understanding tool to define fields with other names in which the bulk of the content of your documents now are stored.

When you train a project from the API, results are taken from all of the root-level fields and they are all considered to have equal significance.
Querying the view for a specific key then provides all the documents with that tag.





It all comes down to the number of HTTP requests that your application makes. There's a cost to opening HTTP connections, particularly HTTPS. While reusing connections helps, making fewer requests overall speeds up the rate that your application can process data.

As a side benefit, when you use de-normalized documents and pre-computed views, you often have the value that your application requires generated ahead of time. Rather than it being constructed while in progress at query time.





In conflict with the advice to de-normalize your data is this advice, use fine-grained documents to reduce the chance of concurrent modifications that create conflicts. This practice is somewhat like normalizing your data. There's a balance to strike between reducing the number of HTTP requests and avoiding conflicts.

For example, see the medical record that includes a list of operations:

{
"_id": "Joe McIllness",
"operations": [
{ "surgery": "heart bypass" },
{ "surgery": "lumbar puncture" }
]
}

If Joe is unfortunate enough to have lots of operations at the same time, the concurrent updates to a document are likely to create conflicted documents. Better to break out the operations into separate documents, which refer to Joe's person document, and use a view to connect things together. To represent each operation, you’d upload documents like the following two examples:

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "heart bypass"
}

{
"type": "operation",
"patient": "Joe McIllness",
"surgery": "lumbar puncture"
}

Emitting the "patient" field as the key in your view would then allow querying for all operations for a specific patient. Again, views are used to help knit together a full picture of a specific entity from separate documents. Views help keep the number of HTTP requests low, even though IBM Cloudant splits up the data for a single-modeled entity.
[API Details](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/16-examples.png)



--------------------





 View and test the APIs 

This section shows how a portal user would view and test the available APIs for a product.



1. Navigate to the API details in the API Connect Developer Portal as outlined in the previous section.
![API Details](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/21-details.png)
2. You can download and view the APIs OpenAPI (Swagger) yaml information by selecting Open API.
![Download OpenAPI yaml](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/22-OpenAPI.png)
3. Scroll down to one of the operations to view its details. You can also click on the operations link to jump to it on the page. ![Operation](https://cloud.ibm.com/docs-content/v1/content/78bb71851d95d7580503eb9ba10cf3ae31490ade/apiconnect/tutorials/images/23-operation.png)
4. In the right panel under the examples, scroll to the Try this operation section. Enter the parameters and select Call operation.
!
Getting database details 

Sending a GET request to https://$ACCOUNT.cloudant.com/$DATABASE returns details about the database, such as how many documents it contains.

See the following example of using HTTP to get database details:

GET /$DATABASE HTTP/1.1

See the following example to get database details:



* Curl
* Java
* Python
* Go
* Node



curl -H "Authorization: Bearer $API_BEARER_TOKEN" -X GET "$SERVICE_URL/products"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.DatabaseInformation;
import com.ibm.cloud.cloudant.v1.model.GetDatabaseInformationOptions;

Cloudant service = Cloudant.newInstance();

GetDatabaseInformationOptions databaseInfoOptions =
new GetDatabaseInformationOptions.Builder()
.db("products")
.build();

DatabaseInformation response =
service.getDatabaseInformation(databaseInfoOptions).execute()
.getResult();

System.out.println(response);

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()

response = service.get_database_information(db='products').get_result()

print(response)

getDatabaseInformationOptions := service.NewGetDatabaseInformationOptions(
"products",
)

databaseInformation, response, err := service.GetDatabaseInformation(getDatabaseInformationOptions)
if err != nil {
panic(err)
}

b, _ := json.MarshalIndent(databaseInformation, "", " ")
fmt.Println(string(b))
Exploring APIs and Products 



1. To view APIs that have been shared with your IBM Cloud org, go to the APIs landing screen and click Explore APIs.
2. Expand the API to view more information about it.
3. To subscribe to an API, click an API product. The Developer Portal opens and you can subscribe to a Plan to access an API.

Note: If the Plan that you select has billing, you must provide a credit card number for your organization. That requires that you have owner privileges.





 Viewing and testing APIs in the Developer Portal 



1. From the Developer Portal home screen, click API Products.
2. To see more details, click APIs for the Product that contains the required API, then click the API.
3. More details of the individual operations associated with an API are available under the API Operations heading.
4. To view the example code of an operation:



* In the API Operations section, click Paths, and click the operation that you want to see the example code for.
* Click the operation's tab to expand and show its details, including code examples. You can view the example code for your operation in several languages.





To test an API, complete the following steps.



1. Click APIs. All of the APIs that can be used by application developers are displayed.
2. Click the name of the API that you want to test.
3. Find the operation you want to work with under the API Operations heading.
4. Click Try this operation.
5. Supply the values for required headers or parameters.
6. If the operation is secured with Basic Authentication, supply the credentials.
7. Click Send Request. The result is displayed in the Response Body field. You can continue to test different parameter values as necessary.







 Subscribing to use APIs 



1. In the Developer Portal, click API Products.
2. Click the Product that contains the Plan that you want to work with.
3. In the Plan section of the Product, click the Plan that you want to use.
The parameter was replaced with the endpoint POST /{db}/_design/{ddoc}/_view/{view}/queries and is supplied as a queries request body parameter. You can also make multiple queries with the following new endpoints:



* POST /{db}/_all_docs/queries
* POST /{db}/_design_docs/queries



Sending several queries to a view
: Sending multiple queries to a view that uses a POST request to /$DATABASE/_design/$DDOC/_view/$VIEWNAME is deprecated with [multi-querying a MapReduce view](https://cloud.ibm.com/apidocs/cloudantpostviewqueries). For more information, see the previous deprecation note about replacing the queries parameter.





 4 April 2018 

The following changes were made in build 6875:

New! Audit facility
: Internal audit facility is added to the platform.

IBM Cloudant Query error messages
: Improve error messages for IBM Cloudant Query.







 March 2018 



 30 March 2018 

The following changes were made in build 6870:

kill command
: Fix how the kill command works when you terminate an operating system process.

_changes endpoint
: Fix _changes endpoint shard substitution.

Compaction resumption
: Fix compaction resumption for terminated compactions.





 13 March 2018 

The following changes were made in build 6761:

New! _dbs_info endpoint
: Introduce new _dbs_info endpoint to get information from a list of databases. See [Get a list of all databases in the instance](https://cloud.ibm.com/apidocs/cloudantgetalldbs).

New! Pluggable storage engine
: Add a pluggable storage engine.

Improvement
: Update MochiWeb to version 2.17.

Attachments
: Ensure deterministic revisions for attachments. See [COUCHDB-3255](https://issues.apache.org/jira/browse/COUCHDB-3255).
Setting up GraphQL 

GraphQL is a query language for APIs. You can run queries on a single /graphql endpoint. GraphiQL uses a built-in introspection system to expose information through the schema.

GraphQL analytics API endpoint is only available for Enterprise-level plans.



 Prerequisites 

You must have the following permissions to proceed:



* You must have Viewer permission either at the instance level or zone level.
* You must have Reader permission at the zone level.





 GraphiQL tool 

Using GraphiQL, you can explore the schema and test queries for the GraphQL endpoint.



1. [Download and install GraphiQL](https://github.com/graphql/graphiql).
2. Open the GraphiQL application and enter the HTTP headers that you want to authorize.
3. Enter the correct endpoint in the GraphQl Endpoint field. Use https://api.cis.cloud.ibm.com/v1/<crn>/zones/<zoneid>/graphql.



The window is divided into a query pane and a response pane. You can define the query variables, which are interpolated into the query in the query pane.

To build a query, follow these steps.



1. Select POST from the Method list menu.
2. Click Edit HTTP Headers.
3. Enter the X-Auth-User-Token or Authorization token, and set the Content-Type: application/json. Be sure to save them.



Start building your query in the query pane. Use the Document Explorer to explore the schema and documentation. The Document Explorer shows the different available options for datasets, dimensions, operations, and functions. The words "zone" and "domain" are synomyous in the Document Explorer.

Paste the following test snippet into the query pane and observe the response in the response pane, adjusting the datetime to meet your needs.

query {
viewer {
* [Accessing the API](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refaccess-grep11-functions)
* [Error handling](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-error-handling)
* [GREP11 function list](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11_function_list)
* [Supported mechanisms](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-mechanism-list)
* [Supported attributes and key types](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-attribute-list)
* [Supported curves](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refsupported-grep11-curve-name)
* [Performing cryptographic operations with GREP11 functions](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-functions)
* [Retrieving supported crypto algorithms](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-operation-retrieve-mechanisms)
* [Generating and deriving keys](https://cloud.ibm.com/docs/hs-crypto?topic=hs-crypto-grep11-api-refgrep11-operation-generate-keys)
IBM Watson Query 

Welcome to the Watson Query product documentation.

  Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/data-virtualization-on-cloud)

  Recommended content 

[FAQs Answers to frequently asked questions about the Watson Query service.](https://cloud.ibm.com/docs/data-virtualization?topic=data-virtualization-faq_dv)[Provision a service instance Create and deploy a Watson Query service instance.](https://cloud.ibm.com/docs/data-virtualization?topic=data-virtualization-provisioning)

  Latest updates 

[View more](https://cloud.ibm.com/docs/data-virtualization?topic=data-virtualization-dv-wn)24 June 2021 | Watson Query GA

Coming soon

26 April 2021 | Watson Query beta

The IBM Watson Query closed beta is now live.
* For API Manager UI-based issues, additionally include:



* All error codes returned
* Time (including time zone) that the problem occurred



* For Portal-based issues, additionally include the user name of person who encountered the problem
* For issues with invoking APIs, additionally include:



* Full API URL impacted - the host name should contain apiconnect.ibmcloud.com
* HTTP method used
* Frequency and time (with time zone) of the issue
* Expected result
* Actual (incorrect) result
* Evidence from analytics to show that the API issue is not a problem with the customer's down-stream system. For more information, see the IBM Developer article [Why isn’t my API working](https://developer.ibm.com/apiconnect/why-isnt-my-api-working/)
* Complete cURL example for recreating the call (include all information required for the call, including headers, client ID, and client secret)





Optionally attach files to help explain your issue (for example, screen captures). Click the Information icon next to "Supported file types" to see the complete list of allowed file types. Helpful attachments include:



* A (compressed) CSV file that contains exported analytics for failing APIs, to ensure we see the correct calls and the failure patterns
* A copy of the API YAML file, with personal data such as IDs, secrets, tokens, and passwords removed
* A copy of a browser HAR trace file, if the problem is related to the behavior of the API Connect UI



4. Click Create to submit your new case.



To view a case's status, look in the "Recent cases" section of the Support Center page. If you don't see the case you're looking for, click View all open cases.