vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Getting changes to documents in the database 

Sending a GET request to https://$ACCOUNT.cloudant.com/$DATABASE/_changes returns a list of changes that were made to documents in the database, including insertions, updates, and deletions.

When a _changes request is received, one replica for each shard of the database is asked to provide a list of changes. These responses are combined and returned to the original requesting client.

The _changes endpoint accepts several optional query arguments:



Table 1. Query arguments for _changes endpoint

 Argument Description Supported values Default 

 conflicts Can be set only if include_docs is true. Adds information about conflicts to each document. Boolean False 
 descending Return the changes in sequential order. Boolean False 
 doc_ids To be used only when filter is set to _doc_ids. Filters the feed so that only changes to the specified documents are sent. Note: The doc_ids parameter works only with versions of IBM Cloudant that are compatible with CouchDB 2.0. For more information, see [GET /](https://cloud.ibm.com/apidocs/cloudantgetserverinformation) documentation. A JSON array of document IDs 
 feed Type of feed required. For more information, see the [feed information](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-get-changesthe-feed-argument). "continuous", "longpoll", "normal" "normal" 
 filter Name of [filter function](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-design-documentsfilter-functions) to use to get updates. The filter is defined in a [design document](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-design-documentsdesign-documents). string No filter. 
 heartbeat If no changes occurred during feed=longpoll or feed=continuous, an empty line is sent after this time in milliseconds. Any positive number No heartbeat 
 include_docs Include the document as part of the result.
[View more](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-etcd-relnotes)19 October 2022 | Deploying and Connecting an IBM Cloud® Databases Instance Tutorial

This tutorial guides you through the process of deploying a Cloud Databases instance and connecting it to a web front end by creating a webpage that allows visitors to input a word and its definition. These values are then stored in a database running on Cloud Databases. You install the database infrastructure by using Terraform and your web application uses the popular Express framework. The application can then be run locally, or by using Docker.

11 October 2022 | Protecting IBM Cloud® Databases for etcd resources with context-based restrictions

Context-based restrictions (CBR) give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies.

25 March 2022 | IBM Cloud® Databases for etcd Security Goals Updated

Available security and compliance goals updated.
* For more information about browser support for deployments that are provisioned with Cloud Pak for Data as a Service, see [Which web browsers are supported for Cloud Pak for Data as a Service](https://dataplatform.cloud.ibm.com/docs/content/wsj/getting-started/browser-support.html).







 Language support 

Language support by feature is detailed in the [Supported languages](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-language-support) topic.





 Beta features 

IBM releases services, features, and language support for your evaluation that are classified as beta. These features might be unstable, might change frequently, and might be discontinued with short notice. Beta features also might not provide the same level of performance or compatibility that generally available features provide and are not intended for use in a production environment.





 Terms and notices 

IBM Cloud



* [IBM Cloud Terms of use](https://cloud.ibm.com/docs/overview/terms-of-use?topic=overview-terms)
* [Service terms (Search for Watson Discovery)](https://www-03.ibm.com/software/sla/sladb.nsf/sla/saas?OpenDocument)
* [Data Processing and Protection Datasheet](https://www.ibm.com/software/reports/compatibility/clarity-reports/report/html/softwareReqsForProduct?deliverableId=A1417A507E8211E6BA51E79BE9476040)



IBM Cloud Pak for Data



* [Security on Cloud Pak for Data](https://www.ibm.com/docs/SSQNUZ_4.6.x/cpd/plan/security.html)



Trademarks are listed in the [Trademarks](https://cloud.ibm.com/docs/overview?topic=overview-noticestrademarks) page for all IBM Cloud services.

Zoom

![Java integrated logo.]
For more information about adding a document classifier by using the product user interface, see [Classifying documents](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-cm-doc-classifier).





 21 March 2022 

Visualize enrichments found in your documents
: When you click to view the passage from a search result, a document preview page is displayed that shows a representation of the original document where the search result was found. For most document types, you can open a new advanced view of the document to see useful summary information, such as the number of occurrences of any enrichments that are detected in the document. You also can select one of the enrichments to highlight every occurrence of the element within the document text.

Currently, only the Entities and Keywords enrichments are listed.

Improved format of search results from PDF documents
: When you click to view a passage from a search result that is extracted from a PDF document, a document preview page is displayed that shows the returned passage in the context of the original PDF page.

The in-context view is available for PDF files to which a Smart Document Understanding model is applied. The rich preview does not work on images, meaning it doesn't work on scanned PDF documents. The in-context view is available for PDFs in all languages; however, the enrichment highlighting might be misaligned in some languages.

Tell us what you think
: Share your opinions and ideas with us at any time by clicking the Share feedback button from the page header of the product user interface.





 10 March 2022 

Manage the data in a collection from the new Manage data page
: You can now access the Manage data page for a collection from the Manage collections navigation pane. Go there to see a list of the documents in your collection and get a quick view of information about the documents. You can also delete documents from a collection with just a few clicks. For more information, see [Excluding content from query results](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-hide-data).





 15 February 2022 

An alternative authentication mechanism is available for Microsoft Sharepoint Online connectors
IBM Cloud® Databases for EnterpriseDB docs 

EDB is a PostgreSQL-based database engine optimized for performance, developer productivity, and compatibility with Oracle.

 Developer tools 

[API & SDK reference](https://cloud.ibm.com/apidocs/cloud-databases-api/cloud-databases-api-v5introduction)[CLI reference](https://cloud.ibm.com/docs/databases-cli-plugin)[Terraform reference](https://registry.terraform.io/providers/IBM-Cloud/ibm/latest/docs/resources/database)

 Recommended content 

[Getting Started A brief tutorial to introduce using an IBM Cloud® Databases for EnterpriseDB deployment.](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-getting-started)[Setting the Admin Password The Databases for EnterpriseDB service is provisioned with an admin user. You must set the admin password before you use it to connect.](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-admin-password&interface=ui)[Getting Connection Strings To connect to IBM Cloud® Databases for EnterpriseDB, you need some users and connection strings. Connection Strings for your deployment are displayed on the Dashboard Overview, in the Endpoints panel. These strings can be used with any set of credentials that you generate.](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-connection-strings&interface=ui)

 Learn more 

[IBM Developer<br><br>Visit IBM Developer for technical articles, code patterns, tutorials, and more.<br><br>!
IBM Cloud Object Storage 

Crawl documents that are stored in an IBM Cloud® Object Storage data source.

IBM Cloud

IBM Cloud only

This information applies only to managed deployments.



 What documents are crawled 

During the initial crawl of the content, documents from all of the content that can be accessed from the storage endpoint are crawled and added to your collection. You cannot crawl private endpoints.

During subsequent scheduled recrawls, only new and modified documents are crawled and any changes are reflected in your collection. Documents that are deleted from the external data source are not deleted from the collection.

All Discovery data source connectors are read-only. Regardless of the permissions that are granted to the crawl account, Discovery never writes, updates, or deletes any content in the original data source.

The following table illustrates the objects that Discovery can crawl.



Table 1. Data sources crawling support

 Data source Objects that are crawled 

 IBM Cloud Object Storage Buckets, files 







 What you need before you begin 

Obtain any required service licenses for the content on the website that you want to connect to. For more information about licenses, contact the system administrator of the data source.

Endpoint
: The endpoint for your IBM Cloud Object Storage data. For example, s3.us-south.cloud-object-storage.appdomain.cloud.

Do not include http:// or https:// in the endpoint value. For more information, see [Regional Endpoints](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-endpointsendpoints-region).

In addition to the endpoint, you must provide credentials to enable authentication with the object store. You can choose to use one of the following authentication methods:

HMAC
: Uses a hash-based message authentication code to authenticate users. HMAC is a cryptographic authentication technique that uses a hash function and a secret key. The data is scrambled before it is sent over the internet. Then, the intended recipient uses the secret key to unscrambles the data.
This tutorial is a short introduction to using an IBM Cloud® Databases for EnterpriseDB deployment. pgAdmin is an open source administration platform for PostgreSQL, and provides many tools for managing your data and databases. Download and install the version that is appropriate to your environment, and then follow the steps to connect it to your Databases for EnterpriseDB deployment.

Databases For EnterpriseDB



* 30 minutes
* 2023-04-27



[Getting Started](https://cloud.ibm.com/docs/databases-for-etcd?topic=databases-for-etcd-getting-started)Getting Started

This tutorial is a short introduction to using an IBM Cloud® Databases for etcd deployment.

Databases For etcd



* 30 minutes
* 2023-04-27



[Getting started with Databases for MongoDB](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-getting-started)Getting started with Databases for MongoDB

IBM Cloud® Databases for MongoDB allows developers to take advantage of the latest MongoDB features: rich JSON documents, powerful query language, multi-document transactions, and authentic APIs. The service also automates common database administration tasks like high availability, backups, encryption, and infrastructure planning.

Databases For MongoDB



* 30 minutes
* 2023-04-27



[Modern web application using MEAN stack](https://cloud.ibm.com/docs/solution-tutorials?topic=solution-tutorials-mean-stack)Modern web application using MEAN stack

This tutorial walks you through the creation of a web application using the popular MEAN stack. It is composed of a MongoDB, Express web framework, Angular front end framework and a Node.js runtime. You will learn how to run a MEAN sample app locally, create and use a managed database-as-a-service (DBasS), deploy the app to IBM Cloud and scale both the runtime and database resources.

Code Engine Databases For MongoDB



* 1 hour
* 2023-05-05
Under Search for Documents, click Use the Discovery Query Language, then:



* Click the Field drop-down and choose enriched_text.concepts.text, for the Operator choose contains, then enter the Value of world wide web.

The query enriched_text.concepts.text:"world wide web" returns all documents that include the concept of world wide web, and those documents are ranked in order of relevance.



4. Click More options, then Fields to return and choose Specify. Select text, which limits the response to the text of the relevant articles and exclude everything else.
5. Click Run query. There is one matching document: "matching_results": 1







 Step 5: Building an aggregation 

Aggregations return a set of data values; for example, top keywords, overall sentiment of entities, and more.

Try building this aggregation. It returns the top 10 concepts in the IBM press releases collection.



1. Click on the Build queries icon ![Query icon](https://cloud.ibm.com/docs-content/v1/content/ded4adc3ea0bd2a81b113c579f2b1183926da211/discovery/images/icon_queryBuilder.png) to open the query page. Select the collection that contains the IBM Press Releases and click Get started.
2. Under Include analysis of your results:



* Click the Output drop-down and choose Top values, for the Field choose enriched_text.concepts.text, then enter the Count of 10.

Term returns the most common values for the conceptstext field. Count specifies the number of results that you want returned. The query term(enriched_text.concepts.text,count:10) displays under the Visual Query Builder.



3. Click More options, then enter 0 in the Number of documents to return field.
4. Click Run query. The top 10 concepts are displayed in both the Summary and JSON tabs.







 Step 6: Build a query in Watson Discovery News 

Discovery News, is a public data set that is pre-enriched with cognitive insights. It is included with Discovery.
For more information, see [Deploying and Connecting a Cloud Databases Instance](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-create-instance-tutorial).





 11 October 2022 

Protecting IBM Cloud® Databases for EnterpriseDB resources with context-based restrictions
: Context-based restrictions (CBR) give account owners and administrators the ability to define and enforce access restrictions for IBM Cloud® resources based on the context of access requests. Access to Cloud Databases resources can be controlled with CBR and identity and access management (IAM) policies. For more information, see [Protecting Cloud Databases resources with context-based restrictions](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=cloud-databases-cbr&interface=ui).





 29 March 2022 

IBM Cloud® Databases for EnterpriseDB supports synchronous replication
: IBM Cloud® Databases for EnterpriseDB now supports synchronous replication. See documentation [here](https://cloud.ibm.com/docs/databases-for-enterprisedb?topic=databases-for-enterprisedb-high-availabilitysync-repl).





 06 July 2020 

General Availability of IBM Cloud® Databases for EnterpriseDB
: IBM Cloud® Databases for EnterpriseDB added to the [IBM Cloud Databases](https://www.ibm.com/cloud/databases) family. See blog post announcement [here](https://www.ibm.com/cloud/blog/announcements/ibm-cloud-databases-for-enterprisedb).
POST /{db}/_all_docs <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostalldocs), [CouchDB](https://docs.couchdb.org/en/latest/api/database/bulk-api.html?highlight=_all_docspost--db-_all_docs) Query a list of all documents in a database by using the HTTP POST method where the query is submitted as a JSON object in the request body. <br>CouchDB documentation references more query parameters (end_key, endkey_docid, end_key_doc_id, stale, start_key, startkey_docid, start_key_doc_id, update_seq). 
 POST /{db}/_bulk_docs <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostbulkdocs), [CouchDB](https://docs.couchdb.org/en/latest/api/database/bulk-api.html?highlight=_all_docspost--db-_bulk_docs) Create and update multiple documents at the same time within a single request. <br>CouchDB documentation references an extra request object field (new_edits). 
 POST /{db}/_bulk_get <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantpostbulkget), [CouchDB](https://docs.couchdb.org/en/latest/api/database/bulk-api.html?highlight=_all_docspost--db-_bulk_get) Retrieve specific revisions or revision histories for multiple documents. 
 GET /{db}/_changes <br>[Cloudant](https://cloud.ibm.com/apidocs/cloudantgetchanges-databases), [CouchDB](https://docs.couchdb.org/en/latest/api/database/changes.htmlget--db-_changes) List of changes that are made to documents in the database, including insertions, updates, and deletions.
Databases[Getting started with IBM Cloudant](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-getting-started-with-cloudant)Getting started with IBM Cloudant

The IBM® Cloudant® for IBM Cloud® Getting started tutorial demonstrates how to use the IBM Cloud® dashboard to create an IBM Cloudant service instance and obtain service credentials to connect to it. Finally, it guides you through the creation of a simple, locally hosted web application that uses your IBM Cloudant database.

Cloudant



* 20 minutes
* 2023-04-03



[Creating a backup](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-a-backup)Creating a backup

This tutorial demonstrates how to use the CouchBackup utility to back up and restore a CouchDB or IBM® Cloudant® for IBM Cloud® instance. CouchBackup backs up the database to a file. If the database fails, you can use the backup file to restore the information to an existing database.

Cloudant



* 20 minutes
* 2023-02-20



[Creating and populating a database](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-and-populating-a-simple-ibm-cloudant-database-on-ibm-cloud)Creating and populating a database

This tutorial shows you how to use the Python programming language to create an IBM® Cloudant® for IBM Cloud® database in your IBM Cloud service instance. You also learn how to populate the database with a simple collection of data.

Cloudant



* 15 minutes
* 2023-04-20



[Using a Dedicated Hardware plan instance](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-and-leveraging-an-ibm-cloudant-dedicated-hardware-plan-instance-on-ibm-cloud)Using a Dedicated Hardware plan instance

This tutorial shows you how to create an IBM® Cloudant® for IBM Cloud® Dedicated Hardware plan instance that uses the IBM Cloud® Dashboard.

Cloudant



* 20 minutes
* 2023-03-30
High-Availability 

IBM Cloud® Databases for MongoDB is a managed cloud database service that is fully integrated into the IBM Cloud environment. The database, storage, and supporting infrastructure all run in IBM Cloud.

Both Databases for MongoDB Standard and Enterprise Edition deployments consist of [three data nodes](https://docs.mongodb.com/manual/core/replica-set-architecture-three-members/primary-with-two-secondary-members-p-s-s), one a primary node and the other two as secondary nodes. They are configured as a replication set without sharding, so they always provide three complete copies of the data set, in addition to the primary. If the primary is unavailable, the replica set elects a secondary to be primary and continues normal operation. The old primary rejoins the set when available.

Connections to a MongoDB replica are made by supplying the driver or binary the replica set name and a seedlist for the hosts in the replica set. This configuration allows drivers and applications the ability to change which host they are connecting to if one of the hosts becomes unavailable. Your deployment has a [connection string](https://cloud.ibm.com/docs/databases-for-mongodb?topic=databases-for-mongodb-connection-strings) in the [DNS Seedlist Connection format](https://docs.mongodb.com/manual/reference/connection-string/dns-seedlist-connection-format), so your applications can take advantage of this feature.



 Application-level High-Availability 

Applications that communicate over networks and cloud services are subject to transient connection failures. You want to design your applications to retry connections when errors are caused by a temporary loss in connectivity to your deployment or to IBM Cloud.

Because Databases for MongoDB is a managed service, regular updates and database maintenance occurs as part of normal operations. This can occasionally cause short intervals where your database is unavailable.

Your applications must be designed to handle temporary interruptions to the database, implement error handling for failed database commands, and implement retry logic to recover from a temporary interruption.

Several minutes of database unavailability or connection interruptions are not expected.
In this example, the first document is an insert because no revision token is supplied. The second document is an update to a document because a revision token is supplied with a new document body. The third document is a deletion. A revision token is supplied, but the body is simply _deleted: true, which tells IBM Cloudant to mark the document as deleted.

It's important to note that this scenario isn't like a transaction in a relational database - all or none of these operations could succeed or fail individually. The response data to this request shows you the response for each operation in turn.

In summary, with two API calls _bulk_docs and _all_docs, we can perform all create, read, update, and delete operations on IBM Cloudant documents and be able to do so in bulk too. _all_docs retrieves documents by _id or ranges of IDs. _bulk_docs creates, updates, and deletes documents in bulk. Generally, we recommend that bulk writes are executed in batches of 500; more for tiny documents and fewer for large documents.

See a screen capture of using IBM Cloudant from a command-line terminal:

That's the end of this part. The next part is called Accessing IBM Cloudant programmatically.







 Accessing IBM Cloudant Programmatically video 

Learn how to access IBM Cloudant programmatically.



* Accessing IBM Cloudant Programmatically video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 9 - Accessing IBM Cloudant programmatically.

So far, our API interactions were triggered by the dashboard or by using curl from the command line. In the following section, we see how IBM Cloudant is accessed programmatically.

The examples use Node.js, so if you want to try the code yourself, you need to install node and npm from nodejs.org.

We can then install the official IBM Cloudant Node.js library with npm install @cloudant/cloudant.