vania-janet/multiturn-rag-retrieval-data huggingface dataset:
However, as the number of unresolved conflicts grows, the performance of the database goes down a black hole, especially when you replicate.

As a developer, it’s your responsibility to check for, and to resolve, conflicts, or even better, employ data models that make conflicts impossible.

If you routinely create conflicts, you must really consider model changes: even if you resolve your conflicts diligently, the conflict branches in the revision tree remain with no easy way to tidy that up. For more information, see the following websites:



* IBM Cloudant guide to [conflicts](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-conflictsconflicts)
* IBM Cloudant guide to versions and [MVCC](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-document-versioning-and-mvccdocument-versioning-and-mvcc)
* Three-part blog series on [conflicts](https://developer.ibm.com/blogs/)







 Deleting documents doesn't delete them 

Deleting a document from an IBM Cloudant database doesn’t purge it. Deletion is implemented by writing a new revision of the document under deletion, with an added field _deleted: true. This special revision is called a tombstone. Tombstones still take up space and are also passed around by the replicator.

Models that rely on frequent deletions of documents are not suitable for IBM Cloudant. For more information, see IBM Cloudant tombstone [docs](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-documents-tombstone-documents).





 Be careful with updates 

It is more expensive in the end to mutate existing documents than to create new ones. IBM Cloudant always needs to keep the document tree structure around. This rule applies even if internal nodes in the tree are stripped of their payloads. If you find that you create long revision trees, your replication performance suffers. Moreover, if your update frequency is higher than, say, once or twice every few seconds, you’re more likely to produce update conflicts.

Prefer models that are immutable.
* A tables array that breaks down the tables identified in the input document.
* A document_structure object that lists section titles and leading sentences identified in the input document.
* A parties array that lists the parties, roles, addresses, and contacts of parties identified in the input document.
* Arrays defining effective_dates, contract_amounts, termination_dates, contract_types, contract_terms, payment_terms, and contract_currencies.



This feature is currently supported in English only. For more information, see [Language support](https://cloud.ibm.com/docs/discovery?topic=discovery-language-supportfeature-support).



 Classification requirements 

The Element Classification enrichment is deprecated and will no longer be available, effective 10 July 2020.

To classify documents using Element Classification your configuration and source documents must meet the following requirements:



* Files to be analyzed are in PDF format.
* The PDF contents are in text form. Scanned documents cannot be parsed, even if they are OCRed.

You can identify a PDF that is in text by opening the document in a PDF viewer and using the Text select tool to select a single word. If you cannot select a single word in the document, the file cannot be parsed.
* Files are no larger than 50Mb in size.
* Secure PDFs (with a password to open) and editing restricted PDFs (with a password to edit) cannot be parsed.
* The Discovery tooling includes a configuration named Default Contract Configuration that can be used to enrich your collection of PDF documents. You also have the option of creating a custom configuration that includes the elements enrichment. See [Collection requirements](https://cloud.ibm.com/docs/discovery?topic=discovery-element-classificationelement-collection) for details.
* Lite plans can process a maximum of 500 pages per month.
* Not available in Dedicated environments.
* Post-enrichment normalization cannot be performed when using Element Classification.



The Default Contract Configuration file was updated on September 25, 2018.
Avoiding invariants makes it more likely that a simple merge operation, if you take the changed field from each conflicted document revision, is suitable. This practice makes simpler and more robust application code.
* Allow documents to be independent. If you have to retrieve other documents to work out the correct resolution, it increases latency in conflict resolution. There's also a chance you get a version of the other documents that aren't consistent with the document you're resolving, making correct resolution difficult.







 Why do I need to build in conflict resolution 

Heavily conflicted documents exert a heavy toll on the database. Building in the capability to resolve conflicts from the beginning is a great help in avoiding pathologically conflicted documents.





 Where can I get more information? 

These tips demonstrate how modeling data affects your application’s performance. IBM Cloudant’s data store has some specific characteristics, both to watch out for and to take advantage of, that ensure the database performance scales as your application grows. IBM Cloudant support understands the shift can be confusing, so they are always available to give advice.

For more information, see the [data model for Foundbite](https://cloudant.com/blog/foundbites-data-model-relational-db-vs-nosql-on-cloudant/), or the [example from our friends at Twilio](https://www.twilio.com/blog/2013/01/building-a-real-time-sms-voting-app-part-3-scaling-node-js-and-couchdb.html).
If a view definition contains an attribute that is called dbcopy, as is the case with the previous example design document, then the dbcopy feature is in use and is affected by the feature's removal.





 Alternatives to dbcopy 

No alternative to the dbcopy feature exists. Simply removing the "dbcopy" attribute from a design document instructs IBM Cloudant to build a normal MapReduce view. This view contains the same data that was being copied to the secondary database, for example:

{
"_id": "_design/myview",
"views": {
"view": {
"map": "function(doc) { emit(doc.date, doc.price) }"
}
}
}

The dbcopy key was removed, so this view becomes a normal MapReduce view that you can [query by using the IBM Cloudant API](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-using-viewsquerying-a-view).

If you are concerned about the removal of the dbcopy feature, you can open a support ticket and ask to consult with our Client Architecture team.







 IBM Cloudant replications no longer support HTTP 

As of 1 October 2023, the replicator for IBM Cloudant no longer supports the HTTP protocol – it supports only the HTTPS protocol to ensure that customer data is always encrypted in flight.

These changes only affect replications leaving IBM Cloudant. Customers who only replicate between Cloudant instances have no action to take.



 What is replication? 

IBM Cloudant replications allow you to copy one database's changes to another IBM Cloudant database. This database is in the same IBM Cloudant instance or in another IBM Cloudant instance on the other side of the world. IBM Cloudant also replicates to and from Apache CouchDB and PouchDB databases on the public internet.





 What happens after HTTP support is removed? 

Following this change, a replication that is configured with the http:// protocol fails permanently. The replication job is not retried. Any replications using the https:// protocol are unaffected.





 How do I know whether my replications use HTTP?
2. Populate the database with some test data.
3. Query the database for some subset of this test data.
4. Verify that the data that you got back is the data that you expected to get back.



Nothing wrong with that test? That works on every other database that you ever used, correct?

Not on IBM Cloudant.

Or rather, it works 99 times out of 100.

The reason for this difference is a (mostly) small inconsistency window between writing data to the database and this data becoming available on all nodes of the cluster. As all nodes in a cluster are equal in stature, no guarantee exists that a write, and a subsequent read, are serviced by the same node. So, in some circumstances, the read might be hitting a node before the written data makes it to the node.

So why don’t you just put a short delay in your test between the write and the read? That delay makes the test less likely to fail, but the problem is still there.

IBM Cloudant has no transactional guarantees. While document writes are atomic (you’re guaranteed that a document can either be read in its entirety, or not at all), no way exists to close the inconsistency window. It’s there by design.

A serious concern that every developer must consider is that you can’t safely assume that data you write is available to anyone else at a specific point in time. This state takes some getting used to if you come from a different kind of database tradition.

Testing tip: what you can do to avoid the inconsistency window in testing is to test against a single-node instance of IBM Cloudant or CouchDB running say in Docker ([docker information](https://hub.docker.com/_/couchdb/)). A single node removes the eventual consistency issue, but beware that you are testing against an environment that behaves differently from what you target in production. Caveat Emptor.





 Replication isn't magic 

“So let’s set up three clusters across the world, Dallas, London, Sydney, with bi-directional synchronization between them to provide real-time collaboration between our 100,000 clients.”
Testing tip: what you can do to avoid the inconsistency window in testing is to test against a single-node instance of IBM Cloudant or CouchDB running say in Docker ([docker information](https://hub.docker.com/_/couchdb/)). A single node removes the eventual consistency issue, but beware that you are testing against an environment that behaves differently from what you target in production. Caveat Emptor.





 Replication isn't magic 

“So let’s set up three clusters across the world, Dallas, London, Sydney, with bi-directional synchronization between them to provide real-time collaboration between our 100,000 clients.”

No. Just… No. IBM Cloudant is good at replication. It might seem like magic, but note that it makes no latency guarantees. In fact, the whole system is designed with eventual consistency in mind. Treating IBM Cloudant’s replication as a real-time messaging system does not end up in a happy place. For this use case, put a system in between that was designed for this purpose, such as [Apache Kafka](https://kafka.apache.org/).

It’s difficult to put a number on replication throughput. The answer is always, “It depends.” Things that impact replication performance include, but are not limited to:



1. Change frequency
2. Document size
3. Number of simultaneous replication jobs on the cluster as a whole
4. Wide (conflicted) document trees
5. Your reserved throughput capacity settings



For more information, see the following websites:



* Blog post on [replication topology](https://dx13.co.uk/articles/2017/11/07/cloudant-replication-topologies-and-failover/)
* IBM Cloudant [guide to replication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-replication-guidereplication)







 Use the bulk API 

IBM Cloudant has nice API endpoints for bulk loading (and reading) many documents in a single request.
* Developers and Researchers are spread across the globe and across organizational boundaries, which make PaaS and SaaS the best option for each user group.
* Regional differences in health regulations require some data and data processing to remain within that region.







 Solution 

Securely hosting and sharing disease data for Research Nonprofit.

The research nonprofit wants to aggregate cancer research data across the globe. So they create a division that is dedicated to solutions for their researchers.



* INGEST - Apps to ingest research data. Researchers today use spreadsheets, documents, commercial products, and proprietary or home-grown databases to record research results. This situation is unlikely to change with the nonprofit's attempt to centralize data analysis.
* ANONYMIZE - Apps to anonymize the data. SPI must be removed to comply with regional health regulations.
* ANALYZE - Apps to analyze the data. The basic pattern is to store the data in a regular format and then to query and process it by using AI and machine learning (ML) technology, simple regressions, and so forth.



Researchers need to affiliate with a regional cluster, and apps ingest, transform, and anonymize the data.



1. Syncing the anonymized data across regional clusters or shipping them to a centralized data store
2. Processing the data, by using ML like PyTorch on bare metal worker nodes that provide GPUs



INGEST
: IBM Cloudant is used at each regional cluster that stores researchers’ rich data documents and can be queried and processed as needed. IBM Cloudant encrypts data at rest and in transit, which complies with regional data-privacy laws.
: IBM Cloud® Functions is used to create processing functions that ingest research data and store them as structured data documents in IBM Cloudant. IBM® Secure Gateway for IBM Cloud® provides an easy way for IBM Cloud® Functions to access on-premises data in a safe and secure manner.
: Web apps in the regional clusters are developed in nodeJS for manual data entry of results, schema definition, and research organizations affiliation. IBM Key Protect helps to secure access to IBM Cloudant data, and IBM Vulnerability Advisor scans app containers and images for security exploits.

ANONYMIZE
Split documents to make query results more succinct 

Split your documents so that the search function can find more concise information to return in query results.

For more information about the benefits of splitting documents, read the [Using IBM Watson Discovery’s New Document Segmentation Feature](https://medium.com/ibm-watson/using-ibm-watson-discoverys-new-document-segmentation-feature-7a58b44d32c2) blog post on Medium.com.

You can split only documents to which a user-trained Smart Document Understanding model is applied.

When you split a document, the original document is broken into segments. Each segment contains a more uniform set of information. By splitting the content in your documents into segmented groups, you can enrich and index your data at a more granular level.

To control how your documents are split, you specify a field, such as subtitle or question, to use as the page break marker. The page break options are populated with fields that are created when you apply a user-trained Smart Document Understanding (SDU) model to the documents. For more information, see [Using Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fields). You cannot split documents with fields that are generated by a pretrained Smart Document Understanding model.

As a document is reprocessed, it is evaluated from start to end. Whenever the page break marker field occurs, the original document is split and a new segment is created. The splitting continues at each marker field until the original document is broken into multiple segments.

Before you begin, decide which field to use as the page break marker.



* You can use any of the fields that are indexed by default. To see your choices, check the Fields to index list. Fields that have a Type value are stored in the index.
* The number of segments per document is limited to 1,000. After segment number 999 is created, any remaining document content is stored within segment 1,000.
* Metadata from PDF and Microsoft Word documents and any custom metadata is extracted and included in the index with each segment.
You can now define JSON normalizations by using the Collections API
: The Create a collection and Update a collection methods now support the addition of conversions and normalizations objects that you can specify to apply normalization operations to the documents in the collection. For example, you can define an operation to copy or merge one field to another in the JSON representation of the documents. The conversions object defines normalization operations that occur during ingestion and the normalizations object defines normalization operations that occur after enrichments are applied. For more information, see the [Collections API reference](https://cloud.ibm.com/apidocs/discovery-datalistcollections).





 31 March 2023 

Update to API version
: The current API version (v2) is now 2023-03-31. One change was made with this version.

Changed how fields named document_id are handled
: If you add a JSON file that contains a field named document_id to a collection, the field is ignored. The system assigns a new unique document ID to the document when it is added to the index. To assign a document ID to a document regardless of its file type, use the Update document method from the API.

Previously, when you uploaded a JSON file with a field named document_id from the product user interface or by using the Add document API method, the document ID from the file was shown as the document_id value in query results. However, a different document ID was assigned to the document, and the assigned ID had to be used for certain other tasks, such as deleting the document. If your application relies on the previous behavior, specify a version number earlier than 2023-03-31, such as 2020-08-30, in your API calls.





 2 March 2023 

Now you can specify the types of files to add to a collection
: When you connect to an external data source, you can limit the types of files to add to the collection from the external data source. For example, you can choose to add only PDF files from a Box data source.





 21 Febuary 2023 

Optical character recognition v2 technology is used
: Add the replication scheduler. Learn more about [replication scheduler](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-advanced-replicationthe-replication-scheduler).

_revs-diff endpoint
: POST requests to the _revs_diff endpoint require either the _reader or _replicator role.







 July 2017 



 24 July 2017 

Retire Shared plan
: IBM Cloudant Shared Plan accounts can no longer be created from the [IBM Cloudant product page](https://www.ibm.com/cloud/cloudant).





 4 July 2017 

The following changes were made in build 6276:

Error message
: An error message changed that occurs when you try to put a document attachment with a nonexistent revision. Now, the error is a 409 error with the following information:

{"error":"not_found","reason":"missing_rev"}







 June 2017 



 26 June 2017 

The following changes were made in build 6233:

IBM Cloudant Query and indexes
: Fixes an IBM Cloudant Query issue where indexes that excluded potentially matching documents were selected by the query planner.





 14 June 2017 

Revised error message
: The error message that occurs when you try to put a document attachment with a nonexistent revision. This error is changed to a 409 error with the following information:

{"error":"not_found","reason":"missing_rev"}







 May 2017 



 11 May 2017 

The following changes were made in build 6069:

New! $allmatch operator support
: Added support for the $allmatch operator.

Replication
: Previously, a replication job that failed for some reason, which resulted in an update to the replication document, was followed by a fresh attempt to start a new replication. Under some circumstances, this behavior might continue indefinitely: many duplicates of the same error message. A fix was introduced so that the replication document is not updated unless the reason for the error changes.







 February 2017 



 13 February 2017 

The following changes were made in build 5834:

Document id length
: An array of changes.

last_seq
: A token that can be supplied to the changes endpoint in a subsequent API call to get the next batch of changes.

See how to fetch the next batch of changes in the following example:

GET /orders/_changes?limit=5&since=5-g1AAAAB5eJzLYWBg
{
"results": [ ...],
"last_seq": "10-g1AAAACbeJzLY"
}

The since parameter is used to define where in the changes feed you want to start from:

since=0
: The beginning of the changes feed.

since=now
: The end of the changes feed.

since=<a last seq token>
: From a known place in the changes feed.

At face value, following the changes feed seems as simple as chaining _changes API calls together. Then, IBM Cloudant passes the last_seq from one changes feed response into the next request's since parameter. But some subtleties to the changes feed need further discussion.





 Why does the changes feed deliver each change at least one time? 

The IBM Cloudant Standard changes feed promises to return each document at least one time, which isn't the same as promising to return each document only one time. Put another way, it is possible for a consumer of the changes feed to see the same change again, or indeed a set of changes repeated.

A consumer of the changes feed must treat the changes idempotently. In practice, you must remember whether a change was already dealt with before you trigger an action from a change. A naive changes feed consumer might send a message to a smartphone on every change received. But a user might receive duplicate text messages if a change is not treated idempotently when replayed changes occur.

Usually these "rewinds" of the changes feed are short, replaying only a handful of changes. But in some cases, a request might see a response with thousands of changes replayed - potentially all of the changes from the beginning of time. The potential for rewinds makes the changes feed unsuitable for an application that expects queue-like behavior.
SOC 1 Type 2, SOC 2 Type 2, and SOC 3 Certification 

IBM® provides a Service Organization Controls (SOC) report for Event Streams. The reports evaluate IBM's operational controls according to the criteria set by the American Institute of Certified Public Accountants (AICPA) Trust Services Principles. The Trust Services Principles define adequate control systems and establish industry standards for service providers such as IBM Cloud to safeguard their customers' data and information.

You can request a SOC report from the customer portal or contact your sales representative. Alternatively, you can open a support ticket with [IBM Cloud support](https://cloud.ibm.com/unifiedsupport/supportcenter).

The Enterprise and Standard plans are SOC certified.





 Payment Card Industry Data Security Standard (PCI DSS) 

Event Streams is compliant with the Payment Card Industry Data Security Standard (PCI DSS). IBM Cloud completes annual PCI DSS assessments by using an approved Qualified Security Assessor (QSA), and the resulting Attestations of Compliance (AOCs) and Service Responsibility Matrix (SRM) guides are available upon customer request. Auditors reviewed Event Streams for compliance under PCI DSS version 3.2.1 at Service Provider Level 1.

If you intend to store sensitive information in an IBM Cloudant database, you must use client-side encryption to render data unreadable to IBM Cloudant operators. For example, for PCI DSS compliance, you must encrypt the Primary Account Number (PAN) before sending a document that contains it to the database.

Customers are responsible for the storing, processing, and transmission of their cardholder data, and can create cardholder data environments (CDEs) that can store, transmit, or process cardholder data by using Event Streams. Customers can request and use the IBM Cloud AOCs and SRM guides when they seek their own PCI DSS certifications. It is the responsibility of the customer to document and operate CDEs and applications that are built by using IBM Cloud Platform services in a PCI DSS-compliant manner.

Event Streams documentation on service security and deletion of data covers methods to manage cardholder data within the environment in accordance with PCI requirements.