vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Due to sharding, IBM Cloudant offers no guarantees that the output of any two specific map functions passes to the same instance of a reduce call. You must not rely on any ordering. The reduce function that you use must consider all the values that are passed to it and return the correct answer irrespective of ordering. IBM Cloudant is also guaranteed to call your reduce function with rereduce=true at query time even if it didn't need to do so when it built the index. It's essential that your functions work correctly in that case (rereduce=true means that the keys parameter is null and the values array is filled with results from previous reduce function calls).





 Reduced value size 

IBM Cloudant computes view indexes and the corresponding reduce values then caches these values inside each of the B-tree node pointers. Now, IBM Cloudant can reuse reduced values when it updates the B-tree. You must pay attention to the amount of data that is returned from reduce functions.

It's best that the size of your returned data set stays small and grows no faster than log(num_rows_processed). If you ignore this restriction, IBM Cloudant does not automatically throw an error, but B-tree performance degrades dramatically. If your view works correctly with small data sets but quits working when more data is added, your view might violate the growth rate characteristic restriction.





 Execution environment 

Your indexing functions work in a memory-constrained environment where the document forms part of the memory used in the environment. Your code's stack and document must fit within the memory. We limit documents to a maximum size of 64 MB.





 No JavaScript reducers when options.partitioned is true 

Design documents with options.partitioned set to true can't contain JavaScript reduce functions, only built-ins Erlang reducers such as _stats.







 Storing the view definition 

Each view is a JavaScript function. Views are stored in design documents. So, to store a view, IBM Cloudant simply stores the function definition within a design document.
If you encounter this issue, open the file in a more recent version of Microsoft Office and convert the file to the DOCX, PPTX, or XLSX format respectively, and then upload the DOCX, PPTX, or XLSX file.

Line breaks are inserted randomly
: When some files in Microsoft Office format are added to a collection, line breaks are inserted seemingly at random to the text that is stored in the html field in the collection's index. The unexpected line breaks can impact the efficiency of enrichments, such as custom rule recognition.

Cause: As part of their ingestion into Discovery, such files are converted from Office format to PDF format. When the conversion happens, textual content is sometimes lost due to the nature of a PDF file. While the new lines appear to be added at random, they typically get inserted in areas where text wraps in the original document, such as in narrow text boxes or to accommodate other inline elements, such as images or diagrams.

Solution: To avoid new line insertions, increase the width of text boxes in the original document. If the original document has a section where text wraps to accommodate an inline element, such as an image, move the image so that it is situated in its own section and the nearby text doesn't need to wrap around it. To test whether your fixes address the issue, you can convert the original file to a PDF file to check for unexpected carriage returns in the text.

After applying a pretrained Smart Document Understanding model to a PPT file, table boundaries are not recognized properly
: During the conversion process, text that is extracted from the table is confused with text that is outside the table in some PPT pages. This issue is more likely to occur in tables with a lot of text and that have footnotes displayed just outside the table border. If you encounter this issue, export the PPT file as a PDF file, and then upload the PDF file instead. Apply a user-trained Smart Document Understanding (SDU) model to the document, and then use the SDU tool to identify the tables in the document. The resulting model handles table boundaries properly and can extract text from the tables cleanly.





 PDF file troubleshooting tips 

Failed to parse document due to invalid encoding
: Enable OCR for the file.
Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 4 - The rev token.

The second fundamental IBM Cloudant rule is that each document revision is given its own unique revision token. Let's find out what it means.

You never need to generate a revision token - one is created for you when you add, update, or delete a document that uses the API.

A revision token consists of two parts:

A number 1, 2, 3, and so on, and a cryptographic hash of the document's body. (For the uninitiated, a hash is a digital "fingerprint" of some data. If the data changes, the fingerprint changes. No 2 fingerprints are the same, that is, no 2 documents with different content can have the same hash.)

You can see from the example on the screen that our document has a revision token (the key starts with _rev) that starts with a 1 followed by a dash. That indicates that this revision is the first revision of the document. The digits that start 04aa8... are the cryptographic hash of the document.

If we follow the lifecycle of a document, it starts with a revision 1. When it is modified later, it gets a revision 2 and so on. With each incrementing revision number, the hash changes because the content of the document is being modified too.

It is possible for a document to have more than one revision with the same number. In this case, two revision 3s exists. This scenario is called a conflict and is "normal" in some circumstances. We see why later in the course, but for now, we can assume that the revision number increments with each update to a document.

Let's follow the lifecycle of an example IBM Cloudant document:

When a new document is created (whether with an auto-generated _id or user-supplied _id), it is allocated a revision 1. You are sent the token in the response to your API request. Normally, you can discard the rev (unless you intend to modify the document soon).
"rev": "1-9ce...8d4"
}







 Quorum - writing and reading data 

In a distributed system, it's possible that a request might take some time to complete. A "quorum" mechanism is used to help determine when a request, such as a write or read, completes successfully.

For more information about quorum settings and their implications on dedicated IBM Cloudant systems, contact IBM Cloudant support.





 Time to live 

[Time to live](https://en.wikipedia.org/wiki/Time_to_live) (TTL) is a property of data, where after a relative amount of time, or at an absolute time, the data is considered expired. The data itself might be deleted or moved to an alternative (archive) location.

IBM Cloudant does not support Time to Live functions. The reason is that IBM Cloudant documents are only "soft" deleted, not deleted. The soft deletion involves replacing the original document with a [smaller record](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-tombstone-docs). This small record or "tombstone" is required for replication purposes. It helps ensure that the correct revision to use can be identified during replication.

If the TTL capability was available in IBM Cloudant, the resulting potential increase in short-lived documents and soft deletion records would mean that the database size might grow in an unbounded fashion.
Field name restrictions 

Field names that begin with the underscore character (_) are reserved in IBM Cloudant. This rule means that you can't normally have your own field names that begin with an underscore. For example, the field example would be accepted, but the field _example would result in a doc_validation error message.

See an example of a JSON document that attempts to create a field with an underscore prefix:

{
"_top_level_field_name": "some data"
}

See an error message that is returned when you attempt to create a field with an underscore prefix:

{
"error": "doc_validation",
"reason": "Bad special document member: _top_level_field_name"
}

However, if the field name is for an object that is nested within the document, you can use an underscore prefix for the field name.

See an example of a JSON document that attempts to create a field with an underscore prefix, nested within an object:

{
"another_top_level_field_name": "some data",
"another_field": {
"_lower_level_field_name": "some more data"
}
}

See an example success message (abbreviated) returned when a nested field with an underscore prefix is created:

{
"ok": true,
"id": "2",
"rev": "1-9ce...8d4"
}







 Quorum - writing and reading data 

In a distributed system, it's possible that a request might take some time to complete. A "quorum" mechanism is used to help determine when a request, such as a write or read, completes successfully.

For more information about quorum settings and their implications on dedicated IBM Cloudant systems, contact IBM Cloudant support.





 Time to live
At those times, IBM Cloudant returns the document normally, as though no conflict exists. However, the version that is returned isn't necessarily the most current version. Instead, the version is selected based on an internal algorithm that considers multiple factors. You must not assume that when documents are returned they're always the most current.





 How do I identify a document with a conflict? 

If a conflict with a document exists and you try to update it, IBM Cloudant returns a 409 response. If you try to update a document while you're offline, IBM Cloudant can't check for potential conflicts, and you don't receive a 409 response.

When this situation happens, it's best to check for document conflicts when you're back online. If you need to find document conflicts, use the following example map function:

function (doc) {
if (doc._conflicts) {
emit(null, [doc._rev].concat(doc._conflicts));
}
}

If you want to find conflicts within multiple documents in a database, write a [view](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-creating-views-mapreduce).





 What happens if I ignore conflicts? 

If you don't check for conflicts, or don't fix them, your IBM Cloudant database has the following problems:



* Document inconsistency increases because conflicting documents continue to multiply.
* Database size increases because documents with conflicts must be kept until the conflict is resolved.
* Performance degrades because it takes more work for IBM Cloudant to respond to each request since it has to go through all the conflicted documents to find the "best possible" version.







 How do I resolve conflicts? 

After you find a conflict, follow these four steps to resolve it.



1. [Get](https://cloud.ibm.com/docs/services/Cloudant?topic=Cloudant-document-versioning-and-mvccget-conflicting-revisions-mvcc) the conflicting revisions.
2.
In this part, we introduce another way of configuring secondary indexes that are called MapReduce.

MapReduce used to be the only way to configure secondary indexes in CouchDB and is still a popular way of querying data from within the document body.

To create a MapReduce index, you need to supply a JavaScript function that is wrapped in a special document that is called a design document to IBM Cloudant. Design documents' _id fields begin with _design/ for example, _design/mydesigndoc.

When IBM Cloudant receives the design document, it sets up a background indexing task, passing each document from the database to your JavaScript function in turn. The key-values that are emitted by your JavaScript function form the basis of the index that is persisted.

Let's look at some example JavaScript functions.

The function accepts one parameter - the document that is passed to it by the IBM Cloudant indexer. Every time your function calls emit the parameters that you pass from the key-value of the index.

The first example emits a key of doc.name, so in this case, it is an index for lookups by the name field. There is nothing (null) for the value.

The second example pre-processes the data before emitting. This pre-processing is a useful way of tidying up strings, trimming whitespace, lower and uppercasing text, applying default values to missing data, or constraining values to certain ranges, and so on.

The third example adds logic: only documents that are published make it to the index. This logic is equivalent to the partial filter selector that we saw with IBM Cloudant Query.

Indexes build asynchronously and cannot be used until they build completely. Once built, they can be used for selection by key, lists of keys, ranges of keys, and aggregation of data. For example, Find orders between two dates, and calculate the total value of the orders, which are grouped by month.

IBM Cloudant includes the four built-in reducers (or five if you count none).

_count
: For counting things.

_sum
: For totalizing values.

_stats
ibmcloud fn action invoke /_/myCloudant/read --blocking --result --param dbname <database_name> --param id <document_id>

Invoke the action to test reading a file. This example reads a file with the id of 9f86f4955e7a38ab0169462e6ac0f476, which is an empty file.

ibmcloud fn action invoke /_/myCloudant/read --blocking --result --param dbname test --param id 9f86f4955e7a38ab0169462e6ac0f476

The following example shows the command syntax for the action invoke command.

{
"_id": "9f86f4955e7a38ab0169462e6ac0f476",
"_rev": "1-967a00dff5e02add41819138abb3284d"
}





 Writing a document to an IBM Cloudant database 

You can use an action to create or update documents in an IBM Cloudant database.

Before you begin

Create a /whisk.system/cloudant[package binding](https://cloud.ibm.com/docs/openwhisk?topic=openwhisk-pkg_cloudantcloudant_db) that is configured for your IBM Cloudant account.



1. Store a document by using the write action in the package binding you created. Replace /_/myCloudant with your package name, replace <database_name> with the name of your database, <document_id> with your document ID, and <test_name> with a custom name.

Command syntax

ibmcloud fn action invoke /_/myCloudant/write --blocking --result --param <database_name> test --param doc "{"_id":"<document_id>","name":"<test_name>"}"

Example write action to a test database
* A .zip file that contains files in UIMA CAS XMI format





 CSV files 

You can upload a two-column CSV file that contains sample text from your local machine. Upload one CSV file at a time. The first column in the CSV file specifies the file name of the document. The second column in the file contains the document text. For an example of the required format, see the [documents-new.csv![External link icon](https://cloud.ibm.com/docs-content/v1/content/50eddb8fd81f33092880335ff107a78ff5cd0f65/icons/launch-glyph.svg)](https://watson-developer-cloud.github.io/doc-tutorial-downloads/knowledge-studio/documents-new.csv) file in the tutorial sample files.





 PDF files 

Text cannot be extracted from a PDF in some cases, depending on how the PDF was created. Typically, text can't be extracted from embedded fonts that don't map to unicode characters. If you are unsure whether text from a PDF can be extracted, you can try copying the text from the PDF and then pasting it into a text editor. If you do not see the same characters that are visible in the PDF itself, then the text extraction would likely fail.





 Formatted documents 

When formatted documents are converted to plain text, it's possible that losing the formatting could result in poor tokenization of words. For example, if a table row in a DOCX file contains cell values that do not end with a period, the values might be converted as one sentence. As another example, if a PDF document contains a very long word that is hyphenated at the end of a line, that word might be converted as two words. In cases like these, the documents might not be suitable for machine learning unless you pre-process the files to fix formatting limitations.





 Documents from another Watson Knowledge Studio workspace 

If you previously downloaded documents from a Knowledge Studio workspace, you can upload the .zip file that you downloaded.
{
"results": [ ...],
"last_seq": "10-g1AAAACbeJzLY"
}

The since parameter is used to define where in the changes feed you want to start from:

since=0
: The beginning of the changes feed.

since=now
: The end of the changes feed.

since=<a last seq token>
: From a known place in the changes feed.

At face value, following the changes feed seems as simple as chaining _changes API calls together. Then, IBM Cloudant passes the last_seq from one changes feed response into the next request's since parameter. But some subtleties to the changes feed need further discussion.





The IBM Cloudant Standard changes feed promises to return each document at least one time, which isn't the same as promising to return each document only one time. Put another way, it is possible for a consumer of the changes feed to see the same change again, or indeed a set of changes repeated.

A consumer of the changes feed must treat the changes idempotently. In practice, you must remember whether a change was already dealt with before you trigger an action from a change. A naive changes feed consumer might send a message to a smartphone on every change received. But a user might receive duplicate text messages if a change is not treated idempotently when replayed changes occur.

Usually these "rewinds" of the changes feed are short, replaying only a handful of changes. But in some cases, a request might see a response with thousands of changes replayed - potentially all of the changes from the beginning of time. The potential for rewinds makes the changes feed unsuitable for an application that expects queue-like behavior.

To reiterate, IBM Cloudant's changes feed promises to deliver a document at least one time in a changes feed, and gives no guarantees about repeated values across multiple requests.





The changes feed doesn't guarantee how quickly an incoming change appears to a client that consumes the changes feed. Applications must not be developed with the assumption that data inserts, updates, and deletes are immediately propagated to a changes reader.
Replication is complete when the most recent version of each document in the source transfers to the destination database. Transfers include new documents, updates to existing documents, and deletions. Only the most recent version of a document remains after replication; older versions are omitted.

The source database remains unaltered by replication, apart from checkpoint data that is written to it to allow partial replications to resume from the last known position. Any pre-existing data in the destination database remains.



 How to start replicating with the dashboard 

The IBM Cloudant Dashboard provides a convenient user interface to trigger replication. Click Replication on the IBM Cloudant Dashboard, and click Start Replication. Complete the following Replication form:

Zoom

![This screen capture shows the replication form with all the proper fields filled out. There is a source section, which is the local database, and a target section, which is the new database and the authentication. In the Options section, you select one time replication or repeating, and add a replication document.](https://cloud.ibm.com/docs-content/v1/content/522c6f62358b063f921283400a601c3f9bc66f08/Cloudant/images/replication_guide_2.png)

Figure 2. Replication form

For security purposes, the IBM Cloudant team recommends that you use IAM API keys or IBM Cloudant legacy authentication [API keys](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountapi-keys) rather than account-level credentials for replication jobs. For more information, see [Managing access](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-managing-access-for-cloudant) or legacy [authentication](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountauthentication) and [authorization](https://cloud.ibm.com/docs/Cloudant?topic=Cloudant-work-with-your-accountauthorization) documentation.
ISO-8601 is compact, readable, and sorts nicely, as do a timestamp (milliseconds since 1970).

If you need to use IBM Cloudant Query on one of the component parts, then that would need to be broken out explicitly in the document.

That's the end of this part. The next part is called Replication.







 Replication video 

Learn what replication means in IBM Cloudant, as well as different types of replication and how they work.



* Replication video script

Welcome to the Introduction to IBM Cloudant course, an 17-part video series that gives you an overview of the IBM Cloudant database-as-a-service.

This video is part 14 - Replication.

Replication is a core feature of IBM Cloudant. It is the transfer of data from one database (the source) to another (the target).

The source and target databases can reside on the same IBM Cloudant service, or be geographically separated, for example, a US IBM Cloudant database that replicates to one in Europe.

The IBM Cloudant replication protocol is shared with Apache CouchDB, so replication is often used by enterprises that copy data from a cloud-based database to one running CouchDB in their own location.

PouchDB, a JavaScript-based CouchDB clone that runs in Node.js stacks or the web browser, can also be used to replicate data between IBM Cloudant or CouchDB, each way, either to or from.

The IBM Cloudant Sync libraries are native iOS or Android apps that sync data to and from an IBM Cloudant service.

Replication is a one-way operation from source to target, which moves all data (deletions, conflicts, attachments, as well as documents) and can be triggered in one of two ways:



1. Run until all the data from the source reaches the target and then stop.
2. The same as one, but the replication runs continuously forever, transferring new data from the source to the target as it arrives.



Replication can also be resumed from where it last stopped. IBM Cloudant keeps a note of checkpoints between replicating parties to allow the resumption of a pre-existing replication from its last known position.

The IBM Cloudant Dashboard includes a replication tab.