vania-janet/multiturn-rag-retrieval-data huggingface dataset:
When you issue a view request that specifies the keys parameter, the results are returned in the same order as the supplied keys array.

See the example of using HTTP to request the records in reversed sort order:

GET $SERVICE_URL/$DATABASE/_design/$DDOC/_view/$VIEW_NAME?descending=true HTTP/1.1
Accept: application/json

See the example of requesting the records in reverse sort order.

Client libraries use POST method instead of GET because they have a similar behavior.



* Curl
* Java
* Node
* Python
* Go



curl -X GET "$SERVICE_URL/users/_design/allusers/_view/getVerifiedEmails?descending=true"

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostViewOptions;
import com.ibm.cloud.cloudant.v1.model.ViewResult;

Cloudant service = Cloudant.newInstance();

PostViewOptions viewOptions = new PostViewOptions.Builder()
.db("users")
.ddoc("allusers")
.view("getVerifiedEmails")
.descending(true)
.build();

ViewResult response =
service.postView(viewOptions).execute()
.getResult();

System.out.println(response);

const { CloudantV1 } = require('@ibm-cloud/cloudant');

const service = CloudantV1.newInstance({});

service.postView({
db: 'users',
ddoc: 'allusers',
view: 'getVerifiedEmails',
descending: true
}).then(response => {
console.log(response.result);
});

from ibmcloudant.cloudant_v1 import CloudantV1

service = CloudantV1.new_instance()
* [Smart formatting](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting) converts dates, times, numbers, currency values, phone numbers, and internet addresses into more readable, conventional forms in final transcripts. For US English, you can also provide keyword phrases to include certain punctuation symbols in final transcripts. Smart formatting is beta functionality.
* [Numeric redaction](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingnumeric-redaction) redacts, or masks, numeric data from a final transcript. Redaction is intended to remove sensitive personal information, such as credit card numbers, from final transcripts. Numeric redaction is beta functionality.
* [Profanity filtering](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingprofanity-filtering) censors profanity from transcripts and metadata.







 Response metadata 



* [Maximum alternatives](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metadatamax-alternatives) provide possible alternative transcripts. The service indicates final results in which it has the greatest confidence.
* [Word confidence](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metadataword-confidence) returns confidence levels for each word of a transcript.
* [Word timestamps](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metadataword-timestamps) return timestamps for the start and end of each word of a transcript.







 Processing and audio metrics 



* [Processing metrics](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-metricsprocessing-metrics) provide detailed timing information about the service's analysis of the input audio.
If present and if a description is added to it, then the text is taken from the external node name field instead.The description that is displayed in the disambiguation list comes from the name (or external node name) of the last node that is processed in the branch where the intent match occurs. For more information, see Disable jumped-to utility nodes]Choosing nodes to disable] Service prompts the user to choose from a list of dialog options, including Cancel an account, Cancel a product order, and None of the above.Service prompts the user to choose from a list of dialog options_ including Cancel an account_ Cancel a product order_ and None of the above_.] Service prompts the user to choose from a list of dialog options, including Capture date information.Service prompts the user to choose from a list of dialog options_ including Capture date information_.] ! ! ! .!]Notice that your assistant recognizes the term today in the user input as a date, a mention of the @sys-date entity. If your dialog tree contains a node that conditions on the @sys-date entity, then it is also likely to be included in the list of disambiguation choices. This image shows it included in the list as the Capture date information option.!]<-- </section "id="section-dialog-runtime-disambig-example" "> --><-- <section "id="section-dialog-runtime-disambig-edit" "> --> Editing the disambiguation configuration Disambiguation is enabled automatically for all new dialog skills. You can change the settings that are applied automatically to disambiguation from the Options page.To edit the disambiguation settings, complete the following steps:<-- <ol> -->1. From the Skills menu, click Options.2. Click Disambiguation.3. In the Disambiguation message field, add text to show before the list of dialog node options. For example, What do you want to do?4.
detag Document.text as DetagedDoc
annotate
element 'a' as Anchor;

-- Remap the Anchor Tags
create view AnchorTag as
select Remap(A.match) as anchor
from Anchor A;

If the argument to Remap is not a derived text object or a span over a derived text object , the function generates an error. If the argument is null, the function returns null.





 SpanBetween 

The SpanBetween function takes two spans as input and returns the span that exactly covers the text between the two spans if the spans are based on the same text object, and returns null if they are based on different text objects:

SpanBetween(['IgnoreOrder',] <span1>, <span2>)

When the optional IgnoreOrder parameter is used, the order of the two spans is ignored by the AQL compiler.

If no text exists between the two spans, then SpanBetween returns an empty span that starts at the end of <span1>.

Like CombineSpans, SpanBetween is sensitive to the order of its inputs, unless you use the IgnoreOrder parameter. So

SpanBetween([5, 10], [50, 60])

returns the span [10, 50], while

SpanBetween([50, 60], [5, 10])

returns the span [60, 60].

If the argument to SpanBetween is null, then the function returns null.





 SpanIntersection 

The SpanIntersection function takes two spans as input and returns a span that covers the text that both inputs cover if the spans are based on the same text object, and returns null if they are based on different text objects:

SpanIntersection(<span1>, <span2>)

For example,

SpanIntersection([5, 20], [10, 60])

returns the span [10, 20], while

SpanIntersection([5, 20], [7, 10])

returns the span [7, 10].

If the two spans do not overlap, then SpanIntersection returns null.
The problems were related to a new version of the optical character recognition (OCR v2) feature that was enabled automatically for English, German, French, Spanish, Dutch, Brazilian Portuguese, and Hebrew collections during that timeframe. The new version changes sentence boundaries in ways that can negatively impact other functions, including element identification in contracts and the document labeling view in the entity extractor tool.

If you experience any of these issues with documents that were added or processed during this period, revert the version of OCR that is applied to the documents. Starting on 12 November 2022, OCR v1 is applied to all collections where OCR is enabled. To go back to using OCR v1, make a change that will reprocess the affected documents. For example, you can re-add documents that were added during the timeframe to reprocess them. Or you can reprocess an entire collection.

To reprocess a collection, from the Manage collections page, open the collection, and then go to the Processing settings tab. Expand the More processing settings section, set the OCR switch to Off, and then set it back to On. Click Apply changes and reprocess to reprocess your collection.





 2 November 2022 

A new and improved optical character recognition technology is available
: A new version of optical character recognition technology is now available. This latest version (OCR v2) is used automatically when you enable OCR for English, German, French, Spanish, Dutch, Brazilian Portuguese, and Hebrew collections in all IBM Cloud service plans. The new optical character recognition model was developed by IBM Research to be better at extracting text from scanned documents and other images that have the following limitations:



* Low quality images due to incorrect scanner settings, insufficient resolution, bad lighting (such as with mobile capture), loss of focus, unaligned pages, and badly printed documents
* Documents with irregular fonts or a variety of colors, font sizes, and backgrounds







 1 November 2022 

Entity extractor loads the first 40,000 characters from training data documents
: Even extra long documents from the collection that you use to define custom entity examples are loaded into the document view of the tool.
For instance, the strings "21, and 1983!" include punctuation that the service returns in its text message. Although the punctuation results in silence, the audio timing for the word does not include that silence.

For example, consider input text that contains the following conditional statement:

If it is sunny, I will go to the beach.

The service returns timing information for all strings of the input, including "sunny, and beach., both of which end in punctuation that produces silence. But the timing information for sunny, does not include the silence that is produced by the comma, and the timing information for beach." does not include the silence for the period. The information reflects only the timing of the spoken strings.





 Timings for SSML text 

When the service synthesizes plain text, it returns all input characters except for blank spaces as part of the strings in its word timing response. The same is not true of SSML, since some SSML elements do not generate audio. The following list summarizes the SSML elements that can impact word timing information:



* <say-as> indicates how the text that is enclosed between the opening and closing <say-as> tags is to be handled in the normalization step. Attributes specify how the embedded text is to be spoken. The following example indicates how the date is to be spoken:

The baby was born on <say-as interpret-as="date" format="mdy">3/4/2016</say-as>.

The service returns timing information for the following strings: "The, baby, was, born, on, 3/4/2016. The service normalizes the string 3/4/2016 as march fourth two thousand sixteen. The word timing information for the string reflects the start time of march and the end time of sixteen".

The following example indicates that the word Hello is to be spelled out:

<say-as interpret-as="letters">Hello</say-as>.

The service returns timing information for the string "Hello.. The service spells the word letter-by-letter during the normalization step.
The smart formatting feature understands and returns the multigram units that the model generates. If you apply your own post-processing to transcription results, you need to handle these units appropriately.









 Smart formatting results 

The following table shows examples of final transcripts both with and without smart formatting. The transcripts are based on US English audio.



Table 2. Smart formatting example transcripts

 Information Without smart formatting With smart formatting 

 Dates I was born on ten oh six nineteen seventy I was born on 10/6/1970 
 I was born on the ninth of December nineteen hundred I was born on 12/9/1900 
 Today is June sixth Today is June 6 
 Times The meeting starts at nine thirty AM The meeting starts at 9:30 AM 
 I am available at seven EST I am available at 7:00 EST 
 We meet at oh seven hundred hours We meet at 0700 hours 
 Numbers The quantity is one million one hundred and one The quantity is 1000101 
 One point five is between one and two 1.5 is between 1 and 2 
 Phone numbers Call me at nine one four two three seven one thousand Call me at 914-237-1000 
 Call me at one nine one four nine oh nine twenty six forty five Call me at 1-914-909-2645 
 Currency values You owe me three thousand two hundred two dollars and sixty six You owe me $3202.66 
 The dollar rose to one hundred and nine point seven nine yen from one hundred and nine point seven two yen The dollar rose to 109.79 yen from 109.72 yen 
 Internet email and web addresses My email address is john dot doe at foo dot com My email address is [john.doe@foo.com](mailto:john.doe@foo.com) 
 I saw the story on yahoo dot com I saw the story on yahoo.com 
 Combinations The code is zero two four eight one and the date of service is May fifth two thousand and one The code is 02481 and the date of service is 5/5/2001 
 There are forty seven links on Yahoo dot com now There are 47 links on Yahoo.com now 







 Smart formatting results for long pauses 

In cases where an utterance contains long enough pauses of silence, the service can split the transcript into two or more final results.
The IN predicate compares a value or values with a collection of values. The range of values is either defined by a query or a list of expressions that are enclosed in parentheses. The query must identify a number of columns that are the same as the number of expressions that are specified to the left of the IN keyword. In addition, the number of elements in the list of expressions must be the same as the number of expressions that are specified to the left of the IN keyword. If NOT is specified, the result is reversed.

The LIKE predicate searches for strings that have a certain pattern. The pattern is specified by a string in which certain characters have a special meaning.



* The underscore character _ represents any single character.
* The percent sign % represents a string of zero or more characters.
* Any other character represents itself. Thus trailing blanks in a pattern are part of the pattern. If NOT is specified, the result is reversed.



The RLIKE predicate searches for a regular expression pattern in a string. If the pattern expression is found, the result is true. If the pattern expression is not found, the result is false. If the value of any of the arguments is null, the result of the RLIKE predicate is unknown. If NOT is specified, the result is reversed.

The regular expression pattern must be a Java™ regular expression as defined by Java class java.util.regex.Pattern. Meta characters that start with a must be escaped for the regular expression to work, for example, use \d instead of d in a pattern string to represent a digit. For more information, such as supported meta characters and predefined character classes, see the latest Java documentation.

The IS NULL predicate tests for null values. The result of a NULL predicate cannot be unknown. If the value of the expression is null, the result is true. If the value is not null, the result is false. If NOT is specified, the result is reversed.

The IS DISTINCT FROM predicate compares two expressions and evaluates to TRUE if their values are not identical. The result of a DISTINCT predicate cannot be null. If NOT is specified, the result is reversed.
And the corrected text is added to the input.text field.

If you want to ask users to confirm the assistant's understanding of their meaning, you can do so in a way that takes into account that their input might have been corrected. Set the condition for the dialog node or conditional response that is asking for confirmation to original_text. This means that if the user's input was automatically corrected, show the corresponding response. And the response can contain the expression: You said: <? input.original_text ?>. Did you mean: <? input.text ?>?

Remember, the input.text field stores either the never-corrected original text from the user or the user’s text after it is corrected. The input.original_text field is only created if the input is corrected. And the user’s incorrect input is stored in it.”

Previous topic:[Connecting customers with support](https://cloud.ibm.com/docs/assistant-data?topic=assistant-data-dialog-support)
The default is 10. The maximum for the count and offset values together in any one query is 10000.





 offset 

The number of query results to skip at the beginning. For example, if the total number of results that are returned is 10, and the offset is 8, it returns the last two results. The default is 0. The maximum for the count and offset values together in any one query is 10000.





 return 

A comma-separated list of the portion of the document hierarchy to return. Any of the document hierarchy are valid values.





 sort 

A comma-separated list of fields in the document to sort on. You can optionally specify a sort direction by prefixing the field with - for descending order or + for ascending order. Ascending order is the default sort direction.

The sort parameter is currently available for use only with the API; it is not available through the tooling.





 bias 

Adjusts search results to bias towards certain results, for example, documents that were published most recently. bias must be set to either a date type field or a number type field, for example bias=publication_date or bias=field_1. When a date type field is specified, returned results are biased towards field values closer to the current date. When a number type field is specified, returned results are biased towards higher field values. This parameter cannot be used in the same query as the sort parameter.

The bias parameter is currently available for use only with the API; it is not available through the tooling.





 passages 

A boolean that specifies whether the service returns a set of the most relevant passages from the documents returned by a query that uses the natural_language_query parameter. The passages are generated by sophisticated Watson algorithms to determine the best passages of text from all of the documents returned by the query. The default is false.

The passages parameter can only be used on private collections. It cannot be used on the IBM Watson™ Discovery News collection.

Discovery attempts to return passages that start at the beginning of a sentence and stop at the end using sentence boundary detection.
54dd2745-7718-487f-85e6-a67e758dd945 my-test-key-17
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
90138be5-1dd1-4eea-b7c1-73cc8609e7f7 my-test-key-19
cf0a3d8b-4856-4aa5-be6a-88cca465eab0 my-test-key-20

 this is the same list sorted in KEY_ID order
 this is the order we expect when listing keys

10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
215f592c-3e42-47b6-867c-3cc1cce8a8fd my-test-key-2
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
3b4b0511-00df-48e6-ad65-fad35bb6ce17 my-test-key-13

3d683d56-d340-40cb-b8b6-605dcfded01f my-test-key-15
3ee0d859-2968-4444-8e4d-3d28614f055c my-test-key-10
54dd2745-7718-487f-85e6-a67e758dd945 my-test-key-17
62090ba8-7363-408e-9edc-aa9fb69c09be my-test-key-12
768c9ceb-368b-46f7-9b1d-be211e036fe1 my-test-key-4
The service applies redaction only to the final transcript of a recognition request, just before it returns the results to the client and after text normalization is complete.



 Language differences 

The feature works exactly as described for US English models but has the following differences for Japanese and Korean models.



 Japanese 

Japanese redaction has the following differences:



* In addition to masking strings of three or more consecutive digits, redaction also masks street addresses and numbers, regardless of whether they contain fewer than three digits.
* Similarly, redaction also masks date information in Japanese-style birth dates. In Japanese, date information is usually presented in Common Era format but sometimes follows Japanese style, particularly for birth dates. In this case, the year and month are masked even though they contain just one or two digits.

For example, a Japanese-style birth date without redaction is 平成 30年 2月. With redaction, the date becomes 平成 XX年 X月.







 Korean 

Korean redaction has the following differences:



* The smart formatting feature is not supported. The service still performs numeric redaction for Korean, but it performs no other smart formatting.
* Isolated digit characters are reduced, but possible digit characters that are included as part of Korean phrases are not. For example, the character 이in the following phrase is not replaced by an X because it is adjacent to the following character:

이입니다

If the 이 character were separated from the following character by a space, it would be replaced by an X, as described in [Numeric redaction results](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingnumeric-redaction-results).









 Numeric redaction results 

The following table shows examples of final transcripts both with and without numeric redaction in each supported language.



Table 5. Numeric redaction example transcripts

 Language Without redaction With redaction 

 US English my credit card number is four one four seven two my credit card number is XXXXX
e9b08526-3227-406c-b83f-c28844bd952b my-test-key-9
3ee0d859-2968-4444-8e4d-3d28614f055c my-test-key-10
bc459a2f-56cb-445e-b4ee-9d1866b57560 my-test-key-11
62090ba8-7363-408e-9edc-aa9fb69c09be my-test-key-12
3b4b0511-00df-48e6-ad65-fad35bb6ce17 my-test-key-13
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
3d683d56-d340-40cb-b8b6-605dcfded01f my-test-key-15
10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
54dd2745-7718-487f-85e6-a67e758dd945 my-test-key-17
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
90138be5-1dd1-4eea-b7c1-73cc8609e7f7 my-test-key-19
cf0a3d8b-4856-4aa5-be6a-88cca465eab0 my-test-key-20

 this is the same list sorted in KEY_ID order
 this is the order we expect when listing keys

10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
The split_transcript_at_phrase_end parameter directs the service to split the transcript into multiple final results based on semantic features of the input. Setting the parameter to true causes the service to split the transcript at the conclusion of meaningful phrases such as sentences. The service bases its understanding of semantic features on the base language model that you use with the request along with a custom language model or grammar that you use. Custom language models and grammars can influence how and where the service splits a transcript.

If you apply a custom language model to speech recognition, the service is likely to split the transcript based on the content and nature of the model's corpora. For example, a model whose corpora include many short sentences biases the service to mimic the corpora and split the input into multiple, shorter final results. Similarly, a grammar that expects a series of six digits can influence the service to insert splits to accommodate that series of digits.

However, the degree to which custom language models and grammars influence the service's splitting of a transcript depends on many factors. Such factors include the amount of training data with which the custom model is built and the qualities of the audio itself. Small changes in the audio can affect the results.

Regardless, the service can still produce multiple final results in response to pauses and silence. If you omit the split_transcript_at_phrase_end parameter or set it to false, the service splits transcripts based solely on the pause interval. The pause interval has precedence over splitting due to semantic features. And if used together on a single request, the end_of_phrase_silence_time parameter has precedence over the split_transcript_at_phrase_end parameter. For more information, see [End of phrase silence time](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-parsingsilence-time).



 Split transcript at phrase end results 

When the split transcript at phrase end feature is enabled, each final result in the transcript includes an additional end_of_utterance field that identifies why the service split the transcript at that point:



* full_stop - A full semantic stop, such as for the conclusion of a grammatical sentence.