vania-janet/multiturn-rag-retrieval-data huggingface dataset:
LITERAL LITERAL Treat the expression as a sequence of literal characters, ignoring the normal regular expression escape sequences. 
 MULTILINE MULTILINE Make the characters ^ and $ match the beginning and end of any line, as opposed to the beginning and end of the entire input text. 
 UNIX_LINES UNIX_LINES Treat only the UNIX™ newline character as a line break, ignoring the carriage return character r. 





* Follow these guidelines to make your extractors run faster and be easier to maintain:



* Avoid long, complex regular expressions and use simpler, smaller regular expressions that are combined with AQL statements.
* Avoid unnecessary use of lookahead and lookbehind in regular expressions. You can usually achieve the same effect by adding predicates to the having clause of your extract statement.
* Use token constraints in your regular expression extraction specifications when possible.









 Examples 

Example 1: Using canonical Unicode equivalence to determine matches

This example shows you how to find capitalized words that are not given names. Canonical Unicode character equivalence is used to determine matches. Notice the usage of the flag ‘CANON_EQ’ and that regex is performed on tokens:

create dictionary FirstNamesDict as
(
'Aaron', 'Matthew', 'Peter'
);
create view NotFirstName as
extract
regex /[A-Z]/
with flags 'CANON_EQ'
on 1 token in D.text
as word
from Document D
having Not(ContainsDict('FirstNamesDict', word));

Example 2: Usage of capturing groups

The following example demonstrates the usage of capturing groups in an extract regex statement. The code extracts the fields of a United States phone number by using capturing groups:

create view Phone as
extract regex /(d{3})-(d{3}-d{4})/
on between 4 and 5 tokens in D.text
return
group 1 as areaCode
and group 2 as restOfNumber
and group 0 as fullNumber
from Document D;

Example 3: Applying multiple regex over input text
Changing the order helps to avoid bias that can be introduced by a percentage of people who always pick the first option without carefully reviewing all of their choices beforehand.<-- </section "id="section-dialog-runtime-choose-nodes" "> --><-- <section "id="section-dialog-runtime-handle-none" "> --> Handling none of the above When a user clicks the None of the above option, your assistant strips the intents that were recognized in the user input from the message and submits it again. This action typically triggers the anything else node in your dialog tree.To customize the response that is returned in this situation, you can add a root node with a condition that checks for a user input with no recognized intents (the intents are stripped, remember) and contains a suggestion_id property. A suggestion_id property is added by your assistant when disambiguation is triggered.Add a root node with the following condition: intents.size()==0 && input.suggestion_id
This condition is met only by input that has triggered a set of disambiguation options of which the user has indicated none match her goal.Add a response that lets users know that you understand that none of the options that were suggested met their needs, and take appropriate action.Again, the placement of nodes in the tree matters. If a node that conditions on an entity type that is mentioned in the user input is higher in the tree than this node, its response is displayed instead.<-- </section "id="section-dialog-runtime-handle-none" "> --><-- <section "id="section-dialog-runtime-disambig-test" "> --> Testing disambiguation To test disambiguation, complete the following steps:<-- <ol> -->1. From the Try it out pane, enter a test utterance that you think is a good candidate for disambiguation, meaning two or more of your dialog nodes are configured to address utterances like it.2.
The direction value is asc for ascending, and desc for descending.

If you exclude the direction value, the default asc is used.

See the following example of simple sort syntax:

[
{
"fieldName1": "desc"
},
{
"fieldName2": "desc"
}
]

See the following example of simple sort, assuming default direction of "ascending" for both fields:

[
"fieldNameA", "fieldNameB"
]

A typical requirement is to search for some content by using a selector, then to sort the results according to the specified field, in the direction preferred.

To use sorting, ensure that the following are true:



* At least one of the sort fields is included in the selector.
* An index is already defined, with all the sort fields in the same order.
* Each object in the sort array has a single key.



If an object in the sort array doesn't have a single key, the resulting sort order is implementation-specific and might change.

Currently, IBM Cloudant Query doesn't support multiple fields with different sort orders, so the direction must either be all ascending or all descending.

If the direction is ascending, you can use a string instead of an object to specify the sort fields.

For field names in text search sorts, it's sometimes necessary for a field type to be specified, for example:

{
"<fieldname>:string": "asc"
}

If possible, an attempt is made to find the field type based on the selector. In ambiguous cases, the field type must be provided explicitly.



Table 1. When to specify the field type

 Which index is used by query? Field type requirement 

 JSON index It's not necessary to specify the type of sort fields in the query. 
 Text index of all fields in all documents Specify the sort field in the query if the database contains documents where the sort field has one type. Also, specify the sort field in the query if it contains documents where the sort field has a different type. 
 Any other text index Specify the type of all sort fields in the query.
This beta feature is currently supported in English only. For details, see Language support]] ! ! ! ! ! .Each query is deduplicated independently, so deduplication across offsets is not supported.Deduplication is performed after passages are extracted and aggregations are calculated, so if you include the passages parameter in your query, passages are returned from all documents in the query results, before deduplication. If you run an aggregation and query together, the aggregation results include data from all documents returned, before deduplication.Deduplication is performed on returned fields only. If you choose to specify the return= in your query, include the field you are deduplicating on.To apply deduplication, use the following syntax in your query. Replace {field} with the name of the field you wish to deduplicate on. The specified {field} must be a string, such as title. &deduplicate.field={field}
When deduplicating, the JSON response includes "duplicates_removed": x, where x is the number of documents removed from the results.<-- <section "id="section-deduplicatewds" "> --> Deduplicating documents in Watson Discovery News News articles might be syndicated to several news outlets, and IBM Watson™ Discovery News picks up each of them, resulting in duplicate articles. Because IBM Watson™ Discovery News has articles that might be syndicated to several news outlets, a query to IBM Watson™ Discovery News might potentially return several identical or nearly identical articles in query results. Using deduplication removes most duplicate articles from your search queries.Discovery deduplicates by using approximate matching on the title field and therefore a field doesn't need to be specified.To apply deduplication, use the following parameter in your query. This query automatically deduplicates on the title field in IBM Watson™ Discovery News. &deduplicate=true
If you prefer to deduplicate on a field other than title, use the following syntax in your query. Replace {field} with the name of the field you wish to deduplicate on. The specified {field} must be a string.
The service extracts and translates speaker names separately to guarantee consistent translation.
* Paragraphs - The text to be translated is grouped into paragraphs. Each paragraph can span multiple cues, but it always consists of a full set of cue lines. In other words, each paragraph consists of one or more cues in their entirety, with each cue contained fully in a single paragraph.

A single cue can contain one or more lines of text (for example, two short sentences). The service creates paragraph breaks only at cue line boundaries to preserve the count of lines in the cue. For languages with punctuation, a paragraph generally maps to a complete sentence. For languages without punctuation, a paragraph can contain multiple sentences, which can adversely affect the distribution of lines into cues in the translated document.
* Comments, notes, and titles - For formats that permit these elements, the service preserves the original text and adds translation that is prefixed by language code. Because this information is intended for use by the author, the service maintains the text in both its original and translated forms.







 Other file formats 

Table 4 lists all other file formats that the service supports for translation.



Table 4. Other file formats

 File format File extension Content type 

 Adobe® Portable Document Format [1] pdf application/pdf 
 Extensible Markup Language .xml text/xml 
 HyperText Markup Language .htm text/html 
 .html text/html 
 .xhtml text/html 
 JavaScript Object Notation [2] .json text/json 
 Plain text .txt text/plain 
 Rich Text Format .rtf application/rtf 



Notes:



1. For PDF files, translation is experimental functionality. The quality of PDF translation is still largely an alpha release. The translation works best for single-column PDF documents that do not include many tables or images.
2. For JSON files, values with type string or string array are translated.
In the following example, the new lines are replaced with n and carriage returns are deleted to make sure the content fits in one line between the inverted commas (see [scalars in double-quoted style](https://yaml.org/spec/1.2.2/double-quoted-style)). You can also use other valid YAML variations.

Example:

env:
logging:
syslog:
hostname: 9.20.7.92
port: 16999
server: "-----BEGIN CERTIFICATE-----nMIIFCTCCAvECFEp7wJLz4jNStIsVH2dUeHDN26ZyMA0GCSqGSIb3DQEBCwUAMEExnCzAJBgNVBAYTAlVTMRkwFwYDVQQKDBBMb2dzdGFzaCBUZXN0IENBMRcwFQYDVQQDnDA5jYS5leGFtcGxlLm9yZzAeFw0yMzAxMDUxNjU0MTNaFw0yNDAxMDUxNjU0MTNanMEExCzAJBgNVBAYTAlVTMRkwFwYDVQQKDBBMb2dzdGFzaCBUZXN0IENBMRcwFQYDnVQQDDA5jYS5leGFtcGxlLm9yZzCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCnggIBANe7PR4XaTXtF6h3FhWe/R4BSTVylXWopA51+ppcJ3BOMPjmRMNJ3tFAFE3hnF4d0RHBNJOZF0+ogT0ZEseTe4mqJXk3RgfMSrLaymNgzaefD67uhQ9ZzznE3kIXenmzh/A8aDwhaUMifIKxekisrmpvjDwUJtaSs3pb27W+cOmzAPZ3cmOs09tELLY134nf52sp0ZqFSOgvCwcdt88PFVMm2rrFgwxP2gLgOkZL4OsM9sQykYEPR28unS+P90VnqnYPy27xqJNss4OdZCJrjkS7lv2PbBxSoFjDq/yLjnDV8khW2+6w0MFRvamoKL34nn/XoXN6VCathSxcwvXg0x3wwTxa5Hevb0iziNGXHjZ9bXt+8bnu/Bhsa7KwaoUt9nrJJeMy0KNsdQyWhMJE904YKm9Eo/S92rrcNWzmzBIV0iecOHc24iw3SIXOnoAKNYn1GtDOQChSEeb7en25s1fjWTqIDyDOktWjp9DXu1ips9YDb7GKZ7raOoQnsPkGrREnKOWClkWQ4qIXJ9LH73ytR1h8+AsGyInaan5ehnz7JC5SFhE96wPzJDaXCKNHBP/entfwQ0BTbgO6z8gPE8JlPGXTmdf9YF5NxMd4oJA7u7Y6x
If there are misspelled terms that you expected your assistant to correct, but it did not, then review the rules that your assistant uses to decide whether to correct a word to see if the word falls into the category of words that your assistant intentionally does not change.





 Autocorrection rules 

To avoid overcorrection, your assistant does not correct the spelling of the following types of input:



* Capitalized words
* Emojis
* Location entities, such as states and street addresses
* Numbers and units of measurement or time
* Proper nouns, such as common first names or company names
* Text within quotation marks
* Words containing special characters, such as hyphens (-), asterisks (*), ampersands (&), or at signs (@), including those used in email addresses or URLs.
* Words that belong in this skill, meaning words that have implied significance because they occur in entity values, entity synonyms, or intent user examples.

Mentions of a contextual entity can be corrected inadvertently. That's because terms that function as contextual entity mentions are fluid; they cannot be predetermined and avoided by the spell checker function in the way a list of dictionary-based terms can be.



If the word that is not corrected is not obviously one of these types of input, then it might be worth checking whether the entity has fuzzy matching enabled for it.



 How is spelling autocorrection related to fuzzy matching? 

Fuzzy matching helps your assistant recognize dictionary-based entity mentions in user input. It uses a dictionary lookup approach to match a word from the user input to an existing entity value or synonym in the skill's training data. For example, if the user enters boook, and your training data contains a @reading_material entity with a book value, then fuzzy matching recognizes that the two terms (boook and book) mean the same thing.

When you enable both autocorrection and fuzzy matching, the fuzzy matching function runs before autocorrection is triggered. If it finds a term that it can match to an existing dictionary entity value or synonym, it adds the term to the list of words that belong to the skill, and does not correct it.
"document_id":"ab7be56bcc9476493516b511169739f0",
"passage_score":15.230205287402338,
"passage_text":"a privately held company that provides hybrid cloud recovery, cloud migration and business continuity software for enterprise data centers and cloud infrastructure.",
"start_offset":120,
"end_offset":300,
"field":"text"
},
{
"passage_text":"Disaster Recovery Services for Hybrid Cloud</title></head>n<body>nnn<p>Published: Thu, 27 Oct 2016 07:01:21 GMT</p>n",
"passage_score":10.153470191601558,
"document_id":"fbb5dcb4d8a6a29f572ebdeb6fbed20e",
"start_offset":70,
"end_offset":120,
"field":"html"
}
] !
}
Show more
<-- <section "id="section-passages_fields" "> --> passages.fields A comma-separated list of fields in the index that passages are drawn from. If this parameter is not specified, then passages from all root-level fields are included.You can specify fields in both the return and passages.fields parameters. When you specify both parameters, each with different values, they are treated separately.For example, the request might include the parameters "return": "docno"] and "passages":{"fields": "body"] ! ! ! ! ! ! ! ! ! ! ! . The body field is specified in passages.fields, but not in return.
"text" : string,
"text_normalized" : string,
"row_index_begin" : int,
"row_index_end" : int,
"column_index_begin" : int,
"column_index_end" : int
},
...
],
"row_headers" :
{
"cell_id" : string,
"location" : {
"begin" : int,
"end" : int
},
"text" : string,
"text_normalized" : string,
"row_index_begin" : int,
"row_index_end" : int,
"column_index_begin" : int,
"column_index_end" : int
},
...
],
"body_cells" :
{
"cell_id" : string,
"location" : {
"begin" : int,
"end" : int
},
"text" : string,
"row_index_begin" : int,
"row_index_end" : int,
"column_index_begin" : int,
"column_index_end" : int,
"row_header_ids": string ],
"row_header_texts": string ],
"row_header_texts_normalized": string ],
"column_header_ids": string ],
"column_header_texts": string ],
"column_header_texts_normalized": string ],
"attributes" :
{
"type" : string,
"text" : string,
"location" : {
"begin" : int,
"end" : int
}
},
...
]
},
...
],
"key_value_pairs":
{
"key": {
"cell_id": string,
"location": {
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
3d683d56-d340-40cb-b8b6-605dcfded01f my-test-key-15
10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
54dd2745-7718-487f-85e6-a67e758dd945 my-test-key-17
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
90138be5-1dd1-4eea-b7c1-73cc8609e7f7 my-test-key-19
cf0a3d8b-4856-4aa5-be6a-88cca465eab0 my-test-key-20

 this is the same list sorted in KEY_ID order
 this is the order we expect when listing keys

10954149-0217-472d-8137-42fc330dbd03 my-test-key-16
16276369-ba81-4eb5-be92-f13512147ad4 my-test-key-14
215f592c-3e42-47b6-867c-3cc1cce8a8fd my-test-key-2
3a5a3b52-6942-463d-9f7a-5b216bbf5123 my-test-key-18
3b4b0511-00df-48e6-ad65-fad35bb6ce17 my-test-key-13

3d683d56-d340-40cb-b8b6-605dcfded01f my-test-key-15
3ee0d859-2968-4444-8e4d-3d28614f055c my-test-key-10
For example, smart formatting is not applied to response data in the timestamps or alternatives fields.
* Speech hesitations, such as "uhm" and "uh", can adversely impact the conversion of phrases and strings by smart formatting for some languages. Previous-generation models produce hesitation markers to replace such hesitations in a transcript. Smart formatting has the following effect on hesitation markers for previous-generation models:



* For US English, smart formatting suppresses hesitation markers from the transcript field for final results.
* For Japanese, hesitation markers continue to appear in final results.
* For both US English and Japanese, hesitation markers continue to appear in interim results.
* For Spanish, the service does not produce hesitation markers for any results.



Next-generation models do not produce hesitation markers. They instead include the actual hesitations in transcription results. Smart formatting has no effect on hesitations that are included by next-generation models. For more information, see [Speech hesitations and hesitation markers](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-hesitation).







 Language differences 

Smart formatting is based on the presence of obvious keywords in the transcript. Because of differences among the supported languages, smart formatting works slightly differently for each language. The following sections describe the strings and content that trigger smart formatting changes for [US English and Spanish](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting-english-spanish) and for [Japanese](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-formattingsmart-formatting-japanese).



 US English and Spanish 



* Times are identified by keywords such as AM, PM, or EST.
* Military times are converted if they are identified by the keyword hours (US English) or horas (Spanish).
* Phone numbers must be either 911 or a number that contains 10 or 11 digits and starts with the number 1.
* Currency symbols are substituted for the following strings in appropriate contexts:
* I saw the story on yahoo dot com becomes I saw the story on yahoo.com.
* Vi la historia en yahoo punto com becomes Vi la historia en yahoo.com.



The following examples show incomplete conversions:



* I saw the story on w w w dot yahoo dot com becomes I saw the story on w w w .yahoo.com.
* Vi la historia en w w w punto yahoo punto com becomes Vi la historia en w w w .yahoo.com.



* Conversion of large numbers and currency values can be challenging. The service converts digits and many numbers well. But larger and more complex numbers and currency values work best with more precise phrasing. For example, the service correctly converts the following transcripts because of their precise wording:



* sixty nine thousand five hundred sixty dollars and twenty five cents becomes $69560.25,
* sixty nine thousand five hundred sixty dollars point twenty five becomes $69560.25.



But the service cannot correctly convert the following transcripts due to their looser phrasing:



* sixty nine thousand five sixty dollars and twenty five cents becomes 60 9000 $560.25.
* sixty nine thousand five sixty dollars point twenty five becomes 60 9000 $560.25.



To correctly convert a greater possible variety of complex numbers, you need to experiment with the results of smart formatting and customize your own post-processing utilities.
* For US English, certain punctuation symbols are added for special keywords that occur in appropriate places. When you use smart formatting, the service substitutes punctuation symbols for the following keyword strings based on where it finds them in a transcript:



* Comma (,)
* Period (.)
* Question mark (?)
* Exclamation point (!)



The service converts these keyword strings to symbols only in appropriate positions of a transcript. In the following example, the speaker says the word period at the end of the sentence:



* the warranty period is short period becomes the warranty period is short.



The service correctly differentiates between the noun that appears earlier in the sentence and the concluding punctuation.







 Japanese 



* Phone numbers must be 10 or 11 digits and begin with valid prefixes for telephone numbers in Japan.
The Speech to Text service supports grammars that are defined in the following standard formats:



* Augmented Backus-Naur Form (ABNF), which uses a plain-text representation that is similar to traditional BNF grammar. The media type for this format is application/srgs.
* XML Form, which uses XML elements to represent the grammar. The media type for this format is application/srgs+xml.



Both grammar formats have the expressive power of a Context-Free Grammar (CFG). However, the service can decode only Type-3 Regular grammars in the Chomsky hierarchy. Such grammars represent finite state automata.

For general information about grammars, see the following Wikipedia pages:



* [Speech Recognition Grammar Specification](https://wikipedia.org/wiki/Speech_Recognition_Grammar_Specification)
* [Augmented Backus-Naur form](https://wikipedia.org/wiki/Augmented_Backus%E2%80%93Naur_form)
* [Chomsky hierarchy](https://wikipedia.org/wiki/Chomsky_hierarchy)







 The Speech Recognition Grammar Specification 

The Speech to Text service supports grammars as defined by the W3C [Speech Recognition Grammar Specification Version 1.0](https://www.w3.org/TR/speech-grammar/). The specification provides detailed information about the supported formats and about defining a grammar. For information about the supported media types, see [Appendix G. Media Types and File Suffix](https://www.w3.org/TR/speech-grammar/AppG) of the specification.

The service does not currently support all features of the Speech Recognition Grammar Specification. Specifically, the service does not support the features described in the following sections of the specification:



* [Section 1.4 Semantic Interpretation](https://www.w3.org/TR/speech-grammar/S1.4). IBM is working to support this feature in a future release of the service.
* [Section 1.5 Embedded Grammars](https://www.w3.org/TR/speech-grammar/S1.5).