vania-janet/multiturn-rag-retrieval-data huggingface dataset:
q14Sm13Sp14","jq16q19Sp16Sm19","jq16q19Sm16Sp19","jq7q10Sp7Sm10","jq7q10Sm7Sp10","jq12q15Sp12Sm15","jq12q15Sm12Sp15","jq22q25Sp22Sm25","jq22q25Sm22Sp25","jq23q24Sp23Sm24","jq23q24Sm23Sp24","jq8q11Sp8Sm11","jq8q11Sm8Sp11","jq0q1Sp0Sm1","jq0q1Sm0Sp1","jq1q2Sp1Sm2","jq1q2Sm1Sp2","jq19q20Sp19Sm20","jq19q20Sm19Sp20","jq6q7Sp6Sm7","jq6q7Sm6Sp7","jq24q25Sp24Sm25","jq24q25Sm24Sp25","jq18q21Sp18Sm21","jq18q21Sm18Sp21","jq4q7Sp4Sm7","jq4q7Sm4Sp7","jq21q23Sp21Sm23","jq21q23Sm21Sp23","jq3q5Sp3Sm5","jq3q5Sm3Sp5","jq5q8Sp5Sm8","jq5q8Sm5Sp8","jq1q4Sp1Sm4","jq1q4Sm1Sp4","jq2q3Sp2Sm3","jq2q3Sm2Sp3","jq19q22Sp19Sm22",
If you configure your collection using [Smart Document Understanding](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu), the PDF and Word conversion settings listed are not used, so changing these conversion settings are ignored.

The word conversion object defines how to convert PDF documents into HTML and has the following structure:

"word": {
"heading": {
"fonts": [
{
"level": 1,
"min_size": 24,
"bold": true,
"italic": false,
"name": "arial"
},
{
"level": 2,
"min_size": 18,
"max_size": 23,
"bold": true,
"italic": false
}
],
"styles": [
{
"level": 1,
"names":
"pullout heading",
"pulloutheading",
"header"
]
},
{
"level": 2,
"names":
"subtitle"
]
}
]
}
},
Show more

The Microsoft Word conversion object works in a similar way to the PDF conversion object. However, there are two different arrays that can be specified inside the heading object when extracting headings from Microsoft Word documents. You can use either or both heading extraction arrays to extract heading level elements from your Microsoft Word documents.

Each item in the fonts array specifies a heading level using the following parameters using font characteristics:



* "level": INT - required - the HTML h level that text identified with these parameters are converted into.
* "min_size": INT - optional - the smallest font size that is identified as this heading level.
* "max_size": INT - optional - the largest font size that is identified as this heading level.
* "bold": boolean - optional - When true only bold fonts are identified as this heading level.
* For external dictionaries, when modules are being loaded, you must specify a list of URIs to external dictionaries, as required by the modules that are being loaded.
* Lemmatization of dictionaries: The primary difference between the existing dictionary matching semantics and the lemmatized semantics is that the match is performed against the lemmatized form of the document, instead of the original form of the document.

Dictionary entries that belong to a dictionary with lemma match enabled have these prerequisites:



* A dictionary entry can contain one or more tokens, where each entry token is a lemma. To create a dictionary of lemmas, you can use the [GetLemma scalar function.
* The tokens of a dictionary entry should be separated by a white space. If the token consists of white spaces, then the white spaces should be escaped by using the backslash (\) character.



* The following table shows the differences between the create external dictionary statement and the create dictionary statement:





 create external dictionary create dictionary 

 <br><br> * Defines a placeholder for a dictionary whose content is supplied at initialization time.<br><br><br> <br><br> * Requires that the content of the dictionary is available at compile time.<br> * Serialized in the compiled representation (.tam) of a module.<br><br><br> 







 Examples 

Example 1: Creating an external dictionary

The external dictionary, PersonPositiveClues, expects to be populated with values from an external file at load-time. It also expects to be matched against some Western languages as specified by its flags.

module PersonModuleEnglish;

create external dictionary PersonPositiveClues
allow_empty false
with case exact;

export dictionary PersonPositiveClues;

Example 2: Lemmatization

Consider a dictionary that has lemma match enabled and contains two entries: go shop and went shopping. The document contains the text Anna went shopping. The lemmatized form of the input document is Anna go shop. Lemma match returns went shopping as a match for the entry go shop. The original document text is not compared to the dictionary entries, only the lemmatized document text.
Dutch [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2)
For more information, see [Guidelines for Japanese](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-corporaWords-ngwordLanguages-jaJP-ng).

Defect fix: Correct use of display_as field in transcription results
: Defect fix: For language model customization with next-generation models, the value of the display_as field for a custom word now appears in all transcripts. Previously, the value of the word field sometimes appeared in transcription results.

Security vulnerabilities addressed
: The following security vulnerabilities have been fixed:



* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to issues in OpenSSL (CVE-2022-1434, CVE-2022-1343, CVE-2022-1292, CVE-2022-1473)](https://www.ibm.com/support/pages/node/6890819)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to arbitrary command execution in OpenSSL (CVE-2022-2068)](https://www.ibm.com/support/pages/node/6890831)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a denial of service in protobuf (CVE-2022-1941)](https://www.ibm.com/support/pages/node/6890835)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a buffer overflow in GNU glibc (CVE-2021-3999)](https://www.ibm.com/support/pages/node/6890837)
* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to a security bypass in GNU gzip (CVE-2022-1271)](https://www.ibm.com/support/pages/node/6890841)
You can use the <say-as> element to control how strings are pronounced, including whether they are to be spelled out as individual characters with the letters and digits elements.

For German, you can also control the pace at which the service pronounces the characters. For more information, see [Specifying how strings are spelled out](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-paramsparams-spell-out-mode).



 The interpret-as attribute 

Acceptable values for the interpret-as attribute and examples of each value follow. The service supports the following values as arguments to the interpret-as attribute:



* [cardinal](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-cardinal)
* [date](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-date)
* [digits](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-digits)
* [interjection](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-interjection)
* [letters](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-letters)
* [number](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-number)
* [ordinal](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-ordinal)
* [vxml:boolean](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsvxml-boolean)
Russian [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 1](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-1) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2)
For more information about how the service validates input text, see [SSML validation](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssmlssml-errors).





 Examples of input text 

The following examples show how to specify input text with either method of the HTTP interface. They also show how to escape XML control characters. The examples include line breaks for readability. Do not include the line breaks in actual input.



 Example input with a GET request 

The following examples pass URL-encoded input with the text query parameter of the GET /v1/synthesize method:



* Plain text input:

text=This&20is&20the&20first&20sentence&20of&20the&20paragraph.&20Here
&20is&20another&20sentence.&20Finally,&20this&20is&20the&20last&20sentence.
* SSML input:

text=%22%3Cp%3E%3Cs%3EThis%20is%20the%20first%20sentence%20of%20the%20%3C
break%20time=%225s%22/%3E%20paragraph.%3C/s%3E%3Cs%3EHere%20is%20another
%20sentence.%3C/s%3E%3Cs%3EFinally,%20this%20is%20the%20last%20sentence.
%3C/s%3E%3C/p%3E%22







 Example input with a POST request 

The following examples pass input in the body of the POST /v1/synthesize method:



* Plain text input:

{
"text": "This is the first sentence of the paragraph. Here is another
sentence. Finally, this is the last sentence."
}
* SSML input:

{
"text": "<p>
This is the first sentence of the <break time="5s"/>
Previously, it was not possible to create speaker models. For more information about the feature, which is available for U.S. English voices only, see [Understanding Tune by Example](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-tbe-intro).

Defect fix: Specifying large cardinal numbers with the <say-as> element no longer causes errors for English voices
: Defect fix: You can now use the <say-as> element to pronounce large numbers as cardinal numbers. Previously, enclosing a large number in the <say-as> element with the attribute interpret-as="cardinal" could cause speech synthesis to fail for English voices. For example, <say-as interpret-as="cardinal">3,200</say-as> could cause the service to generate an error. For more information, see [cardinal](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-cardinal) in the topic SSML elements.

Defect fix: Homonyms and other words are now pronounced correctly by English voices
: Defect fix: The service now pronounces homonyms and other words correctly based on their context in English text that is to be synthesized. Previously, words such as advocate and wifi could be pronounced incorrectly by English voices.

Security vulnerability addressed
: The following security vulnerability has been fixed:



* [Security Bulletin: IBM Watson Speech Services Cartridge for IBM Cloud Pak for Data is vulnerable to denial of service in Pypa Setuptools (CVE-2022-40897)](https://www.ibm.com/support/pages/node/6958142)







 30 January 2023 (Version 4.6.2) 

Version 4.6.2 is now available
: Text to Speech for IBM Cloud Pak for Data version 4.6.2 is now available. This version supports IBM Cloud Pak for Data version 4.6.x and Red Hat OpenShift versions 4.8 and 4.10.
* Next-generation models do not produce hesitation markers. They instead include the actual hesitations in transcription results. For more information, see [Speech hesitations and hesitation markers](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-hesitation).
* Next-generation models support automatic capitalization only for German models. Previous-generation models support automatic customization only for US English models. For more information, see [Capitalization](https://cloud.ibm.com/docs/speech-to-text?topic=speech-to-text-basic-responseresponse-capitalization).
The Speech Synthesis Markup Language (SSML) is an XML-based markup language that is designed to provide annotations of text for speech synthesis applications such as the Text to Speech service. You can use SSML elements and their attributes to gain greater control over the synthesis and resulting audio output.

For more information about using SSML to annotate input text, see [Understanding SSML](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssml). For an inventory of all supported elements and attributes, see [SSML elements](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elements).





 Escaping XML control characters 

Because you can submit input text that includes XML-based SSML annotations, the service validates all input to ensure that any SSML is correct and well formed. Therefore, you must escape any XML control characters that are present in the input text, regardless of whether the input includes SSML. Use the equivalent escape strings or character encodings from Table 1 instead of the indicated characters.

Scroll for more

Scroll for more



Table 1. Escaping XML control characters

 Character Escape strings Character encoding 

 " <br>(double quotes) " " 
 ' <br>(apostrophe or single quote) ' ' 
 & <br>(ampersand) & & 
 < <br>(left angle bracket) < < 
 > <br>(right angle bracket) > > 
 / <br>(forward slash) None / 



For more information about how the service validates input text, see [SSML validation](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssmlssml-errors).





 Examples of input text 

The following examples show how to specify input text with either method of the HTTP interface. They also show how to escape XML control characters. The examples include line breaks for readability. Do not include the line breaks in actual input.



 Example input with a GET request
A higher rate causes the text to be spoken more quickly; a lower rate causes the text to be spoken more slowly. The parameter changes the per-voice default rate for an entire request. For more information, see [Modifying the speaking rate](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-paramsparams-rate-percentage).

New beta pitch_percentage query parameter for controlling the global speaking pitch
: The service offers a new pitch_percentage query parameter to modify the speaking pitch for a synthesis request. The speaking pitch represents the tone of the speech that the service synthesizes. It represents how high or low the tone of the voice is perceived by the listener. A higher pitch results in speech that is spoken at a higher tone and is perceived as a higher voice; a lower pitch results in speech that is spoken in a lower tone and is perceived as a lower voice. The parameter changes the per-voice default pitch for an entire request. For more information, see [Modifying the speaking pitch](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-synthesis-paramsparams-pitch-percentage).

Defect fix: Custom word translations now accept commas in all cases
: Defect fix: Word translations added to custom models now accept commas in all cases. Previously, a comma in a translation could occasionally cause the translation to fail to generate valid audio when used for speech syntheses. This problem was identified in US English custom models.

Defect fix: French synthesis of dates is now consistent
: Defect fix: French synthesis no longer includes the article "le" before dates of the form "the ordinal of month." Previously, the article was included only for the first day of the month for French (for example, "the first of September," "le premier septembre").

Defect fix: Japanese synthesis is improved to handle long strings of input text
: Defect fix: The service now correctly synthesizes Japanese requests that include long strings of characters. Previously, the service failed to properly synthesize very long strings of Japanese text.

Defect fix: Add rules for custom model naming documentation
* To select all text that includes number plates for private and commercial vehicles in India formatted as AA-99-AA-9999, AA-99-AAA-9999 and, for states where the 0 is optional, AA-9-AA-9999 with separators that may be hyphens, spaces or not present (for example, TN-86-AF-1199, WB 06 F 5971, and DL4CAF4943), specify:

[A-Z]{2}(-| |)d{1,2}(-| |)[A-Z]{1,3}(-| |)d{4} (M1 1AE, B33 8TH, DN55 1PT, W1A 0AX, CR2 6XH, SW1A 2AA)

{ codeblock}









 Creating a sequence pattern 

Often, you must assess the text for patterns that provide context for the terms of interest. For example, you might want to distinguish references to military or law enforcement personnel from references to civilians. When designing an extractor, several methods are available to identify and extract information on the basis of patterns in text. These methods can be used in a single extractor, or you can combine extractors.

Before defining your own pattern to meet your needs, check to see if a provided extractor can give you the desired results.

To create a sequence pattern:



1. Create individual extractors for all needed terms by extending provided extractors, or creating [dictionaries](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractorscreating-a-dictionary), [regular expressions](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractorscreating-a-regular-expression), [literals](https://cloud.ibm.com/docs/watson-knowledge-studio-data?topic=watson-knowledge-studio-data-managing-projects-and-extractorscreating-a-literal).
2.
I live at 21a Front Street, Durham, UK - my email is chris7767@aol.com."



 Standard analyzer 

The Standard analyzer changes the string in the following ways:



* Removes punctuation.
* Splits words based on spaces and punctuation.
* Removes stop words, including "is" and "at".
* Changes words to use lowercase letters.
* Note how "aol.com" stays intact.



{"tokens":["my", "name", "chris", "wright", "smith", "i", "live", "21a", "front", "street", "durham", "uk", "my", "email", "chris7767", "aol.com"]}





 Keyword analyzer 

With the Keyword analyzer, the string stays intact. See the following example:

{"tokens":["My name is Chris Wright-Smith. I live at 21a Front Street, Durham, UK - my email is chris7767@aol.com."]}





 Simple analyzer 

The Simple analyzer changes the string in the following ways:



* Removes punctuation.
* Splits words based on spaces and punctuation.
* No stop words removed (notice "is" and "at").
* Changes words to use lowercase letters.
* Note how chris7767 changes to chris and 21a changes to a.



{"tokens":["my", "name", "is", "chris", "wright", "smith", "i", "live", "at", "a", "front", "street", "durham", "uk", "my", "email", "is", "chris", "aol","com"]}





 White space analyzer 

The White space analyzer changes the string in the following ways:



* Removes some punctuation.
* Splits words on spaces.
* No stop words removed (notice "is" and "at").
await writer.write(encoder.encode("[n"))

for (let i = 0; i < bodies.length; ++i) {
if (i > 0) {
await writer.write(encoder.encode(",n"))
}
writer.releaseLock()
await bodiesi].pipeTo(writable, { preventClose: true })
writer = writable.getWriter()
}

await writer.write(encoder.encode("]"))

await writer.close()
}
Show more

Again, there are a couple of important details to note:



* The runtime expects to receive TypedArrays on the readable side of the TransformStream. Therefore, we never pass a string to writer.write(), only Uint8Arrays. If you need to write a string, use a TextEncoder.









 Custom load balancer with Edge functions 

Load balancing helps you maintain the scalability and reliability of websites you host. You can use Edge functions to create custom load balancers designed to address your specific needs.

const US_HOSTS = [
"0.us.example.com",
"1.us.example.com",
"2.us.example.com"
];

const IN_HOSTS = [
"0.in.example.com",
"1.in.example.com",
"2.in.example.com"
];

var COUNTRIES_MAP = {
IN: IN_HOSTS,
PK: IN_HOSTS,
BD: IN_HOSTS,
SL: IN_HOSTS,
NL: IN_HOSTS
}
addEventListener('fetch', event => {
var url = new URL(event.request.url);

var countryCode = event.request.headers.get('CF-IPCountry');
var hostnames = US_HOSTS;
if (COUNTRIES_MAP[countryCode]) {
]
},
{
"start_time": 0.63,
"end_time": 0.77,
"alternatives":
{
"word": "that",
"confidence": 0.72
},
{
"word": "the",
"confidence": 0.27
}
]
},
{
"start_time": 0.77,
"end_time": 1.2,
"alternatives":
{
"word": "tuna",
"confidence": 0.94
}
]
}
]
}
]
}
Show more
"several",
1.01,
1.52
],

"tornadoes",
1.52,
2.15
],

"touch",
2.15,
2.5
],

"down",
2.5,
2.81
],
. . .

"on",
5.62,
5.74
],

"Sunday",
5.74,
6.34
]
],
"confidence": 0.96,
"transcript": "several tornadoes touch down as a line of
severe thunderstorms swept through Colorado on Sunday "
}
],
"final": true
}
]
}
Show more
: Specify the REGION where the metadata associated with route and target definitions is stored.

backup-metadata-region
: Specify the REGION where the metadata associated with route and target definitions is stored as a backup.

private-api-endpoint-only
: Specifies whether nor not a private endpoint can be used. If TRUE only a private endpoint can be used.

default-targets
: Is a list of target IDs. If no routing rules cause metrics to be sent to other targets, these targets will received the metrics. TARGETS is a comma-separated list of target IDs.

permitted-target-regions
: Is the list of regions that can be used to define a target. REGIONS is a comma-separated list of regions.

--output FORMAT
: If JSON specified, output will be returned in JSON format. If JSON is not specified, output will be returned in a tabular format.

help | --help | -h
: List options available for the command.

If the update is successful, the current settings will be displayed.







 API prerequsites 

To make API calls to manage settings, complete the following steps:



1. Get an IAM access token. For more information, see [Retrieving IAM access tokens](https://cloud.ibm.com/docs/metrics_router?topic=metrics_router-retrieve-iam-token).
2. Identify the API endpoint in the region where you plan to configure or manage settings. For more information, see [Endpoints](https://cloud.ibm.com/docs/metrics_router?topic=metrics_router-endpointsendpoints_api).







 Getting settings using the API 

You can use the following cURL command to get existing settings information:

curl -X GET ENDPOINT/api/v3/settings -H "Authorization: $ACCESS_TOKEN"

Where:

ENDPOINT
: Is the API endpoint in the region where you plan to configure or manage a target.