vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Epiphany observed (Many regions) epiphanyobserved 
 Valentine's Day valentinesday 
 Andalusia Day andalusiaday 
 Baleares Day balearesday 
 Ash Wednesday ashwednesday 
 San Jose (Many regions) sanjose 
 Palm Sunday palmsunday 
 Maundy Thursday (Many regions) maundythursday 
 Good Friday goodfriday 
 Easter Sunday eastersunday 
 Easter Monday (Many regions) eastermonday 
 Aragon Day aragonday 
 Saint George Day stgeorgeday 
 Labor Day / May Day laborday 
 Community Day communityday 
 San Isidro saintisidro 
 Mothers' Day mothersday 
 Whit Sunday/Pentecost whitsunday 
 Feast of Saint James the Apostle (GA, PV, S) feastofsaintjamestheapostle 
 Assumption of Mary assumptionofmary 
 Hispanic Day hispanicday 
 All Saints' Day (All) allsaintsday 
 Constitution Day constitutionday 
 Immaculate Conception immaculateconception 
 Immaculate Conception observed (Many regions) immaculateconceptionobserved 
 Christmas Eve christmaseve 
 Christmas Day christmasday 
 Feast of the Holy Family feastoftheholyfamily 
 Corpus Christi corpuschristi 
 All Saints' Day' observed (Most regions) allsaintsdayobserved 
 Assumption observed assumptionobserved 
 Cantabria Day cantabriaday 
 Christmas Day observed (Balearic Islands) christmasdayobserved 
 Labor Day observed (Many regions) labordayobserved 
 Christmas Day observed (Many regions) christmasdayobserved 
 Immaculate Conception observed immaculateconceptionobserved 
 Hispanic Day observed (Many regions) hispanicdayobserved 
 Almuneda Day almunedaday 
 New Year's Eve newyearseve 







 Holidays (ar-ar) 



Arabic holidays

 Holiday name String to use to check for a festival match 

 new year newyear 
 Valentine valentine 
 easter sunday easter 
 christmas eve christmaseve 
 christmas christmas 
 Eid al-Fitr eidalfitr 
 Eid al-Adha eidaladha 
 Ramadan ramadan 
 Islamic New Year islamicnewyear 
 Ashura ashura
R ɜ<br><br>ɜː<br><br>ɚ<br><br>ɝ 025C<br><br>025C+02D0<br><br>025A<br><br>025D butter, hurt 







 Diphthongs 



Table 2. Diphthongs (AU English)

 SPR symbol IPA symbol IPA Unicode Example words 

 O ɔɪ 0254+026A toil, boy 
 W aʊ 0061+028A out, cow 
 Y aɪ 0061+026A life, fine 
 Ex eɪ<br><br>æɪ 0065+026A<br><br>00E6+026A square, bared 
 Ix ɪə 026A+0259 near, here 
 Ux ʊə 028A+0259 tour, lured 







 Reduced vowels 



Table 3. Reduced vowels (AU English)

 SPR symbol IPA symbol IPA Unicode Example words 

 x ə 0259 sofa, alone, suppose, America 
 X i<br><br>ɨ 0069<br><br>0268 roses, hinted 







 Consonants 



Table 4. Consonants (AU English)

 SPR symbol IPA symbol IPA Unicode Example words 

 b b 0062 bad, sob 
 C tʃ<br><br>ʧ 0074+0283<br><br>02A7 chip, witch, nature 
 d d 0064 dip, had 
 D ð 00F0 this, breathe 
 f f 0066 field, if, graph 
 g g<br><br>ɡ 0067<br><br>0261 good, bug 
 G ŋ 014B sing, finger 
 h h<br><br>x<br><br>χ<br><br>ɦ 0068<br><br>0078<br><br>03C7<br><br>0266 hot, hero 
 J dʒ<br><br>ʤ 0064+0292<br><br>02A4 Jane, huge 
 k k 006B kill, make, back
"bold": true,
"italic": false,
"name": "ariel"
}
]
}
},
Show more

When converting PDF files, headings in those files can be identified and converted into an appropriate HTML "h" tag by identifying the size, font, and style of each heading level. Heading levels can be specified multiple times, if necessary, to correctly identify all relevant sections. HTML heading levels are important to identify, if you plan on extracting the contents, using CSS selectors, or if you intend to split the document, using document splitting. The heading object contains the fonts array and each item in that array specifies a heading level, using the following parameters:



* "level": INT - required - the HTML h level that text identified with these parameters are converted into.
* "min_size": INT - optional - the smallest font size that is identified as this heading level.
* "max_size": INT - optional - the largest font size that is identified as this heading level.
* "bold": boolean - optional - When true only bold fonts are identified as this heading level.
* "italic": boolean - optional - When true only italic fonts are identified as this heading level.
* "name": "string" - optional - The name of the font that is identified as this heading level.



For an area of text to be identified as a heading it must match all of the parameters defined in the specific array item. If the parameters defined are too flexible, you might have more headings identified than you anticipated. Your results are better if you strictly define each heading level multiple times to ensure all heading level variations are included without invalid matches.





 Word 

If you configure your collection using [Smart Document Understanding](https://cloud.ibm.com/docs/discovery?topic=discovery-sdu), the PDF and Word conversion settings listed are not used, so changing these conversion settings are ignored.

The word conversion object defines how to convert PDF documents into HTML and has the following structure:

"word": {
"heading": {
"fonts": [
{
"level": 1,
Like audio/basic and audio/mulaw, this format provides single-channel audio that is encoded by using 8-bit u-law (or mu-law) data that is sampled at 8 kHz. For more information, see [Using audio formats](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-audio-formats).





 19 May 2022 

Defect fix: Multiple consecutive SSML <phoneme> tags are now parsed correctly
: Defect fix: The service now correctly synthesizes text that contains consecutive <phoneme> tags. Previously, if the text contained two or more consecutive <phoneme> tags, the service synthesized only the first tag, ignoring the others.





 31 March 2022 

Important: Deprecation of all neural voices
: Important: Effective 31 March 2022, all neural voices are deprecated. The deprecated voices remain available to existing users until 31 March 2023, when they will be removed from the service and the documentation. No enhanced neural voices or expressive voices are deprecated.

The following neural voices are now deprecated:



* Arabic: ar-MS_OmarVoice
* Chinese (Mandarin): zh-CN_LiNaVoice, zh-CN_WangWeiVoice, and zh-CN_ZhangJingVoice
* Czech: cs-CZ_AlenaVoice
* Dutch (Belgian): nl-BE_AdeleVoice and nl-BE_BramVoice
* Dutch (Netherlands): nl-NL_EmmaVoice and nl-NL_LiamVoice
* English (Australian): en-AU_CraigVoice, en-AU_MadisonVoice, and en-AU_SteveVoice
* Korean: ko-KR_HyunjunVoice, ko-KR_SiWooVoice, ko-KR_YoungmiVoice, and ko-KR_YunaVoice
* Swedish: sv-SE_IngridVoice



The deprecated neural voices continue to be available to existing users but are no longer available to new users:



* Existing users: Instances of the service created before 31 March 2022 can continue to use the deprecated voices for speech synthesis.
French liaison in invalid location In the ph attribute of a <phoneme> element, the liaison character does not follow a consonant or it occurs in the middle of a word's pronunciation. 
 Japanese : symbol does not precede a vowel In the ph attribute of a <phoneme> element in Japanese, a : character does not occur before a vowel (possibly with other symbols, such as syllable boundary, in between). 
 Invalid use of the SSML <prosody> element You cannot use the contour, duration, range, and volume attributes of the <prosody> element with any voice. 
 Invalid use of the SSML <express-as> element You can use the <express-as> element only with expressive neural voices.
Hindi [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2)
r r<br><br>ɹ<br><br>ʀ<br><br>ʁ 0072<br><br>0279<br><br>0280<br><br>0281 parer, rare, carreau 
 s s 0073 sans, ambition, façon 
 S ʃ 0283 cheval, lâche, schéma 
 t t 0074 ton, patte, théâtre 
 T ts<br><br>ʦ 0074+0073<br><br>02A6 petit, tuque 
 v v 0076 laver, wagon, visiter 
 w w 0077 oui, watt 
 z z 007A jaser, réseau, zigzaguer 
 Z ʒ 0292 rage, gîte, jouer 







 Liaison 

In Canadian French, the _ (underscore) can be used immediately following the final consonant of a word (but within the double-quotes that enclose the SPR) to indicate that it is a liaison consonant. A liaison consonant is pronounced only if the word that follows begins with a vowel.



Table 3. Liaison (Canadian French)

 SPR symbol IPA symbol IPA Unicode 

 _ ‿ 203F 



Examples of words with and without the liaison symbol follow:



* "p0'oe't1it_": The t is pronounced only if the following word begins with a vowel.
* "nEt": The t is always pronounced.



A dictionary entry petit with the translation value "p'oe't1it_" has the final t pronounced in the input string un petit ami but not in the input string un petit chien. An entry with the translation value "nEt" has the final t pronounced regardless of context.
* When multiple IPA symbols (or symbol combinations) are documented for an SPR symbol, all of the IPA symbols are equivalent to the single SPR symbol. The service treats all of these IPA symbols the same and does not realize the subtle or regional differences that IPA is meant to describe.



For more information, see



* [The phoneme element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsphoneme_element)
* [Rules for creating custom entries](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-rules)
* [Creating and managing custom entries](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customWords)







 Working with IBM SPR 

IBM SPR is an alternative representation to standard IPA. The following examples of valid SPR notations define the words through and shocking in US English:

<phoneme alphabet="ibm" ph=".1Tru">through</phoneme>
<phoneme alphabet="ibm" ph=".1Sa.0kIG">shocking</phoneme>

In the definitions, the letters represent specific sounds of US English speech. A . signals the beginning of a new syllable, and the digits 1 and 0 indicate syllable stress. For more information, see [Specifying syllables](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-symbolssyllables).



 Speech sound symbols 

Each language uses its own inventory of SPR symbols to represent the speech sounds of that language. The following rules apply to specifying an SPR symbol:



* Letters are case-sensitive, so e and E, for example, represent two different sounds.
* Two- and three-character symbols must be enclosed in single quotes when indicated in the symbol tables. The single quotes indicate that the multiple characters are actually a single symbol. For example, the symbol 'aj' in the German word heim is specified as "h'aj'm".
* By specifying the appropriate file extension for the format. For example, to translate an HTML file named example.html from English to French, you would use the following command:

curl -X POST --user "apikey:{apikey}" --form "file=@example.html" --form "source=en" --form "target=fr" "{url}/v3/documents?version=2018-05-01"
* By specifying the content type (MIME type) of the format as the type of the file parameter. For example, to translate an HTML file named just example from English to French, you would use the following request:

curl -X POST --user "apikey:{apikey}" --form "file=@example;type=text/html" --form "source=en" --form "target=fr" "{url}/v3/documents?version=2018-05-01"



The tables in the following sections list the valid file extensions and content types for each supported format. In most cases, specifying the correct file extension is preferred because it can eliminate ambiguity and is simpler. For subtitles, the table makes clear where the file extension or the content type is needed.

All file formats other than Adobe® PDF (Portable Document Format) are generally available. PDF is currently experimental functionality.



 Microsoft Office file formats 

Table 1 lists the Microsoft Office file formats that the service supports for translation.



Table 1. Microsoft Office file formats

 File format File extension Content type 

 Microsoft Excel .xls application/vnd.ms-excel 
 .xlsx application/vnd.openxmlformats-officedocument.spreadsheetml.sheet 
 Microsoft PowerPoint .ppt application/vnd.ms-powerpoint 
 .pptx application/vnd.openxmlformats-officedocument.presentationml.presentation 
 Microsoft Word .doc application/msword 
 .docx application/vnd.openxmlformats-officedocument.wordprocessingml.document 



Microsoft, Microsoft Excel, Microsoft Office, Microsoft PowerPoint, and Microsoft Word are trademarks of the Microsoft group of companies.
Currency code Example of recognized values (case insensitive) 

 USD $ 
 GBP £ 
 INR ₹ 
 EUR € 
 JPY ¥ 
 MXN mxn 
 ILS שקל 







 Currencies (zh-cn) 



Simplified Chinese currencies

 Currency code Example of recognized values (case insensitive) 

 USD $ 
 GBP £ 
 INR ₹ 
 EUR € 
 CNY ¥ 







 Currencies (zh-tw) 



Traditional Chinese currencies

 Currency code Example of recognized values (case insensitive) 

 USD $,usd,美金 
 GBP £ 
 INR ₹ 
 EUR €,歐元 
 CNY ¥,人民幣 
 TWD nt$,twd,台幣 
 HKD 港幣＄ 







 Currencies (ja-jp) 



Japanese currencies

 Currency code Example of recognized values (case insensitive) 

 USD $,usd,米ドル,usドル 
 GBP £,gbp 
 INR ₹,inr 
 EUR €,eur 
 JPY ¥,jpy 
 MXN mxn 







 Currencies (ko-kr) 



Korean currencies

 Currency code Example of recognized values (case insensitive) 

 USD 
 GBP £ 
 INR ₹ 
 EUR € 
 JPY ¥ 
 KRW ￦ 







 Currencies (universal model) 



Universal model currencies

 Currency code Example of recognized values (case insensitive) 

 USD '$','usd' 
 CAD '$','c$','ca$','can$','cad' 
 AUD 'a$','au$','aud' 
 CHF 'chf' 
 GBP '£','gbp','quid' 
 INR '₹','inr','rs','rs.' 
 EUR '€','eur' 
 JPY '¥','jpy' 
 MXN 'mxn' 
 RUB '₽','rub'
Romanian [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2) [Version 2](https://cloud.ibm.com/docs/natural-language-understanding/?topic=natural-language-understanding-entity-types-version-2)
Change the srclang attribute of the <header> tag to the [language code](http://www-01.sil.org/iso639-3/codes.asp) of the source language that your glossary or parallel corpus uses.
3. Change the xml:lang attribute of the <tuv> tags to the correct [language code](http://www-01.sil.org/iso639-3/codes.asp) for each term or translation.
4. Save the document with the extension .tmx and UTF-8 encoding.







 Changing a TMX file to UTF-8 encoding 

You can use a number of tools to create or generate TMX files. Often, generated TMX files are UTF-16 encoded by default. The Language Translator service accepts only UTF-8 encoded TMX files. To change the encoding of a TMX file, complete the following steps:



1. Open the TMX file in a text editor.
2. Change the XML header (if present) from <?xml ... encoding="utf-16"?> to <?xml ... encoding="utf-8"?>.
3. Save the file with a new name and with UTF-8 encoding output.



Mac users can also change file encoding by updating the XML header of the document as described in step 2, and then running the iconv command as shown in the following example:

iconv -f utf-16 -t utf-8 <utf-16_file_name.tmx> <new_utf-8_file_name.tmx>







 XLIFF 

The service supports [XLIFF version 1.2](http://docs.oasis-open.org/xliff/v1.2/os/xliff-core.html). Submit files to the service either with the file extension .xliff or a content-type of application/xliff+xml. The service interprets only the part of XLIFF that can be used to encode aligned text segments. You must ensure that your data has a structure like the following example:
For an example translation that uses IPA Unicode values, see [The phoneme element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsphoneme_element).



 Working with an existing phonetic translation 

Unless you are an expert in phonology, composing phonetic translations is not an easy task. It is always easier to edit an existing phonetic translation than to compose one from scratch. To help you create phonetic translations, the service's API includes a GET /v1/pronunciation method. The method returns the IPA or SPR representation that is generated by the service's regular pronunciation rules for a word in a specified language. You can also request the pronunciation for a word from a specified custom model to see the translation in the language of that model.

You can use the /GET v/1/pronunciation method to obtain an initial phonetic translation for a word. You can then modify the translation to achieve the pronunciation that you want. As with the sounds-like method, you follow a trial-and-error process. You submit your candidate translation to the service, synthesize the word as input text, listen to the resulting audio, and edit the candidate translation. You can repeat the process until you are satisfied with the pronunciation.

For more information, see [Querying a word from a language](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customWordscuWordsQueryLanguage).





 More information about phonetic translation 

The following resources provide information about phonetic translation:



* For more information about using SSML and its <phoneme> element, see [Understanding SSML](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssml).
* For more information about specifying SPR and IPA symbols and translations, see [Understanding phonetic symbols](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-symbols).









 Mixed sounds-like and phonetic translation 

You can mix the sounds-like and phonetic methods in the same translation.
curl -X POST --user "apikey:{apikey}" --header "Content-Type: application/json" --data "{"text":"Hello","model_id":"96221b69-8e46-42e4-a3c1-808e17c787ca"}" "{url}/v3/translate?version=2018-05-01"

The /v3/translate method accepts a maximum of 50 KB of input text encoded in UTF-8 format for translation.







 Forced glossary customization 

Use a forced glossary to set mandatory translations for specific terms and phrases. If you want specific control over translation behavior, use a forced glossary.



* Encoding: UTF-8.
* Maximum file size: 10 MB.
* You can apply a forced glossary to a base model or to a model that has been customized with parallel corpora.
* You are limited to one forced glossary per custom model.
* You can submit a single forced glossary by using the forced_glossary multipart form parameter.



Forced glossary translations are case-sensitive. Make sure that your training data reflects the capitalization of the content that your application will encounter.



 Forced glossary example 

The following example shows a TMX file with two translation pairs. The first pair forces "international business machines" to be preserved in English during translation. This type of forced translation might be useful if you want to preserve improperly capitalized company names in informal communications. The second pair forces the model to always use "brevet" when translating "patent".

<?xml version="1.0" encoding="UTF-8"?>
<tmx version="1.4">
<header creationtool="" creationtoolversion=""
segtype="phrase" o-tmf="" adminlang="en"
srclang="en" datatype="PlainText" o-encoding="UTF-8" />
<body>
<tu>
<tuv xml:lang="en">
r r<br><br>ɹ<br><br>ʀ<br><br>ʁ<br><br>x<br><br>χ 0072<br><br>0279<br><br>0280<br><br>0281<br><br>0078<br><br>03C7 parer, rare, carreau 
 s s 0073 sans, ambition, façon 
 S ʃ 0283 cheval, lâche, schéma 
 t t 0074 ton, patte, théâtre 
 v v 0076 laver, wagon, visiter 
 w w 0077 oui, bouée, watt 
 z z 007A jaser, réseau, zigzaguer 
 Z ʒ 0292 rage, gîte, jouer 







 Liaison 

In French, the _ (underscore) can be used immediately following the final consonant of a word (but within the double-quotes that enclose the SPR) to indicate that it is a liaison consonant. A liaison consonant is pronounced only if the word that follows begins with a vowel.



Table 3. Liaison (French)

 SPR symbol IPA symbol IPA Unicode 

 _ ‿ 203F 



Examples of words with and without the liaison symbol follow:



* "p0'oe't1it_": The t is pronounced only if the following word begins with a vowel.
* "nEt": The t is always pronounced.



A dictionary entry petit with the translation value "p'oe't1it_" has the final t pronounced in the input string un petit ami but not in the input string un petit chien. An entry with the translation value "nEt" has the final t pronounced regardless of context.
Usage notes 



* In general, AQL supports the same features as the Java 5 regular expression implementation, as described in [Class Pattern: java.util.regex. The runtime component contains several regular expression engine implementations, including the built-in implementation of Java. During compilation, the Optimizer examines each regular expression and chooses the fastest engine that can run the expression.
* The alternative execution engines might have slightly different semantics for certain corner cases. In particular, AQL does not guarantee the order in which alternatives are evaluated.

For example, suppose that an extract statement matches the regular expression /fish|fisherman/, over the text 'fisherman'. The statement might match either 'fish' or 'fisherman', depending on which regular expression engine is used internally.





 AQL flag string Java flag Description 

 CANON_EQ CANON_EQ Canonical equivalence: Different Unicode encodings of the same character are considered equivalent. 
 CASE_INSENSITIVE CASE_INSENSITIVE Perform matching that is not case-sensitive.By default, case-insensitive matching assumes that only characters in the US-ASCII character set are matched. Unicode-aware case-insensitive matching can be enabled by specifying the UNICODE flag with this flag. 
 UNICODE UNICODE_CASE If case-insensitive matching is specified, use Unicode case folding to determine whether two characters are equivalent in a case-insensitive comparison in a manner that is consistent with the Unicode Standard. By default, case-insensitive matching assumes that only characters in the US-ASCII character set are matched.Note: The behavior of this flag is not defined when it is used without the CASE_INSENSITIVE flag. 
 DOTALL DOTALL Make the dot character . match all characters, including newlines. 
 LITERAL LITERAL Treat the expression as a sequence of literal characters, ignoring the normal regular expression escape sequences. 
 MULTILINE MULTILINE Make the characters ^ and $ match the beginning and end of any line, as opposed to the beginning and end of the entire input text. 
 UNIX_LINES UNIX_LINES Treat only the UNIX™ newline character as a line break, ignoring the carriage return character r. 





* Follow these guidelines to make your extractors run faster and be easier to maintain: