vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Each returned match must start at the beginning of a token and end at the end of a token. If multiple overlapping matches exist, the extract statement returns all of them.

The locations of token boundaries depend on what tokenizer the runtime component is using to tokenize the document. If the engine is using the Standard tokenizer, then a token is defined as a sequence of word characters or a single punctuation character.

For example, consider the string:

"The fish are pretty," said the boy.

The token boundaries are identified at these locations:

["] [fish] [are] [pretty]["] [said] [the] [boy]
* <name>.<column>

The view name and column name on which the regular expression is applied.
* <grouping spec>

Determines how to handle the capturing of groups in the regular expression. Capturing groups are regions of the regular expression match that are identified by parentheses in the original expression. For example, the expression (fish)(cakes) has three capturing groups:



* Group 0 is the entire match fishcakes.
* Group 1 is fish.
* Group 2 is cakes. When you specify group IDs in the return clause of the syntax, each group ID must correspond to a valid group within each regular expression specified as part of the extract regex clause in the syntax.



This is the format of the grouping specification:

return
group <number> as <name>
[and group <number> as <name>]

To return only Group 0 (the entire match), you can use a shorter, alternative format as in Example 1. This format is equivalent to return group 0 as <name>. <name> can be a simple identifier or a double-quoted identifier.







 Usage notes 



* In general, AQL supports the same features as the Java 5 regular expression implementation, as described in [Class Pattern: java.util.regex. The runtime component contains several regular expression engine implementations, including the built-in implementation of Java. During compilation, the Optimizer examines each regular expression and chooses the fastest engine that can run the expression.
* The alternative execution engines might have slightly different semantics for certain corner cases.
Tagging objects in IBM Cloud Object Storage 

Your data can be expressly defined, categorized, and classified in IBM Cloud® Object Storage using associated metadata, called "tags." This document will show you how to take full control in "tagging" the objects representing your data.



 Objects and metadata 

Organizing your data can be a complex task. Basic methods, such as using key prefixes like organizational "folders" are a great start to hierarchical structures. But for more complex organization, you will need custom "

tags." Your metadata can describe the relationships inherent to your data, and provide more organization than titles or folders. Unlike mere labels, there are two parts to a tag: a key and a value, defined individually according to your needs.



 Tagging Objects 

Managing tags describing your objects can be performed through various interfaces and architectures. Using the [Console](https://cloud.ibm.com) provides a graphical user interface. Using the command line requires tools like [curl](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-curl) and the knowledge of how it interacts with Object Storage.





 Before you begin 

You need:



* An [IBM Cloud® Platform account](https://cloud.ibm.com/login)
* An [instance of IBM Cloud Object Storage](https://cloud.ibm.com/docs/cloud-object-storage/basics?topic=cloud-object-storage-provision) and a bucket created for this purpose
* An [IAM API key](https://cloud.ibm.com/docs/cloud-object-storage/iam?topic=cloud-object-storage-iam-overview) with Writer access to your Object Storage bucket or instance
* Either existing or new objects that will have tags applied to them.







 Reading tags 

Tags are accessible throughout an instance with the proper permissions. While the true organizational power of using tags as an organizational principle scales with you, you can access tags on an individual basis as well.
For example, the following input includes a single <sub> tag:

I work at <sub alias="International Business Machines">IBM</sub>.

The service produces timing information for the following strings: "I, work, at, IBM.. The service normalizes the string IBM as International Business Machines. The timing information for the string reflects the start time of International and the end time of Machines".
* <break> inserts a pause in the spoken text. The service reflects the resulting silence in the word timings as a gap between the end time of the word that precedes the <break> element and the start time of the word that follows the element.
* <paragraph> (or <p>) can add silence to the audio. The service does not return timing information for the silence.
* <sentence> (or 
) can add silence to the audio. The service does not return timing information for the silence.



SSML elements that are not mentioned in the list do not impact word timing information. For more information about the service's support for SSML, see [Understanding SSML](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-ssml).







 Examples with mark elements 

The following examples show a simple WebSocket session between a client and the service. The examples focus on the exchange of data, not on opening the connection. The client sends a text message that includes two <mark> elements, named SIMPLE and EXAMPLE, and it requests the audio to be returned in WAV format:

{
"text": "This is a <mark name="SIMPLE"/>simple <mark name="EXAMPLE"/> example.",
"accept": "audio/wav"
}

The service first sends a message to confirm the audio format. It then sends multiple messages with the results. The service cannot guarantee the number of audio chunks that it sends to the client or the order in which the text and audio messages are delivered.

Both of the following responses are possible. In each case, the service sends two text messages that identify the locations of the marks in the binary stream.
"keywords": "keyword"],
"short_description": "SHORT DESCRIPTION",
"long_description": "LONG DESCRIPTION",
"provider_name": "PROVIDER NAME",
"offering_docs_url": "DOCS URL",
"offering_icon_url": "ICON URL",
"support_details": "SUPPORT DETAILS",
"features":
{
"title": "TITLE",
"description": "DESCRIPTION"
}
],
"module_info":
{
"works_with":
{
"catalog_id": "CATALOG ID",
"name": "PRODUCT PROGRAMMTIC NAME",
"kind": "VERSION'S FORMAT KIND",
"version": "VERSION NUMBER",
"flavors": "FLAVOR"]
}
]
}
],
"flavors":
{
"label": "LABEL",
"name": "VARIATION'S PROGRAMMATIC NAME",
"working_directory": "PATH",
"usage": "USAGE",
"license": {
"name": "LICENSE NAME",
"url": "LICENSE URL"
},
"compliance": {
"authority": "scc-v3",
"profiles":
{
"profile_name": "PROFILE NAME",
"profile_version": "PROFILE VERSION"
}
],
"controls":
{
"profile": {
"name": "PROFILE NAME",
"version": "PROFILE VERSION"
},
"names":
"CONTROL NAME",
"CONTROL NAME"
]
}
]
},
"iam_permissions":
{
"service_name": "SERVICE PROGRAMMATIC NAME",
"role_crns":
"ROLE",
"ROLE"
],
"cell_id": string,
"location": {
"begin": int,
"end": int
},
"text": string
},
...
]
},
...
]
},
...
],
"document_structure": {
"section_titles": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"level": int,
"element_locations":
{
"begin": int,
"end": int
},
...
]
},
...
],
"leading_sentences": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"element_locations":
{
"begin": int,
"end": int
},
...
],
"paragraphs":
{
"location": {
"begin": int,
"end": int
}
},
...
]
},
...
]
},
"parties": [
{
"party": string,
"role": string,
"importance": string,
"addresses":
{
"text": string,
"location": {
"begin": int,
"end": int
}
},
...
],
"contacts":
{
"name": string,
"role": string
},
...
],
"mentions":
{
"text": string,
"location": {
"begin": int,
"end": int
}
[<lexicon> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementslexicon_element) None <br><br> * [rate attribute](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsprosody-rate)<br><br><br> Full 
 [<mark> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsmark_element) Partial <br><br> * volume attribute<br><br><br> None 
 [<meta> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsmm_element) None [<say-as> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as_element) Partial 
 [<metadata> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsmm_element) None <br><br> * [interpret-as attribute](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementssay-as-interpret-as)<br><br><br> Partial 
 [<paragraph> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsps_element) Full [<sentence> element](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-elementsps_element) Full
After the text is labeled, the user clicks the Submit page button.

Your browser does not support the video tag.





 Labeling the documents 

Before you begin, get a feel for the structure of the document you plan to annotate. Are there subtitled sections that you want Discovery to return per answer? If so, identify all subtitles. Later you can split the document into discrete subdocuments, each starting with a subtitle. For more information, see [When to use Smart Document Understanding](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-when).

To label documents, complete the following steps:



1. Review the document preview.

A view of the original document is displayed along with a representation of the document, where the text is replaced by blocks.

The blocks are all the color of the text field label because all of the current text is considered to be standard text and will be indexed in the text field.

Label blocks that represent specific types of information, such as titles or page footers, with other field labels. For example, when you apply the title field label to a document title that would otherwise be indexed as text, you are defining a more precise representation of the document content.

The process of using labels to identify different parts of the document's structure is called annotating the document.
2. Review the field labels that you can use to annotate the document. They are displayed in the Field labels panel.

See the [Default field labels](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-configuring-fieldssdu-default-fields) table for a list of the fields and their descriptions.
3. To create a custom field label, click Create new.



* Specify a field label with no spaces. For example, complex_task is a valid field label.

Avoid using a field label name or including characters, such as a number sign (#) or period (.), in the name that have special meaning for Discovery.
By default, the search term is placed in <em> tags to highlight it, but the highlight can be overridden by using the highlights_pre_tag and highlights_post_tag parameters.

The length of the fragments is 100 characters by default. A different length can be requested with the highlights_size parameter.

The highlights_number parameter controls the number of fragments that are returned, and defaults to 1.

In the response, a highlights field is added, with one subfield per field name.

For each field, you receive an array of fragments with the search term highlighted.

For highlighting to work, store the field in the index by using the store: true option.

See the following example that uses HTTP to search with highlighting enabled:

GET /movies/_design/searches/_search/movies?q=movie_name:Azazel&highlight_fields=["movie_name"]&highlight_pre_tag=" "&highlight_post_tag=" "&highlights_size=30&highlights_number=2 HTTP/1.1
HOST: $ACCOUNT.cloudant.com
Authorization: ...

See the following example that the command line to search with highlighting enabled:



* Curl
* Java
* Node
* Python
* Go



curl "https://$ACCOUNT.cloudant.com/movies/_design/searches/_search/movies?q="movie_name:Azazel"&highlight_fields=["movie_name"]&highlight_pre_tag=" "&highlight_post_tag=" "&highlights_size=30&highlights_number=2" 	-X GET

import com.ibm.cloud.cloudant.v1.Cloudant;
import com.ibm.cloud.cloudant.v1.model.PostSearchOptions;
import com.ibm.cloud.cloudant.v1.model.SearchResult;

import java.util.Arrays;

Cloudant service = Cloudant.newInstance();

PostSearchOptions searchOptions = new PostSearchOptions.Builder()
.db("movies")
.ddoc("searches")
"words": [
{
"word": "ＮＹ",
"translation": "ニューヨーク",
"part_of_speech": "Mesi"
},
{
"word": "ＮＹＣ",
"translation": "ニューヨークシティ",
"part_of_speech": "Mesi"
},
{
"word": "ＹＣ",
"translation": "ヨコハマチューカガイ",
"part_of_speech": "Mesi"
}
]
}
Show more

With these entries, assume that the service receives the following input text: 一週間ＮＹＣを訪問した. In this case, the service matches the word ＮＹＣ because ＮＹＣ is longer than ＮＹ and because ＮＹＣ matches before ＹＣ.





 Japanese parts of speech 

The following table lists the parts of speech that are supported for Japanese custom entries. For more information about specifying the part of speech for a Japanese custom entry, see [Adding words to a Japanese custom model](https://cloud.ibm.com/docs/text-to-speech?topic=text-to-speech-customWordscuJapaneseAdd).



Table 1. Japanese parts of speech

 part_of_speech argument Japanese meaning English meaning 

 Dosi Doushi Verb 
 Fuku Fukishi Adverb 
 Gobi Gobi Inflection 
 Hoka Hoka Other (Words that have a special grammatical meaning of their own that does not fit into any other part of speech. For example, ありがとう for "thank you.") 
 Jodo Jodoushi Auxiliary verb 
 Josi Joshi Postpositional particle (For example, が の を for "of.") 
 Kato Kantoushi Interjection 
 Kedo Keiyodoushi Adjective verb 
 Keyo Keiyoshi Adjective (For example, 美し for "beautiful" or 明る for "bright.") 
 Kigo Kigou Symbol 
 Koyu Koyuumeishi Proper noun 
 Mesi Meishi Noun 
 Reta Rentaishi Determiner 
 Stbi Setsubiji Suffix 
 Stto Settoji Prefix
Related material includes related sentences; footnotes; and sentences from other parts of the document that refer to the table. The list is represented as an array. Each object in the array consists of the following elements:



* text: The text contents of a related material from the input document, without HTML markup.
* location: The location of the related material in the input document as defined by its begin and end indexes.









 Notes on the table output schema 



* Row and column index values per cell are zero-based and so begin with 0.
* Multiple values in arrays of row_header_ids and row_header_texts elements indicate a possible hierarchy of row headers.
* Multiple values in arrays of column_header_ids and column_header_texts elements indicate a possible hierarchy of column headers.









 Examples 

The following table is an example table from an input document.

Zoom

![Example table](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/example-table.png)

Figure 1. Table example

The table is composed as follows:

Zoom

![Table composition](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/table-comp.png)

Figure 2. Anatomy of the example table

The following syntax is used in the table:



* Bold text indicates a column header
* Italic text indicates a row header
* Unstyled text indicates a body cell



The output from service represents the example's first body cell (that is, the first cell in row 3 with a value of 35.0%) as follows:

{
"tables": [ {
"location": {
"begin": 872,
"end": 5879
},
* UI
* CLI
* API
* Terraform







 Working with tags 

Use tags to organize, track usage costs, and even manage access to your resources and service IDs. You can tag related resources and view them throughout your account by filtering by tags from your resource list.

To see a full list of tags in your account, go to Manage > Account in the IBM Cloud® console, and select Tags.

You can apply user tags to organize your resources and service IDs and easily find them later. User tags can also help you with identifying specific team usage or cost allocation. By creating access management tags, you can control access to your resources and service IDs without requiring updates to your IAM policies.



 Tagging rules 

Tags are not case-sensitive, and the maximum length of a tag is 128 characters. The permitted characters are A-Z, 0-9, spaces, underscore, hyphen, period, and colon. The only supported format for access management tags is key:value. The use of a colon formats the tag into a string that isolates two logical parts, like a env:dev pair. A comma separates multiple tags and can't be used within the tag name itself.

Tags are visible account-wide and can be replicated across geographic regions. Since tags are not regulated information, avoid creating tags that use personal information, such as your name, address, phone number, email address, or other identifying or proprietary information.



 Sample tags and syntax 

You can apply tags to help you organize and manage your resources, service IDs, and access policies. Consider writing tags as key:value pairs to help coordinate your development environments, projects, compliance, and optimization throughout your organization. See the following table for some examples of tags that you might want to use.



Table 1. Tag syntax

 Tag Description 

 env:dev, env:test, env:stage, env:prod Use to identify or even manage access to your development environment 
 project:lw-wizard, app:poc-app Use to identify or even manage access to a project 
 dataresidency:germany, compliance:hipaa, compliance:pii Use to define compliance requirements
* Maximum value length - 256 Unicode characters in UTF-8
* Maximum value byte size - 512 bytes
* A Tag key and value may consist of US Alpha Numeric Characters (a-zA-Z0-9), and spaces representable in UTF-8, and the following symbols: !, _, ., , , (, ), -, :
* Tag keys and values are case-sensitive
* ibm: cannot be used as a key prefix for tags



As noted previously, log in to your instance and navigate to the bucket and object you wish to "tag." In the panel that appears when you select "Manage your tags", start by clicking on the "Add tags +" button. Then, you can add tags by typing text into the key and value fields as desired. Add more tags one at a time, by repeating the steps you've just completed.

![Add tags in place](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/object-add-tags.jpg)

If you do not click on "save" when completing your changes, a dialog box will remind you of the consequences. That is, changes are discarded unless saved.

![Unsaved changes are discarded](https://cloud.ibm.com/docs-content/v1/content/63322f2f3a72050206eb05ba7f590fdcb741105c/cloud-object-storage/images/object-discard-changes.jpg)

As noted previously, you will have to authenticate in order to add tags to your data. If you have questions about bearer tokens, see [this example](https://cloud.ibm.com/docs/cloud-object-storage?topic=cloud-object-storage-curlcurl-token). Again, note the query string for working with tags: ?tagging.
Why aren't all the tags restored when I restore by digest? 

You want to restore an image by [digest](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_digest) from the IBM Cloud® Container Registry trash, but some [tags](https://cloud.ibm.com/docs/Registry?topic=Registry-registry_overviewoverview_elements_tag) weren't restored.

 What’s happening 

You run the [ibmcloud cr image-restore](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_restore) command, but the tags were not restored. If all the tags were unsuccessful, the digest shows in the live repository, but it is not tagged. You can see the digest if you run [ibmcloud cr image-digests](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_digests), but not if you run [ibmcloud cr image-list](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_list).

 Why it’s happening 

The tags that were not restored are already in your live repository. You can't overwrite a tag with a tag that is in the trash.

 How to fix it 

Remove the tag from the existing image in your live repository by running the [ibmcloud cr image-untag](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_untag) command. You can then restore the required image from the trash by running the [ibmcloud cr image-restore](https://cloud.ibm.com/docs/Registry?topic=Registry-containerregclibx_cr_image_restore) command with the option <repo>@<digest>, which restores the digest and its tags to the live repository.
A Filter must specify exactly one Prefix, Tag, or an And child element. 
 Prefix String None Filter An object key name prefix that identifies the subset of objects to which the rule applies. 
 Tag String None Filter A container for specifying a tag key and value. The rule applies only to objects that have the tag in their tag set. 
 And String None Filter A container for specifying rule filters. The filters determine the subset of objects to which the rule applies. This element is required only if you specify more than one filter. 
 Key String None Tag The tag key. 
 Value String None Tag The tag value. 



This example will replicate any new objects, but will not replicate delete markers.

curl -X "PUT" "https://$BUCKET.s3.$REGION.cloud-object-storage.appdomain.cloud/?replication" -H 'Authorization: bearer $TOKEN' -H 'Content-MD5: exuBoz2kFBykNwqu64JZuA==' -H 'Content-Type: text/plain; charset=utf-8' -d $'<ReplicationConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Rule>
<ID>SimpleReplication</ID>
<Priority>1</Priority>
<Status>Enabled</Status>
<DeleteMarkerReplication>
<Status>Disabled</Status>
</DeleteMarkerReplication>
<Filter/>
<Destination>
<Bucket>$DESTINATION_CRN</Bucket>
</Destination>
</Rule>
</ReplicationConfiguration>'
Show more

This example will replicate any objects with a key (name) that begin with project_a/ to the bucket identified with $DESTINATION_CRN_A, and any objects with a key (name) that begin with project_b/ to the bucket identified with $DESTINATION_CRN_B, and any objects that have an object tag with the key Client and the value ACME to a third bucket identified with $DESTINATION_CRN_C, and will replicate delete markers in all cases.