vania-janet/multiturn-rag-retrieval-data huggingface dataset:
By examining the text, we determine that 31-year-old is an age, male is a gender, and together they signify a person. You are dealing with 3 entity types, basically: AGE, GENDER, and PERSON. One approach to representing this would be to annotate 31-year-old as AGE, male as GENDER, and because the distinct mention of a PERSON is missing, but male has come to represent person, the word male can have a role of PERSON. You can then capture relations between the PERSON role and all the attributes of a person that you want to capture. For example, you can define a hasAttribute relation between the PERSON role and AGE mention. Because you applied both a GENDER entity type and a PERSON role type label to the same word male, you cannot define a hasAttribute relation between the two. However, the fact that the two labels are applied to the same mention associates them with one another. This association is considered by the machine learning model without you having to define a hasAttribute relation type to explicitly call out the relationship.

A similar example is when the phrase 2008 Ford F-150 is used as shorthand for 2008 Ford F-150 truck. Here, 2008 is annotated as a MODEL_YEAR entity type, Ford is annotated as a MANUFACTURER entity type and F-150 is given a MODEL entity type. But with the word truck missing, F-150 also represents a vehicle. Saying the F-150 is like saying the truck. You can capture that usage by adding a VEHICLE role to the mention in addition to its MODEL entity type. You can then define hasAttribute relations between the VEHICLE role and the MANUFACTURER and MODEL_YEAR entity mentions.



 How are roles different from subtypes? 

Entity subtypes are for stratifying an inventory of entity types into a hierarchy. For example, if you care to distinguish 40 things, but they logically group into 10 sets of 4 (VITALSIGN_BLOODPRESSURE, VITALSIGN_HEARTRATE, MEDICATION_DOSAGE, MEDICATION_FREQUENCY), you could structure them into types and subtypes, which yields more compact menus, but adds a level of depth and nuisance to the labeling process.
The labels that can be applied are entity types defined in the workspace's type system.



 About this task 

Before starting to annotate entity mentions in a document, it's a good practice to read the entire document. Doing so can help keep the entire context in mind while annotating, and can help provide insights into how entity mentions might relate to each other and which mentions might need to be coreferenced in future passes through the document.

When you open a document to annotate it, you might want to use the concordance tool to annotate repeating entity mentions first, and then annotate individual entity mentions. You can then annotate relation mentions and coreferences in any order that you like, or not at all. Entity mention annotation is mandatory. Whether you also annotate relation mentions and coreferences depends on the purpose of your model and the needs of your domain. However, until and unless you identify coreferences, each entity mention is considered to represent a distinct entity.



 Tips 



* Keep in mind that shorter entity mentions are better for training because it is easier for the machine learning model to recognize the shorter patterns and add the correct annotation tokens.
* If you chose to use a dictionary-based tokenizer with the workspace, and want to handle compound terms and punctuation in your training data, you can add the terms to a dictionary and create a dictionary annotator to pre-annotate the occurrences. For example, to avoid sentence boundary breaks for terms that include punctuation, add terms like Yahoo! and Dr. to a dictionary. Likewise, if your training data includes hyphenated words or alphanumeric acronyms, like Hi-C or MS-60-70, add those terms to the dictionary. To annotate occurrences regardless of case, add the terms in lowercase (such as hi-c). To annotate variations, add the variations as surface forms (MS-60-70 and MS 60 70).

Important: Do not use this approach if you are using the default tokenizer.









 Procedure 

To annotate entity mentions in a document:



1. Log in as a human annotator (or as an administrator who was assigned documents to annotate).
* Flag: --marker KEY



* Optional: Limits the response to keys that begin with the specified PREFIX.



* Flag: --prefix PREFIX



* Optional: The SIZE of each page to get in the service call. This does not affect the number of items returned in the command's output. Setting a smaller page size results in more calls to the service, retrieving fewer items in each call. This can help prevent the service calls from timing out.



* Flag: --page-size SIZE



* Optional: The total NUMBER of items to return in the command's output.



* Flag: --max-items NUMBER



* Optional: Output FORMAT can be only json or text.



* Flag: --output FORMAT













 List in-progress multipart uploads 



* Action: Lists in-progress multipart uploads.
* Usage:ibmcloud cos multipart-uploads --bucket BUCKET_NAME [--delimiter DELIMITER] [--encoding-type METHOD] [--prefix PREFIX] [--key-marker value] [--upload-id-marker value] [--page-size SIZE] [--max-items NUMBER] [--region REGION] [--output FORMAT]
* Parameters to provide:



* The name of the bucket.



* Flag: --bucket BUCKET_NAME



* Optional: A DELIMITER is a character that you use to group keys.



* Flag: --delimiter DELIMITER



* Optional: Requests to encode the object keys in the response and specifies the encoding METHOD to use.



* Flag: --encoding-type METHOD



* Optional: Limits the response to keys that begin with the specified PREFIX.



* Flag: --prefix PREFIX



* Optional: Together with upload-id-marker, this parameter specifies the multipart upload after which listing should begin.



* Flag: --key-marker value



* Optional: Together with key-marker, specifies the multipart upload after which listing should begin. If key-marker is not specified, the upload-id-marker parameter is ignored.



* Flag: --upload-id-marker value
"interpretation": {
"value": string,
"numeric_value": number,
"unit": string,
},
"provenance_ids": string, string, ... ],
"location": { "begin": int, "end": int }
},
...
],
"contract_currencies": [
{
"confidence_level" : string,
"text" : string,
"text_normalized" : string,
"provenance_ids": string, string ..],
"location": { "begin": int, "end": int }
},
...
],
"tables": [],
"document_structure": {
"section_titles": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"level": int,
"element_locations":
{
"begin": int,
"end": int
},
...
]
},
...
],
"leading_sentences": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"element_locations":
{
"begin": int,
"end": int
},
...
],
"paragraphs":
{
"location": {
"begin": int,
"end": int
}
},
...
]
},
...
]
},
"parties": [
{
"party": string,
"role": string,
"importance": string,
"addresses":
{
"text": string,
* Optional: Specifies CACHING_DIRECTIVES for the request and reply chain.



* Flag: --cache-control CACHING_DIRECTIVES



* Optional: Specifies presentation information (DIRECTIVES).



* Flag: --content-disposition DIRECTIVES



* Optional: Specifies what content encodings (CONTENT_ENCODING) are applied to the object and thus what decoding mechanisms must be applied to obtain the media-type referenced by the Content-Type header field.



* Flag: --content-encoding CONTENT_ENCODING



* Optional: The LANGUAGE the content is in.



* Flag: --content-language LANGUAGE



* Optional: A standard MIME type describing the format of the object data.



* Flag: --content-type MIME



* Optional: Copies the object if its entity tag (Etag) matches the specified tag (ETAG).



* Flag: --copy-source-if-match ETAG



* Optional: Copies the object if it has been modified since the specified time (TIMESTAMP).



* Flag: --copy-source-if-modified-since TIMESTAMP



* Optional: Copies the object if its entity tag (ETag) is different than the specified tag (ETAG).



* Flag: --copy-source-if-none-match ETAG



* Optional: Copies the object if it hasn't been modified since the specified time (TIMESTAMP).



* Flag: --copy-source-if-unmodified-since TIMESTAMP



* Optional: A MAP of metadata to store.



* Flag: --metadata MAP JSON Syntax: The --metadata flag takes the file:// prefix that is used to load the JSON structure from the specified file.





{
"file_name": "file_20xxxxxxxxxxxx45.zip",
"label": "texas",
"state": "Texas",
"Date_to": "2019-11-09T16:00:00.000Z",
"Sha256sum": "9e39dxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx8ce6b68ede3a47",
[checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Enable answer finding to return a succinct answer from a passage ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 
 Use table retrieval ![checkmark icon](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/icons/checkmark-icon.svg) 







 Query analysis 

When you submit a query, the query text string is analyzed. During query analysis, the root (or lemma) of each key term in the query is identified. Any stop words that occur in the original query string are removed and synonym expansions that are defined for any terms that occur in the original query string are added. This enhanced version of the query is what gets submitted to Discovery.

The same analysis is performed on all queries, whether they are submitted as natural language queries or by using Discovery Query Language syntax.





 Query flow 

The following diagram shows a conceptual illustration of how a search request is handled by Discovery.

Zoom

![Shows the different ways that query and natural language query requests are handled by Discovery](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/query-flow.png)

Figure 1. Flow chart that shows the processes that are used for Natural Language Queries versus Discovery Query Language queries

The following processes are shown in the flow diagram:

BM25
Why can't I attach or detach access management tags on a resource? 

If you can't attach or detach access management tags on a resource, you might need to check the following items:



*  List of the existing tags
*  The type of access you're assigned
*  Type of the resource



  What’s happening 

You can't attach or detach access management tags on a resource, which you have access to.

  Why it’s happening 

This problem can be commonly caused by the following reasons:



*  You're trying to attach an access management tag that doesn't exist yet in the account, which you are a member of. You need to [create the access management tag](https://cloud.ibm.com/account/tags) first.
*  You might not be assigned the appropriate access to complete the actions. You must have the Administrator role on the resource.
*  The resource is not an IAM-enabled resource. Only a set of IBM Cloud® services is enabled to use IBM Cloud® IAM for access control. These services need to be labeled as "IAM-enabled" in the catalog.
*  Some services don't have full support for access management tags on their UI, so when you're creating an instance of those services, you can only attach user tags. To learn how you can work with access management tags, see the tutorial about [controlling access to resources by using tags](https://cloud.ibm.com/docs/account?topic=account-access-tags-tutorial).



  How to fix it 

Complete the following steps to check your level of access. If you need to request access, contact the account owner.

Go to Manage > Access (IAM) in the IBM Cloud console, and select your name on the Users page.



*  To view IAM access policies that are assigned to you, select Access and view the Access policies table.
*  To determine what access you have through the access groups you are assigned, select Access and view the Access groups table. Check the access policies for each of the access groups.



To view what actions are mapped to each role, click the numbers listed next to each role.
In this example, the extract statement identifies blocks of two to three capitalized words. In AQL, a block refers to a contiguous span of tokens, in this case two to three tokens. This example also uses a consolidation policy to exclude blocks that are contained within larger blocks from the output set of tuples.

create view CapitalizedWord as
extract
regex /[A-Z]/
with flags 'CANON_EQ'
on 1 token in D.text
as word
from Document D;

create view TwoToThreeCapitalizedWords as
extract blocks
with count between 2 and 3
and separation 0 tokens
on CW.word as capswords
from CapitalizedWord CW
consolidate on capswords using 'ContainedWithin';

The consolidate clause is applied to the capswords field by the extract blocks specification. The difference is that the target field referred to by the consolidation clause is an output field of the extract statement. The target field of the select statement is an input field. This behavior is similar to that of the having clause.

Example 3: A nested extract or select statement as a view name

The input view for an extract statement can be either a view name, as in Example 2, or a nested extract or select statement, as in this example:

create view SampleExtract as
extract
regex /foo/ on E.foobar as foo
from
(extract regex /foobar/ on D.text as foobar
from Document D) E;

Example 4: Extract a statement with a select list

In this example, we extract matches for a pattern, and at the same time, select multiple attributes from the input views.

create view Person as
extract.F.first as first, M.initial as middle, L.last as last
pattern ('Mr.'|'Ms.'|'Miss')? (<F.first> <M.initial>? <L.last>)
return group 0 as reference
and group 1 as salutation
and group 2 as name
from FirstName F, MiddleInitial M, LastName L;
* Parameters to provide:



* The name of the bucket.
* Flag: --bucket BUCKET_NAME
* Optional: The REGION where the bucket is present. If this flag is not provided, the program uses the default option that is specified in config.
* Flag: --region REGION
* Optional: Output FORMAT can be only json or text.
* Flag: --output FORMAT









 Complete a multipart upload 



* Action: Complete a multipart upload instance by assembling the currently uploaded parts and uploading the file to the bucket in the user's IBM Cloud Object Storage account.
* Usage:ibmcloud cos multipart-upload-complete --bucket BUCKET_NAME --key KEY --upload-id ID --multipart-upload STRUCTURE [--region REGION] [--output FORMAT]
* Parameters to provide:



* The name of the bucket.



* Flag: --bucket BUCKET_NAME



* The KEY of the object.



* Flag: --key KEY



* Upload ID identifying the multipart upload.



* Flag: --upload-id ID



* The STRUCTURE of MultipartUpload to set.



* Flag: --multipart-upload STRUCTURE
* Shorthand Syntax: --multipart-upload 'Parts=[{ETag=string,PartNumber=integer},{ETag=string,PartNumber=integer}]'
* JSON Syntax: --multipart-upload file://<filename.json> ======= The --multipart-upload command takes a JSON structure that describes the parts of the multipart upload that should be reassembled into the complete file. In this example, the file:// prefix is used to load the JSON structure from the specified file.



{
"Parts": [
{
"ETag": "string",
"PartNumber": integer
}
...
]
}
* Optional: The REGION where the bucket is present.
Regular expression A regular expression, also known as a regex, is a standardized format for defining search patterns. You can define patterns with special significance to your application. For example, the bill of materials (BOM) numbers for parts that you manufacture might have a standard syntax of two uppercase letters followed by four numbers (GT2345). You can teach Discovery to recognize BOM mentions by adding a regular expression that can recognize and tag occurrences of the pattern in text. [Learn more](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-domain-regex). 
 Stopword Words to filter out of queries because they are not useful in a search, such as a, an, and the. You can add words that are common and not useful to your use case as stopwords to improve the relevance of results for natural language queries. [Learn more](https://cloud.ibm.com/docs/discovery-data?topic=discovery-data-search-settingsstopwords).
ibmcloud resource tag-delete (--tag-name TAG_NAME | -a, --all [-f, --force]) [-p, --provider PROVIDER] [--tag-type TAG_TYPE] [--account-id ACCOUNT_ID]



 Command options 

--tag-name value
: Tag name to be deleted.

--provider value, -p value
: Delete the tag in the specified provider (only supported value is classic-infrastructure). Use it for resources of type SoftLayer_Hardware, SoftLayer_Network_Application_Delivery_Controller, SoftLayer_Network_Subnet_IpAddress or SoftLayer_Network_Vlan.

--tag-type value
: Type of the tag. Only allowed values are: user, service or access (default value : user).

account-id value
: The ID of the account that owns the tags to be deleted (required if tag-type is set to service).

--force, -f
: Delete the tags without confirmation.

--all, -a
: Delete all tags not attatched to any resources.

-q, --quiet
: Suppress verbose output.

A tag can be deleted only if it isn't attached to any resource.





 Examples 



* To delete the user tag MyTag from the account:

ibmcloud resource tag-delete --tag-name "MyTag"
* To delete the access management tag project:myproject from the account:

ibmcloud resource tag-delete --tag-name "project:myproject" --tag-type access
* To delete all unused user tags from the account:

ibmcloud resource tag-delete -a
* To delete all unused access management tags from the account:

ibmcloud resource tag-delete -a --tag-type access









 ibmcloud resource reclamations 

List reclaimed resources that can be restored or deleted:

ibmcloud resource reclamations [--resource-instance-id INSTANCE_ID]



 Command options 

--resource-instance-id
The begin index indicates that the string begins at character position 2510 in the transformed HTML, which is the location of the letter A in Amount. The end index indicates that the string ends at character position 2519, which is the location of the letter e in due.







 Document structure 

The output of the Contracts enrichment includes a document_structure object that details the structural composition of the input document. The document structure information is represented in the following JSON sample. The object is located immediately after the root-level tables array.

"document_structure": {
"section_titles": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"level": int
"element_locations":
{
"begin": int,
"end": int
},
...
]
},
...
],
"leading_sentences": [
{
"text": string,
"location": {
"begin": int,
"end": int
},
"element_locations":
{
"begin": int,
"end": int
},
...
]
},
...
],
"paragraphs": [
{
"location": {
"begin": int,
"end": int
}
},
...
]
}
Show more



 Document structure elements 

The elements of the document_structure object contain the following information:



* document_structure: An object that describes the structure of the input document.



* section_titles: An array that contains one object per section or subsection that is detected in the input document. Sections and subsections are not nested. Instead, they are flattened out and can be placed back in order by using the begin and end values of the element and the level value of the section.



* text: A string that lists the section title, if detected.
const params = {
resources: [resourceModel],
tagNames: ["project:myproject"],
tagType: 'access',
};

globalTaggingService.attachTag(params)
.then(res => {
console.log(JSON.stringify(res.result, null, 2));
})
.catch(err => {
console.warn(err)
});







 Detaching tags from a resource by using the API 

You can programmatically detach tags by calling the [Global Search and Tagging - Tagging API](https://cloud.ibm.com/apidocs/taggingdetach-tag) as shown in the following sample requests. The allowed values for the tag_type query parameter are: user for user tags and access for access management tags. The following example shows how to detach an access management tag that's called project:myproject from a service instance:



1. Find the unique identifier for your resource by calling the following command:



* Curl
* Java
* Python
* Go
* Node



curl -v -X POST -k --header 'Content-Type: application/json'
--header 'Accept: application/json'
--header 'Authorization: bearer <your IAM token>'
-d '{"query": "name:myresource"}' 'https://api.global-search-tagging.cloud.ibm.com/v3/resources/search'

SearchOptions searchOptions = new SearchOptions.Builder()
.query("GST-sdk-")
.fields(new java.util.ArrayList<String>(java.util.Arrays.asList("crn")))
.searchCursor(searchCursor)
.build();

Response<ScanResult> response = service.search(searchOptions).execute();
ScanResult scanResult = response.getResult();
[Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) 
 Role: Create, Edit, Delete, or Copy ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) 
 Security Tag Edit ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) 
 Selector Extensions: Manage ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg) ![Available](https://cloud.ibm.com/docs-content/v1/content/11f96f96f8f4b60cf6f3fa4d904856a2c4528bc2/icons/checkmark-icon.svg)
: The access tags of the resource in the form of name:value,name:value,....

--enforcement-mode (string)
: How the rule is enforced. The CLI accepts the values enabled (default), disabled, and report. For more informaiton about enforcement, see [Rule enforcement](https://cloud.ibm.com/docs/account?topic=account-context-restrictions-whatisrule-enforcement).

--file (string)
: The supplied file is used to update the rule. This flag is unique and cannot be used with other flags. The file needs to follow the JSON schema for the rule update API. For more information, see the [Context-based restrictions API](https://cloud.ibm.com/apidocs/context-based-restrictionsreplace-rule-request).







 ibmcloud cbr rule-delete 

This operation deletes the rule that is identified by the specified rule ID.

ibmcloud cbr rule-delete RULE-ID



 Example 

ibmcloud cbr rule-delete 30fd58c9b75f40e854b89c432318b4a2