vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Example output {"log": "hello", "level": "info"}

--message MESSAGE
: Optional: Filters out any logs that contain a specified message anywhere in the log. Example: The messages "Hello", "!", and "Hello, World!", would apply to the log "Hello, World!".

--regex-message MESSAGE
: Optional: Filters out any logs that contain a specified message that is written as a regular expression anywhere in the log. Example: The pattern "hello [0-9]" would apply to "hello 1", "hello 2", and "hello 9"

--force-update
: Force your Fluentd pods to update to the latest version. Fluentd must be at the latest version to change your logging configurations.

--output json
: Optional: Prints the command output in JSON format.

-q
: Optional: Do not show the message of the day or update reminders.

Examples:

This example filters out all logs that are forwarded from containers with the name test-container in the default namespace that are at the debug level or less, and have a log message that contains "GET request".

ibmcloud ks logging filter update --cluster example-cluster --id 885274 --type container --namespace default --container test-container --level debug --message "GET request"

This example filters out all the logs that are forwarded, at an info level or less, from a specific cluster. The output is returned as JSON.

ibmcloud ks logging filter update --cluster example-cluster --id 274885 --type all --level info --output json





 ibmcloud ks logging refresh 

Virtual Private Cloud

Classic infrastructure

Refresh the logging configuration for the cluster. This action refreshes the logging token for any logging configuration that is forwarding to the space level in your cluster.

The logging config refresh alias for this command is deprecated.

ibmcloud ks logging refresh --cluster CLUSTER [--force-update] [-q]

Minimum required permissions
Acceptable values in their canonical order are fatal, error, warn/warning, info, debug, and trace. As an example, if you filtered logs at the info level, debug, and trace are also filtered. Note: You can use this option only when log messages are in JSON format and contain a level field. Example output {"log": "hello", "level": "info"}

--message MESSAGE
: Optional: Filters out any logs that contain a specified message anywhere in the log. Example: The messages "Hello", "!", and "Hello, World!", would apply to the log "Hello, World!".

--regex-message MESSAGE
: Optional: Filters out any logs that contain a specified message that is written as a regular expression anywhere in the log. Example: The pattern "hello [0-9]" would apply to "hello 1", "hello 2", and "hello 9".

--force-update
: Force your Fluentd pods to update to the latest version. Fluentd must be at the latest version to change your logging configurations.

--output json
: Optional: Prints the command output in JSON format.

-q
: Optional: Do not show the message of the day or update reminders.

Examples:

This example filters out all logs that are forwarded from containers with the name test-container in the default namespace that are at the debug level or less, and have a log message that contains "GET request".

ibmcloud ks logging filter create --cluster example-cluster --type container --namespace default --container test-container --level debug --message "GET request"

This example filters out all the logs that are forwarded, at an info level or less, from a specific cluster. The output is returned as JSON.

ibmcloud ks logging filter create --cluster example-cluster --type all --level info --output json





 ibmcloud ks logging filter get 

Virtual Private Cloud

Classic infrastructure

View a logging filter configuration.
Filter log data by using pre-defined filters.

You can filter logs by log source, application, and log level.



* A source can be a host, a computer, a virtual machine, or a Heroku app.
* An application represents a log file, a program, or a container.
* Examples of log levels are: INFO, DEBUG, ERROR



Complete the following steps to filter logs:



1. Click the Views icon ![Views icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/views.png).
2. Select EVERYTHING or a view.
3. Expand Tags to see the list of tags that are identified in the logs. Then, choose the ones that you want.
4. Expand Sources to see the list of log sources that are identified in the logs. Then, choose the ones that you want.
5. Expand Apps to see the list of apps that are identified in the logs. Then, choose the ones that you want.
6. Expand Levels to see the list of log levels that are identified in the logs. Then, choose the ones that you want.



In each section, you can group multiple options into a group. Group tags, log sources, apps, and log levels to reuse these groupings when you filter log data in other custom views. To create a group, select multiple values. Then, click Save as group. Enter a name for the group, and save it.
3. Filter log data by adding a search query.

When you search log data, the search applies any log filters and time queries configured in that view.

You can do simple searches (single term string search), compound search (multiple search terms and operators), field searches if the log line can be parsed, and others. For more information, see [How to Search Logs](https://docs.logdna.com/docs/search).
Step 5. Filter logs 

You can filter logs by log source, application, and log level.



* A source can be a host, a computer, a virtual machine, or a Heroku app.
* An application represents a log file, a program, or a container.
* Examples of log levels are: INFO, DEBUG, ERROR.



Complete the following steps to filter logs:



1. Click the Views icon ![Views icon](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/views.png).
2. Select EVERYTHING or a view.
3. Expand Tags to see the list of tags that are identified in the logs. Then, choose the ones that you want.
4. Expand Sources to see the list of log sources that are identified in the logs. Then, choose the ones that you want.
5. Expand Apps to see the list of apps that are identified in the logs. Then, choose the ones that you want.
6. Expand Levels to see the list of log levels that are identified in the logs. Then, choose the ones that you want.



In each section, you can group multiple options into a group. Group tags, log sources, apps, and log levels to reuse these groupings when you filter log data in other custom views.

To create a group, select multiple values. Then, click Save as group. Enter a name for the group, and save it.





 Step 6. Search logs 

When you search log data, the search applies any log filters and time queries configured in that view.

You can do simple searches (single term string search), compound search (multiple search terms and operators), field searches if the log line can be parsed, and others. For more information, see [How to Search Logs](https://docs.logdna.com/docs/search).
A notification sent to the Event Notifications service must conform to the [CNCF](https://www.cncf.io/)[CloudEvents](https://cloudevents.io/) format.





 More about filters 

A filter is a conditional statement, which connects a source to a topic. Filters are written to route notifications of interest to a particular topic. All notifications that pass through the filters into a topic are then routed to the topic subscribers. Filtering is absent between topic and destination.

To simplify filtering, a source might include event categories in their notifications. Event categories are standard filter keys with the following hierarchy: Event Category -> Event Type -> Severity. Event categories simplify filtering because they appear as dropdown selection boxes when you are creating topics and filters in the Event Notifications UI. For more advance filtering, use [JSONPath](https://jsonpath.com/) in the Custom Filter field. For more on JSONPath, see [JSONPath Online Evaluator](https://restfulapi.net/json-jsonpath/).
An intent name in the response exactly matches order, and an entity name in the response exactly matches beverage. [response.output.intents:intent::order,response.output.entities:entity::beverage] 







 Filtering v1 logs 

If your application is still using the v1 API, you can query and filter logs using the v1 /logs method. The filtering syntax is the same, but the structure of v1 logs and message requests is different. For more information, see [API Reference](https://cloud.ibm.com/apidocs/assistant/assistant-v1listlogs).

With the v1 /logs API, you can filter on the following fields:



* language
* meta.message.entities_count
* request.context.metadata.deployment
* request.context.system.assistant_id
* request.input.text
* response.context.conversation_id
* response.entities
* response.input.text
* response.intents
* response.top_intent
* workspace_id



Filtering on other fields is not currently supported.
Sorting or filtering the shipments list 

The Shipments screen within the [IBM Cloud console](https://cloud.ibm.com/login) displays the shipments that are associated with Data Transfer Service requests. All shipping requests from the last 30 days are displayed. Shipments can be sorted and filtered to view older shipments or specific shipment types.



  Sorting and filtering 



1.  Access the Shipments screen within the [IBM Cloud console](https://cloud.ibm.com/login).
2.  Determine whether the view is to be filtered for specific details or sorted by shipment status.



*  If you want to sort by shipment status, you can select one of the following options from the Display list: Open Shipping Requests, Closed Shipping Requests, Canceled Shipping Requests or All Shipping Requests.
*  If you want to filter for specific details, select one, or more of the filter options available and click Filter.





Zoom

![DTS Shipment Screen](https://cloud.ibm.com/docs-content/v1/content/8188f85967d873aeb00bdd6c5f3fd757e84176ab/DataTransferService//images/DTSShipmentScreen.svg)

Figure 1 shows the shipment screen.



Table 1. Filter options.
Table 1 shows the filter options available in the Shipment screen. The left column lists the available options and the right side column provides their descriptions.

 Filter option     Entry information                                                                   

 Shipped date      Select the time frame of the shipment.                                              
 Shipped from      Select the location where the shipment originates from.                             
 Carrier           Select the courier (for example FedEx, UPS, or USPS) that transports the shipment.  
 Shipment type     Select the type of shipment.                                                        
 Shipped To        Select the location to which the shipment was sent.                                 
 Tracking number   Enter the tacking number that was provided by the courier.                          
 Item Description  Enter the description of the item that is shipped.                                  



After you filtered the Shipments List, all shipments that correspond with the parameters that are set by the filter are displayed in the Shipment List. Click Modify, and update the filter options if you want to change the list. To clear all parameters that are set by the filter, click Clear All.

If the results were sorted by the shipment status, all shipments that are within the selected status are displayed. Select a new status from the list to update the displayed results at any time.
If not provided, the filter is applied to all the cluster logging configurations that are passed to the filter. You can view log configurations that match the filter by using the --show-matching-configs option. 
 <kubernetes_namespace> Optional: The Kubernetes namespace that you want to forward logs from. This option applies only when you are using log type container. 
 <container_name> Optional: The name of the container from which you want to filter logs. 
 <logging_level> Optional: Filters out logs that are at the specified level and less. Acceptable values in their canonical order are fatal, error, warn/warning, info, debug, and trace. As an example, if you filtered logs at the info level, debug, and trace are also filtered. Note: You can use this option only when log messages are in JSON format and contain a level field. To display your messages in JSON, append the --output json option to the command. 
 <message> Optional: Filters out logs that contain a specified message that is written as a regular expression. 
 <filter_ID> Optional: The ID of the log filter. 
 --show-matching-configs Optional: Show the logging configurations that each filter applies to. 
 --all Optional: Delete all your log forwarding filters. 





1. Create a logging filter.

ibmcloud ks logging filter create --cluster <cluster_name_or_ID> --type <log_type> --logging-configs <configs> --namespace <kubernetes_namespace> --container <container_name> --level <logging_level> --regex-message <message>
2. View the log filter that you created.

ibmcloud ks logging filter get --cluster <cluster_name_or_ID> --id <filter_ID> --show-matching-configs
3. Update the log filter that you created.
Therefore, to filter the logs to only include the smart contract events, you need to include app:fluentd in the search input box.
* Finally, you can append the name and optionally the version of the smart contract, marbles 1.0.0.



The full search string, including the namespace, would be similar to the following example:

pod:n4c817fpeer1org1-55885d5666-zq55t app:fluentd marbles 1.0.0

Zoom

![Filtering on smart contract](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/logDNAsc.png)

Figure 3.Filtering logs by smart contract





 Hyperledger Fabric v2.x peer image 

![version 2.x](https://cloud.ibm.com/docs-content/v1/content/c5288bed34c3820e3a5251d820ee91adcd2591a3/blockchain/images/2-x_Pill.png) In order to filter on the smart contract logs, you need to provide the name of the pod where the smart contract is running. Complete the following steps to get the name of the smart contract pod:

Find your cluster namespace

If you don't already know it, you need to find your Kubernetes cluster namespace. From the console, open any CA node and click the Info and Usage icon. View the value of the API URL. For example: https://n2734d0-soorg10524.ibpv2-cluster.us-south.containers.appdomain.cloud:7054. The namespace is the first part of the URL beginning with the letter n and followed by a random string of six alphanumeric characters. So in the example, the value of the namespace is n2734d0.

Find the smart contract pod
Depending on the filter, the subsequent drop-down lists reflect the appropriate options.





 Option Description 

 length Filter according to whether the column content is shorter, longer, or equal to a specified number of characters or tokens. 
 text Filter according to whether the text matches or contains a regular expression or dictionary from the canvas, or you can enter a new regular expression. You can also set case sensitivity. 
 range Filter on the basis of specified criteria within a range of the specified column. For example, you might use starts with to narrow the results within matches so that you only get military rank matches that start with Chief. 





5. Once you complete your filter options, execute the extractor again to see how the results have changed.



Your Military Ranks extractor might produce a match for text Chief Warrant Officer John Doe, but you do not want to include results that have the word except preceding the match.



1. Create a dictionary with the term except and any other terms that you might want to use to exclude results.
2. Select Military Ranks from the canvas and open the Output tab under Extractor Properties.
3. Click New Filter.
4. Select Exclude and choose a column.
5. Select range and occurs after.
6. Select your dictionary that includes except.
7. Select the column and between0 to 2tokens.



This filter excludes any matches that have the word except within 0-2 tokens before a match.

]









 Extracting in languages other than English 

To extract text from non-English documents, you can sometimes use or modify the provided extractors or define new extractors based on linguistic patterns. Also, if target terms are based on a pattern, you can design a sequence pattern.

You can use the workspace to build extractors to be used with any language, such as Spanish and French, that is based on tokens defined by white space and punctuation. For example, the results of using the Person extractor with Spanish text might extract names as shown here.

(images/samplefilter.png "Sample filter")]



 Extending the provided extractors for use in a different language
An intent name in the response exactly matches either hello or goodbye. response.output.intents:intent::(hello goodbye) 
 An intent name in the response exactly matches order, and an entity name in the response exactly matches beverage. [response.output.intents:intent::order,response.output.entities:entity::beverage] 







 Filtering v1 logs 

If your application is still using the v1 API, you can query and filter logs using the v1 /logs method. The filtering syntax is the same, but the structure of v1 logs and message requests is different. For more information, see [API Reference](https://cloud.ibm.com/apidocs/assistant/assistant-v1listlogs).

With the v1 /logs API, you can filter on the following fields:



* language
* meta.message.entities_count
* request.context.metadata.deployment
* request.context.system.assistant_id
* request.input.text
* response.context.conversation_id
* response.entities
* response.input.text
* response.intents
* response.top_intent
* workspace_id



Filtering on other fields is not currently supported.
There are different ways in which you can filter out events sent to Activity Tracker:



* For some Cloud services you can [configure the collection of events.](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-events-opt-in)
* You can define [exclusion rules](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-exclusion_rules) for events before they are stored for search.



* You can drop the events entirely and not see them at all through the UI
* You can view the events in the UI, but you cannot search on them. However, you can define views and alerts based on the data from these logs.



* You can also configure [usage quotas](https://cloud.ibm.com/docs/activity-tracker?topic=activity-tracker-control_usage_quotas) and define conditional usage quota exclusion rules.





Log Analysis



* Where can I find the list of Cloud services that send logs?

You can find the list of Cloud services that generate logs in the following documentation topic: [Cloud services](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-cloud_services).
* Where can I find details about the logs that a Cloud service generates?

You can access more information about the logs that each Cloud service generates from the following documentation topic: [Cloud services](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-cloud_services). For each Cloud service, you can link to the logging topic specific to a Cloud service where you can get more information.
* Where can I find the logs that a Cloud service generates?

First, you must [check whether you have enabled Platform logs in the location where your Cloud service is available](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-config_svc_logs).
[Non-persistent message get - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_non_persistent_message_get_bytes_total) 
 [Persistent - topic MQPUT/MQPUT1 count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_topic_mqput_mqput1_total) 
 [Persistent message MQPUT count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_mqput_total) 
 [Persistent message MQPUT1 count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_mqput1_total) 
 [Persistent message browse - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_browse_bytes_total) 
 [Persistent message browse - count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_browse_total) 
 [Persistent message destructive get - count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_destructive_get_total) 
 [Published to subscribers - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_published_to_subscribers_bytes_total)
[Persistent message destructive get - count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_destructive_get_total) 
 [Published to subscribers - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_published_to_subscribers_bytes_total) 
 [Published to subscribers - message count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_published_to_subscribers_message_total) 
 [Purged queue count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_purged_queue_total) 
 [Put non-persistent messages - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_non_persistent_message_put_bytes_total) 
 [Put persistent messages - byte count](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_persistent_message_put_bytes_total) 
 [Queue Manager file system - bytes in use](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_queue_manager_file_system_in_use_bytes) 
 [Queue Manager file system - free space](https://cloud.ibm.com/docs/mqcloud?topic=mqcloud-monitor_sysdigibm_mq_qmgr_queue_manager_file_system_free_space_percentage)
* filter(enriched_text.entities.text:"Pittsburgh Steelers").term(enriched_text.sentiment.document.label,count:3)



This example aggregation first filters a set of articles in Discovery News to only the ones that include the entities text of twitter. Next, the aggregation divides those articles by the document sentiment types. Only the top 3 document sentiment types (positive, negative, neutral) are returned.



* filter(enriched_text.entities.text:twitter).term(enriched_text.sentiment.document.label,count:3)







 Nested aggregations 

Adding nested before an aggregation restricts the aggregation to the area of the results specified. For example: nested(enriched_text.entities) means that only the enriched_text.entities components of any result are used to aggregate against.

This can be seen easily by looking at the differences between the following two queries:



* filter(enriched_text.entities.disambiguation.subtype::City) - the aggregation counts the number of Results that contain one or more entity with the type City
* nested(enriched_text.entities).filter(enriched_text.entities.disambiguation.subtype::City) - the aggregation counts the number of instances of an entity with the type City in the results.



Additionally, any subsequent operation further restricts the result set that can be aggregated against. For example:



* nested(enriched_text.entities).filter(enriched_text.entities.disambiguation.subtype::City) means that only entities of subtype::City are aggregated.
* nested(enriched_text.entities).filter(enriched_text.entities.disambiguation.subtype::City).term(enriched_text.entities.text,count:3) aggregates the top 3 entities of subtype City
* filter(enriched_text.entities.disambiguation.subtype::City).term(enriched_text.entities.text,count:3) returns the top 3 entities where the result contains at least one entity of subtype City.











 Querying Watson Discovery News