vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Kubernetes audit logging with admission controller 

Kubernetes log integration enables IBM Cloud® Security and Compliance Center Workload Protection to use Kubernetes audit log data for Falco Rules and activity auditing. Integration allows auditing of the creation and destruction of pods, services, deployments, and daemon sets. Creating, updating, and removing config maps or secrets and attempts to subscribe to changes to any endpoint are also monitored.

IBM Cloud Security and Compliance Center Workload Protection implements Sysdig Secure functionality. Information that is provided by the Sysdig Secure documentation applies to Workload Protection as well.

For more information, see [Kubernetes audit logging](https://docs.sysdig.com/en/docs/sysdig-secure/secure-events/kubernetes-audit-logging/)
wget -O- https://assets.logdna.com/logdna.gpg | sudo apt-key add -

sudo apt-get update

sudo apt-get install logdna-agent < "/dev/null"
2. Set the ingestion key that the logging agent will use to forward logs to the IBM Log Analysis instance.

sudo logdna-agent -k <INGESTION_KEY>

where <INGESTION_KEY> contains the ingestion key for the IBM Log Analysis instance where the logs will be forwarded.

You can retrieve the ingestion key using the [IBM Cloud console](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-ingestion_keyibm_cloud_ui), or by the [IBM Cloud CLI](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-ingestion_keyingestion_key_cli).
3. Set the authentication endpoint. The logging agent uses this host to authenticate and get the token to forward logs.

sudo logdna-agent -s LOGDNA_APIHOST=api.us-south.logging.cloud.ibm.com
4. Set the ingestion endpoint.

sudo logdna-agent -s LOGDNA_LOGHOST=logs.us-south.logging.cloud.ibm.com
5. (Optional) Define any additional log paths to be monitored. Run the following command:

sudo logdna-agent -d <PATH_TO_LOG_FOLDERS>

Where <PATH_TO_LOG_FOLDERS> is the path where logs are saved on your system. By default, /var/log is monitored.
6. (Optional) Configure the logging agent to tag your hosts. Run the following command:

sudo logdna-agent -t TAG1,TAG2

Tags must be separated by commas and without any spaces between the comma and tag name.
7. Update the logging agent with your changes. Run the following command:
If you use the Lite (free) pricing plan for IBM Cloud Object Storage, you cannot use an SQL query to confirm the creation of flow log objects in the Object Storage bucket without purchasing the Standard pricing plan. As an alternative, you can access generated flow log object files directly from the Object Storage bucket to verify that the files were created successfully and that the objects are being generated.

To view generated flow log files from the Object Storage bucket, follow these steps:



1. To ensure that the flow logs are being captured, check the Object Storage bucket.
2. Navigate in the folder to find your VPC or the resource where you configured monitoring.
3. Drill-down into the resource to find the network resource that you are monitoring.

If configured correctly, you should see several gzip files for each log entry.



Using SQL queries, you can flatten and view these objects or any other type of view of all these files. In our case, we want to ensure that data is being captured. Download one of the files to view its contents to ensure that traffic is being captured.
sudo apt-get update

sudo apt-get install logging-agent < "/dev/null"
2. Set the ingestion key that the logging agent will use to forward logs to the IBM Log Analysis instance.

sudo logging-agent -k <INGESTION_KEY>

where <INGESTION_KEY> contains the ingestion key for the IBM Log Analysis instance where the logs will be forwarded.

You can retrieve the ingestion key using the [IBM Cloud console](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-ingestion_keyibm_cloud_ui), or by the [IBM Cloud CLI](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-ingestion_keyingestion_key_cli).
3. Set the authentication endpoint. The logging agent uses this host to authenticate and get the token to forward logs.

sudo logging-agent -s LOGDNA_APIHOST=api.us-south.logging.cloud.ibm.com
4. Set the ingestion endpoint.

sudo logging-agent -s LOGDNA_LOGHOST=logs.us-south.logging.cloud.ibm.com
5. (Optional) Define any additional log paths to be monitored. Run the following command:

sudo logging-agent -d <PATH_TO_LOG_FOLDERS>

Where <PATH_TO_LOG_FOLDERS> is the path where logs are saved on your system. By default, /var/log is monitored.
6. (Optional) Configure the logging agent to tag your hosts. Run the following command:

sudo logging-agent -t TAG1,TAG2

Tags must be separated by commas and without any spaces between the comma and tag name.
7. Update the logging agent with your changes. Run the following command:

sudo update-rc.d logging-agent defaults
8. Start the logging agent. Run the following command:
cf env jc-hello-world | awk '/VCAP_SERVICES/{flag=1} /^}/{flag=0} flag' | sed 's/"VCAP_SERVICES"://' > vcap.json





 Output 

Review how your application reads the system and user-defined variables. You will need to ensure that your application continues to have access to these variables.





 Migration steps 

Your application uses these variables for various purposes. You will need to ensure that the small pieces of code that gather these variables for use inside your application are modified to continue to get this information in your new compute service.







 Application access, routes, domains 

You probably know how you access any application you have in production, but if needed, this step will help you summarize how your applications are mapped and accessed.

You also need to know if your application is directly accessed using DNS, or using a load balancer such as IBM Cloud Internet Service or Akamai. For more information, see the following:



* [Cloud Foundry Routes and Domains](https://docs.cloudfoundry.org/devguide/deploy-apps/routes-domains.html)
* [Adding and using a custom domain](https://cloud.ibm.com/docs/cloud-foundry-public?topic=cloud-foundry-public-custom-domains)
* [Managing Cloud Foundry routes and domains](https://www.ibm.com/support/pages/commands-manage-cloud-foundry-applications-domains-or-routes)
* [Multi-Zone Region (MZR) overview](https://cloud.ibm.com/docs/loadbalancer-service?topic=loadbalancer-service-multi-zone-region-mzr-overview)





 Commands 

Commands to pull the needed information:

The following command lists all the routes that are used to access your applications.

ibmcloud cf routes



The following command lists all the domains that are being used in those potential routes.

ibmcloud cf domains

If you use the IBM Cloud Internet Service for routing, or to check, install the CIS command plug-in using the following command.

ibmcloud plugin install cis
calicoctl apply -f /log-denied-packets.yaml --config=<filepath>/calicoctl.cfg



3. Generate log entries by sending requests from your system IP to the NLB IP. These request packets are logged before they are denied.

curl --connect-timeout 10 <loadbalancer_IP>:80
4. Check for log entries that are written to the /var/log/syslog path. The log entry looks similar to the following.

Sep 5 14:34:40 <worker_hostname> kernel: [158271.044316] calico-packet: IN=eth1 OUT= MAC=08:00:27:d5:4e:57:0a:00:27:00:00:00:08:00 SRC=192.XXX.XX.X DST=192.XXX.XX.XX LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52866 DF PROTO=TCP SPT=42962 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0



Nice! You set up logging so that blocked traffic can be monitored more easily.

If you want to clean up the blocklist and the log policies:



1. Clean up the blocklist policy.



* Linux:

calicoctl delete GlobalNetworkPolicy blocklist
* Windows and OS X:

calicoctl delete GlobalNetworkPolicy blocklist --config=filepath/calicoctl.cfg



2. Clean up the log policy.



* Linux:

calicoctl delete GlobalNetworkPolicy log-denied-packets
* Windows and OS X:

calicoctl delete GlobalNetworkPolicy log-denied-packets --config=filepath/calicoctl.cfg









 What's next? 



* Read more about [controlling traffic with network policies](https://cloud.ibm.com/docs/containers?topic=containers-network_policies).
Cloud Type ibm_ctype The cloud type is public, dedicated, or local. 
 Location ibm_location The location of the monitored resource, which can be a region, data center, or global. 
 Resource ibm_resource The resource that is measured by the service, which is typically an identifying name or GUID. 
 Resource Type ibm_resource_type The type of the resource that is measured by the service. 
 Scope ibm_scope The scope is the account, organization, or space GUID associated with the metric. 
 Service name ibm_service_name The name of the service that is generating this metric. 







 More attributes 

The following attributes are available for one or more attributes described in the previous tables. See the individual metrics for options.



Table 4. Other atributes

 Attribute Attribute Name Attribute Description 

 target type ibm_atracker_target_type The target type destination of the event. Valid values are: cloud_object_storage, logdna, or event_streams. 
 reason ibm_atracker_reason_code The reason for the failure of an event delivery to its destination.
selector: ibm.role=='worker_public'
order: 300
types:
- Ingress
Show more
2. Apply the policy.



* Linux:

calicoctl apply -f /log-denied-packets.yaml
* Windows and OS X:

calicoctl apply -f /log-denied-packets.yaml --config=<filepath>/calicoctl.cfg



3. Generate log entries by sending requests from your system IP to the NLB IP. These request packets are logged before they are denied.

curl --connect-timeout 10 <loadbalancer_IP>:80
4. Check for log entries that are written to the /var/log/syslog path. The log entry looks similar to the following.

Sep 5 14:34:40 <worker_hostname> kernel: [158271.044316] calico-packet: IN=eth1 OUT= MAC=08:00:27:d5:4e:57:0a:00:27:00:00:00:08:00 SRC=192.XXX.XX.X DST=192.XXX.XX.XX LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=52866 DF PROTO=TCP SPT=42962 DPT=22 WINDOW=29200 RES=0x00 SYN URGP=0



Nice! You set up logging so that blocked traffic can be monitored more easily.

If you want to clean up the blocklist and the log policies:



1. Clean up the blocklist policy.



* Linux:

calicoctl delete GlobalNetworkPolicy blocklist
* Windows and OS X:

calicoctl delete GlobalNetworkPolicy blocklist --config=filepath/calicoctl.cfg



2. Clean up the log policy.



* Linux:

calicoctl delete GlobalNetworkPolicy log-denied-packets
* Windows and OS X:
[Shows the new label dialog.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-add-label.png)

Figure 15. New label dialog
10. Click the new footnote label that you added, and then label the footnote on the page with the label. Click Submit page to move to the next page.

Zoom

![Shows the footnote that is being labeled in the SDU tool.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-label-footnote.png)

Figure 16. A footnote is being labeled in the Smart Document Understanding tool
11. Repeat this process to label and submit multiple pages.

For most pages, the content includes a subtitle, a footnote, and the bulk of the content on the page remains labeled as text.

Zoom

![Shows the subtitle, footnote, and text labels that are being applied to a page in the SDU tool.](https://cloud.ibm.com/docs-content/v1/content/f16b1f78a92498247223b337bd9303c2758b9ba8/discovery-data/images/tut-sdu-typical-labels.png)

Figure 17. Subtitle, footnote, and text labels are being applied

As you label and submit pages, the model learns from the annotations that you make. Gradually, the labels that are applied automatically become accurate and don't require any adjustments.

If the tool becomes overzealous in the application of labels, apply the text label to sections of standard text to correct it.
12.
* Both public and private network (not available for Databases for MongoDB)





 Provisioning from the CLI and API 

Service Endpoints are enabled through an optional parameter when you provision through the CLI and API. Provisioning is handled by the Resource Controller, and you pass the service-endpoints parameter one of the options public, private, or public-and-private.

ibmcloud resource service-instance-create <service-name> --service-endpoints <endpoint-type>

For more information, see the [Provisioning](https://cloud.ibm.com/docs/cloud-databases?topic=cloud-databases-provisioning) page.

Cloud Databases deployments except Databases for MongoDB allow for both public and private networking to be enabled at the same time.







 Changing Service Endpoints 

After you have a deployment, it is possible to change your public and private service endpoints configuration, except for Databases for MongoDB.

In the Overview tab of your deployment's dashboard, go to the Endpoints section. You can toggle which types of connections are available to your deployment.

You can use the [ibmcloud resource service-instance-update](https://cloud.ibm.com/docs/cli?topic=cli-ibmcloud_commands_resourceibmcloud_resource_service_instance_update) command in the CLI, specifying the endpoint with the --service-endpoints flag.

ibmcloud resource service-instance-update <service-name> --service-endpoints <endpoint-type>

Or you can use the [Resource Controller API](https://cloud.ibm.com/apidocs/resource-controller), with a PATCH request to the [/resource_instances/{id}](https://cloud.ibm.com/apidocs/resource-controllerupdate-a-resource-instance) endpoint.

Changing the type of endpoints available on your deployment does not cause any downtime from a database perspective. However, if you disable an endpoint that is being used by you or your applications, those connections are dropped.





 Credentials for Private Endpoints
* Provide the workspace ID that you generated in the [Creating a workspace](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-creating-workspace&interface=api) task, for example, us-south.workspace.Terraform-Schematics-Python-Workspace.b3bbc9f5.
* Export your IBM Cloud API key by using the following command:



export IBMCLOUD_API_KEY =”<ibm-cloud-api-key>”



* Run the following curl command to create a refresh token:



curl -X POST "https://iam.cloud.ibm.com/identity/token" -H "Content-Type: application/x-www-form-urlencoded" -d "grant_type=urn:ibm:params:oauth:grant-type:apikey&apikey=$IBMCLOUD_API_KEY" -u bx:bx
5. Run the Python script by using python3 <python-file-name> to apply a plan in the IBM Cloud.
6. You get an activity ID in response if the parameters passed as part of the request are valid. You should see the plan being applied in the Schematics workspace that you created in the IBM Cloud console. If you don’t get a successful response, the error response contains the errors that you need to resolve. Resolve those errors and run the script until you are able to get a valid response and apply a plan.
7. If you want to check the logs of the action, see [Retrieving action logs with Schematics API](https://cloud.ibm.com/docs/hpc-spectrum-scale?topic=hpc-spectrum-scale-retrieve-action-logs) to retrieve the logs. The response contains the status of the action, and it appears in either a COMPLETED or FAILED state.
Logging with Linux VPC server instances 

Use the IBM® Log Analysis service to monitor and manage logs from a Linux VPC server instance in a centralized logging system on the IBM Cloud.

These instructions are for Ubuntu Linux systems but can be used for other Linux systems.

You can collect and monitor system and application logs.

By default, the logging agent for Ubuntu monitors log files in the /var/log directory. For example, the Ubuntu system log (/var/log/syslog) is monitored by default.

On the IBM Cloud, configure an Ubuntu server to forward logs to an IBM Log Analysis instance by completing the following steps:



1. Provision a VPC running Ubuntu Linux.
2. Provision an instance of the IBM Log Analysis service.
3. Configure the logging agent in the Ubuntu server.
4. Optionally, add additional directories to be monitored by the agent.



Zoom

![Component overview on the IBM Cloud](https://cloud.ibm.com/docs-content/v1/content/f61dbc5d84bf0c89e477a05cab04c72f429e1b7a/log-analysis/images/Log-Analysis-03-Ubuntu.svg)

Figure 1. Component overview on the IBM Cloud

In this tutorial, you will learn how to configure an Ubuntu server to forward logs to an IBM Log Analysis instance.



 Before you begin 

Read about IBM Log Analysis. For more information, see [About](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-getting-startedgetting-started-ov).

Work in a [supported region](https://cloud.ibm.com/docs/log-analysis?topic=log-analysis-regions).

You can send data from an Ubuntu instance that is located in the same region as your logging instance, in a different region, or not in the IBM Cloud.
* [Total number of events that failed to send to the storage target](https://cloud.ibm.com/docs/atracker?topic=atracker-monitoring_metricsibm_atracker_failed_events_by_target)
* [Total number of events that are discarded](https://cloud.ibm.com/docs/atracker?topic=atracker-monitoring_metricsibm_atracker_discarded_events)





 Total number of events successfully sent to the storage target 

Total number of events that are successfully sent to the storage target.



Table 1. Total number of events successfully sent to the storage target

 Metadata Description 

 Metric Name ibm_atracker_successful_events_by_target 
 Metric Type counter 
 Value Type none 
 Segment By target type 







 Total number of events that failed to send to the storage target 

Total number of events that failed to send to the storage target.



Table 2. Total number of events that failed to send to the storage target

 Metadata Description 

 Metric Name ibm_atracker_bad_config_discarded_events 
 Metric Type counter 
 Value Type none 
 Segment By target type, reason 







 Total number of events that are discarded 

Total number of events that are discarded.



Table 3. Total number of events that are discarded

 Metadata Description 

 Metric Name ibm_atracker_failed_events_by_target 
 Metric Type counter 
 Value Type none 
 Segment By target type, reason 







 Global Attributes 

The following attributes are available for all metrics.



Table 3. Global atributes

 Attribute Attribute Name Attribute Description 

 Cloud Type ibm_ctype The cloud type is public, dedicated, or local. 
 Location ibm_location The location of the monitored resource, which can be a region, data center, or global. 
 Resource ibm_resource The resource that is measured by the service, which is typically an identifying name or GUID. 
 Resource Type ibm_resource_type The type of the resource that is measured by the service. 
 Scope ibm_scope The scope is the account, organization, or space GUID associated with the metric.
You can also expand your graph view by clicking the arrows icon.

Netezza Performance Server separates short queries from longer-running queries. Short queries are those queries that run in two seconds or less. For “lightly loaded” systems, the queues might be zero if no queries are waiting, or small. As the system becomes busy with concurrent short or long queries, these queues can show a larger number of waiting snippets and jobs.

Netezza Performance Server reserves resources specifically for short queries so that it can run and complete those short queries even while it is busy running longer queries.





 Resource allocation 

The resource allocation charts display the percentage of current system resources that are consumed by the 4 most popular resource groups.

The first chart monitors the percentage of system resources that are consumed by the resource groups over the time. The other two pie charts show the current resource allocation and current run jobs of the most popular resource groups.







 Infrastructure 

On the Infrastructure page, you can view information about your system, system disk status, SPUs, and storage utilization.
The [CI toolchain](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-ci-pipeline) provides pipelines that scan your code and build artifacts, catch security issues early, and prevent those issues from being introduced into the product configuration, while collecting evidence that can be used for audits. The [CD toolchain](https://cloud.ibm.com/docs/devsecops?topic=devsecops-cd-devsecops-cd-pipeline) checks for required evidence, generates a change request document with an evidence summary and description of changes, and gates deployment based on compliance with your organization’s security policies. And, because new vulnerabilities that were not previously known might always be discovered, you can leverage the [CC toolchain](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops-cc-pipeline) to continuously assess app security and compliance posture for code that is already in production.

To learn more, see the [DevSecOps documentation](https://cloud.ibm.com/docs/devsecops?topic=devsecops-devsecops_intro).







 Assessing and maintaining compliance of enterprise workloads 

Although IBM Cloud reduces the time and complexity of setting up a compliant enterprise application, you still need to ensure that you're maintaining compliance. To do so, you can use the Security and Compliance Center to run [automatic evaluations](https://cloud.ibm.com/docs/security-compliance?topic=security-compliance-scan-resources) on your resource configurations. The evaluation results are provided in the dashboard of the Security and Compliance Center or you can get notified of changes. You are able to quickly assess the risk to your organization, fix issues, and generate reports so that you're always audit-ready.

Zoom

!
Viewing VMware vSphere instances 

View the summary and detailed information of the VMware vSphere® instances that are provisioned for different user accounts.



 Procedure to view summary for VMware vSphere instances 

To view a summary of all the VMware vSphere instances that are provisioned for a user account, complete the following steps:



1. From the IBM Cloud® for VMware Solutions console, click Resources > vSphere from the left navigation pane.
2. From the console banner, click the Avatar icon ![Avatar icon](https://cloud.ibm.com/docs-content/v1/content/5bdb37c0149145723c5437e8176651928565e722/icons/i-avatar-icon.svg), and then click the Account field. Select the user account that you want to check instances for.
3. In the vSphere table, view the list of instances that are provisioned in the selected user account.





Table 1. VMware vSphere instance items

 Item Description 

 Name The name of the instance. 
 Location The IBM Cloud data center where the instance is hosted. 
 Creation time The date and time when the instance was created. 
 Status The status of the instance. 



The instance can have different statuses.



Table 2. VMware vSphere instances status descriptions

 Status Description 

 Creating The instance is being created. 
 Building The instance is being configured. 
 Available The instance is ready to use. 
 Modifying The instance is being modified. 
 Failed The creation, configuration, or modification process failed. 
 Deleting The instance is being deleted. 
 Deletion error An error occurred when the instance was being deleted. 
 Deleted The instance is deleted. 







 Procedure to view details for VMware vSphere instances 

To view the property details of an instance, complete the following steps.



1. In the vSphere table, click an instance name.
2. On the Summary tab, view the details for the instance.



Table 3. VMware vSphere instance properties

 Property Description