vania-janet/multiturn-rag-retrieval-data huggingface dataset:
(https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-troubleshooting-virtual-servertroubleshoot-vs-portal-shows-server-disconnected-but-running)





For more help, see [Getting help and support](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-gettinghelp).





 What do I do if my bare metal server is down? 

If your bare metal server is down, use the following steps.



1. Remotely access the server by using [IPMI](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-bm-faqhow-do-i-use-ipmi). Depending on the output, you might need to create a [support case](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-gettinghelp).



If your server does not power on, you might need to start your server in rescue mode. For more information, see [Rescue mode](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-launching-rescue)

For more help, see [Getting help and support](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-gettinghelp).





 What is the difference between "Boot from image" and "Load from image"? 

Boot from image and Load from image both use image templates, which are applied to a device to either replace or supplement the existing operating system to help remedy an existing issue. The main difference between the Boot and Load process is the type of image that is used. When either the boot or load from image process is used, make sure that you back up all data that you might want to recover.



* Boot from image starts a device by using an ISO supplied by IBM Cloud® for system recovery or an ISO that was uploaded by using the Import image feature in the IBM Cloud console. The ISO might be a clean version of the device operating system or a recovery disk that is used in an attempt to remedy an issue with the device.
FAQs for virtual server instances 



 Can a vNIC have both a floating IP and private IP address? 

Yes, a vNIC on a virtual server instance has a private IP and can be attached to floating IP.





 Instance1 has two vNICs, called vNIC1 and vNIC2. Can these two vNICs be attached to the same subnet? 

Yes, you can attach multiple network interfaces of an instance to the same subnet.





 Can an instance be created without a subnet, such as with only a floating IP address? 

No, a virtual server instance must be provisioned in a subnet.





 Can an instance be attached to multiple VPCs? 

No, a virtual server instance can be provisioned in only one VPC.





 During floating IP assignment in a VPC, a customer must specify the network interface of the instance, is that correct? 

Yes. Initially, assigning the floating IP to the primary network interface of a server helps establish the data path. Later, you can associate the floating IP to a different network interface if you want. Alternatively, you can manually configure routing for the interface in the guest operating system. For more information, see [Adding or editing network interfaces](https://cloud.ibm.com/docs/vpc?topic=vpc-using-instance-vnicsediting-network-interfaces).





 Imagine that instance1 in a VPC has only vNIC1 and it is attached to Subnet1. Subnet1 is attached to a Public Gateway (PGW). Can a customer still assign a floating IP to instance1? 

Yes, a server can be on a subnet that is attached to a public gateway and also have a floating IP. The assignment of floating IP to an instance is not related to whether a public gateway is attached to the subnet. A floating IP associated to an instance takes precedence over the public gateway that is attached to the subnet.





 What regions are available?
Managing bare metal servers 

You can cancel, stop, and restart your bare metal server.



  Before you begin 



*  Go to your console device menu. For more information, see [Navigating to devices](https://cloud.ibm.com/docs/bare-metal?topic=virtual-servers-navigating-devices).
*  Make sure that you have any necessary account permissions and device access. Only the account owner, or a user with the Manage users Classic infrastructure permission, can adjust the permissions.



For more information about permissions, see [Classic infrastructure permissions](https://cloud.ibm.com/docs/iam?topic=iam-infrapermissioninfrapermission) and [Managing device access](https://cloud.ibm.com/docs/vsi?topic=virtual-servers-managing-device-access).





  Deleting the server instance 

At any time, you can delete the bare metal server instance if you no longer need the instance and do not want to be charged.



1.  Go to the Devices menu and highlight the device to be deleted.
2.  Click Actions and select Cancel server.







  Powering off a server 

You can create a server in advance to prepare for usage. After you create the server, power it off to make sure that nothing gets changed and that no one can access it. When you're ready, you can power on the server and have it ready in minutes.



1.  Go to the Devices menu and highlight the device to be powered off or on.
2.  Click Actions and select Power On / Off.



You are billed for the server while it is powered off. If the server is powered off, it stays powered off and you need to be manually power it on.
[Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/data-virtualization?topic=data-virtualization-activity-tracker) 







 Classic services 

The following table lists services that send auditing events:



Table 2. List of classic services

 Service CRN service name Activity Tracker hosted event search offering Events 

 [IBM Cloud® Virtual Servers (Classic)](https://cloud.ibm.com/docs/vsi?topic=virtual-servers-about-virtual-serversabout-virtual-servers) audit-log ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/vsi?topic=virtual-servers-at_eventsat_events) 
 [IBM Cloud® Bare Metal Servers (Classic)](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-about-bmabout-bm) audit-log ![Checkmark](https://cloud.ibm.com/docs-content/v1/content/6f3c07c34058f637b1c86044ca4f9b24214330a7/activity-tracker//images/checkmark-icon.svg) [Location-based events](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-bm-at-eventsbm-at-events) 







 Compute serverless services 

The following table lists serverless compute services that send auditing events:



Table 3. List of serverless compute services
The more servers in partial failover mode, the greater the impact of latency on the customer routing and security functions.





 Continuously available class 

For [continuously available class](https://cloud.ibm.com/docs/overview?topic=overview-understanding-drplan-objectives), because of the short recovery time, the best-suited DR solution is to adopt application clustering and an application-aware data replication to have a continuously available infrastructure ready to be activated in the case of an emergency.

Application-aware data replication, such as database replication, allows to have an application-consistent copy of data that can be immediately used by the alternative compute instance. The alternative compute instance for DR is already created and always-on.

Compute instances that make up the application or database cluster must be deployed over different fault domains to guarantee that at least one instance will survive an unavailability event.

The DR test will be completed on a new set of on-demand, dynamically provisioned compute instances, that are duplicated by using snapshot functions. This validates the DR runbooks that will be run on the real alternative passive DR compute instances.

Zoom

![Continuously available class network architecture](https://cloud.ibm.com/docs-content/v1/content/adcecdbe4e86955f88aac7f3c1e7ec3ff44992ef/overview/images/BCDR-Architecture-Diagram-image12.svg)

Figure 1. Continuously available class network architecture

On the DR side, you can see a solution where components are deployed on the cloud infrastructure and strictly connected with the correspondent that is deployed on premises. The components on cloud are passive in the sense that they do not perform active production workload, but they are receiving updates from the on premises components that are performing all of the productive workload.

On the HA+DR side, you see a typical 3-site deploy, where two sites are performing productive workload (active-active couples), while the third site has passive components that just receive the updates from the two productive sites.

In both cases, you see an “on-demand” environment that is dynamically created and used solely to run DR test.
Provisioning public instances 

You can provision your public virtual server instances through the IBM Cloud® console.

For the IBM Cloud console, you must have an upgraded account to order virtual servers. For more information about upgrading your account, see [Pay-As-You-Go account](https://cloud.ibm.com/docs/account?topic=account-accountspaygo).



 Before you begin 

Before you begin, review the following prerequisites.



1. Review the deployment options available to you. For more information, see [Public virtual servers](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-about-public-virtual-serversabout-public-virtual-servers).
2. Review virtual server instance capacity considerations. For more information, see [Resource considerations for virtual server instances](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-capacity-considerationscapacity-considerations).
3. Open the virtual server instance page from the IBM Cloud console.







 Provisioning a public virtual server instance 

To provision a virtual server instance, you need to consider the following information.

You must be logged in to see all available options.



 Public instance 



Table 1. Public instance options

 Field Details 

 Billing Depending on your virtual server instance, you can select either hourly or monthly billing. The primary difference, other than cost, is that hourly instances don't have an included bandwidth allocation. At the end of a billing period, the bandwidth usage and the number of hours that each instance was active on the account are calculated. For example, if you cancel an hourly instance after 10 days, you pay for only the hours for those 10 days. If you cancel a monthly instance after 10 days, you pay for the whole month. If you're interested in the suspend billing feature, which is only available on hourly instances, see [About suspend billing](https://cloud.ibm.com/docs/virtual-servers?topic=virtual-servers-requirements). 
 Hostname Can contain labels that are made of alphanumeric characters and dashes, which are separated by periods.
.config("fs.stocator.cos.impl", "com.ibm.stocator.fs.cos.COSAPIClient") 
.getOrCreate()

print(" Got a spark session, listing all tables")
spark.sql('show tables').show()

spark.stop()
Show more

Prepare a JSON file to start that program, as in the following example (listTablesExample.json):

{
"application_details": {
"application": "cos://<your-bucket>.<cos-region>/listTablesExample.py",
"arguments": ["<Data-Engine-instance-CRN>", "<API-key-to-access-data-engine-instance>"],
"conf": {
"ae.spark.executor.count":"1",
"ae.spark.autoscale.enable":"false",
"spark.hadoop.fs.cos.<cos-region>.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-region>.iam.api.key": "<API-key-to-access-python-file>",
"spark.app.name": "DataEngineHiveAccess"
}
}
}

Start the application by using a curl command, as in the following example:

curl -X POST https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<GUID of Analytic Engine>/spark_applications --header "Authorization: Bearer $TOKEN" -H "content-type: application/json" -d @listTablesExample.json





 Apache Spark Data Engine integration 

For self-hosted Apache Spark installations use the following instructions.



1.
Gets the details of a given Spark application.

ibmcloud analytics-engine-v3 spark-app show [--instance-id INSTANCE-ID] --app-id APP-ID



 Command options 

-i, --instance-id (string)
: Identifier of the instance to which the application belongs. When specified, it overrides the target instance ID if it was set. Required, if the target instance was not set.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.

--app-id (string)
: Identifier of the application for which details are requested. Required.

The value must match the regular expression /u0008[0-9a-f]{8}u0008-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-u0008[0-9a-f]{12}u0008/.





 Examples 

ibmcloud analytics-engine-v3 spark-app show --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b --output json

ibmcloud analytics-engine-v3 spark-app show --instance-id=e64c907a-e82f-46fd-addc-ccfafbd28b09 --app-id=ff48cc19-0e7e-4627-aac6-0b4ad080397b --output json





 Example output 

The output is as follows:

{
"application_details" : {
"application" : "/opt/ibm/spark/examples/src/main/python/wordcount.py",
tablesDF=spark.sql("SHOW TABLES")
  tablesDF.show()
  statesDF=spark.sql("SELECT * from COUNTRIESCAPITALS");
  statesDF.show()

def main():
  if __name__ == '__main__':
    if len (sys.argv) < 2:
        exit(1)
    else:
        crn = sys.argv[1]
        apikey = sys.argv[2]
       session_builder = SparkSessionWithDataengine.enableDataengine(crn, apikey, "public", "/opt/ibm/connectors/data-engine/hms-client")
        spark = session_builder.appName("Spark DataEngine integration test").getOrCreate()
      sc = spark.sparkContext
      dataengine_table_test (spark,sc)

if __name__ == '__main__':
  main()
Show more

The following is the payload for the convenience dataengine-job-convenience_api.py:

Example:

Enter:

{
"application_details": {
"conf": {
"spark.hadoop.fs.cos.ALIAS NAME.endpoint": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.access.key": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.secret.key": "CHANGEME"
}
"application": "cos://mybucket.ALIAS NAME/dataengine-job-convenience_api.py",
"arguments": ["<CHANGEME-CRN-DATA-ENGINE-INSTANCE>","<APIKEY-WITH-ACCESS-TO-DATA-ENGINE-INSTANCE>"]
    }
}

Parameter values:

ALIAS NAME: specify the data engine endpoint for your region.
You need this value to perform operations such as getting the state of the application, retrieving the details of the application, or deleting the application.



6. Submit the Spark application:

curl -X POST https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications --header "Authorization: Bearer $token" -H "content-type: application/json" -d @submit-spark-app.json

Sample response:

{
"id": "87e63712-a823-4aa1-9f6e-7291d4e5a113",
"state": "accepted"
}
7. If forward logging was enabled for your instance, you can view the application output in the platform logs that are forwarded to IBM Log Analysis. For details, see [Configuring and viewing logs](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-viewing-logs).









 Passing the Spark configuration to an application 

You can use the "conf" section in the payload to pass the Spark application configuration. If you specified Spark configurations at the instance level, those are inherited by the Spark applications run on the instance, but can be overridden at the time a Spark application is submitted by including the "conf" section in the payload.

See [Spark configuration in Analytics Engine Serverless](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config).





 Parameters for submitting Spark applications 

The following table lists the mapping between the spark-submit command parameters and their equivalent to be passed to the "application_details" section of the Spark application submission REST API payload.



Table 1. Mapping between the spark-submit command parameters and their equivalents passed to the payload

 spark-submit command parameter Payload to the Analytics Engine Spark submission REST API 

 <application binary passed as spark-submit command parameter> application_details -> application 
 <application-arguments> application_details -> arguments 
 class application_details -> class
* IBM Analytics Engine instance ID: 4308f7a6-2cfd-4d7e-b7a7-678e71a27957
* Spark application ID: 5438585f-6413-4bb2-9251-8162d34f3dc3
* Sources for which log forwarding is enabled: spark-driver and spark-executor
* Associated tags: prod and us-south



The following image shows you an example of the returned search results in the Log Analysis dashboard. At the bottom of the image you can see the search entry field where you enter your search queries. In the example, the application ID 5438585f-6413-4bb2-9251-8162d34f3dc3 and the source spark-driver were entered.

Zoom

![Shows the search window for on the Log Analysis dashboard.](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/search-window-log-analysis.png)

Figure 1. Log Analysis dashboard displaying search results

Examples of searches:



* To retrieve the logs for all Spark applications submitted in a specific IBM Analytics Engine instance, pass the instance ID in the search window:

4308f7a6-2cfd-4d7e-b7a7-678e71a27957
* To fetch the logs for a specific Spark application, you can pass:



* A combination of the IBM Analytics Engine instance ID and the Spark application ID:

4308f7a6-2cfd-4d7e-b7a7-678e71a27957 5438585f-6413-4bb2-9251-8162d34f3dc3
* Only the application ID:

5438585f-6413-4bb2-9251-8162d34f3dc3



* To search the Spark master logs of an application, you can pass:
"spark.hadoop.fs.cos.ALIAS NAME.endpoint": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.access.key": "CHANGEME",
"spark.hadoop.fs.cos.ALIAS NAME.secret.key": "CHANGEME"
        },
  "application": "cos://mybucket.ALIAS NAME/create_table_data_engine.py",
  "arguments": ["<CHANGEME-CRN-DATA-ENGINE-INSTANCE>","<APIKEY-WITH-ACCESS-TO-DATA-ENGINE-INSTANCE>"]
    }
}

Parameter values:

ALIAS NAME: specify the data engine endpoint for your region. For more information on the currently supported data engine endpoints, see [Data engine endpoints](https://cloud.ibm.com/docs/sql-query?topic=sql-query-overviewendpoints).

Make sure that you select the standard aliases.
{: important}
Show more







 Reading data from table by passing full list of Data Engine parameters 

You can read the data from the metastore table using the SQL querry.

Read the data from the table by using the Spark SQL in the following application called select_query_data_engine.py:

from pyspark.sql import SparkSession
import time

def init_spark():
spark = SparkSession.builder.appName("dataengine-table-select-test").getOrCreate()
sc = spark.sparkContext
return spark,sc

def select_query_data_engine(spark,sc):
tablesDF=spark.sql("SHOW TABLES")
tablesDF.show()
statesDF=spark.sql("SELECT * from COUNTRIESCAPITALS");
statesDF.show()

def main():
spark,sc = init_spark()
select_query_data_engine(spark,sc)

if __name__ == '__main__':
main()
"spark.hadoop.fs.cos.<cos-reference-name>.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud",
"spark.hadoop.fs.cos.<cos-reference-name>.access.key": "<access_key>",
"spark.hadoop.fs.cos.<cos-reference-name>.secret.key": "<secret_key>",
"spark.app.name": "MySparkApp"
}
}
}

Note:



* You can pass Spark application configuration values through the "conf" section in the payload. See [Parameters for submitting Spark applications](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-spark-app-rest-apispark-submit-parms) for more details.
* <cos-reference-name> in the "conf" section of the sample payload is any name given to your IBM Cloud Object Storage instance, which you are referencing in the URL in the "application" parameter. See [Understanding the Object Storage credentials](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cos-credentials-in-iae-serverless).
* It might take approximately a minute to submit the Spark application. Make sure to set sufficient timeout in the client code.
* Make a note of the "id" returned in the response. You need this value to perform operations such as getting the state of the application, retrieving the details of the application, or deleting the application.



6. Submit the Spark application:

curl -X POST https://api.us-south.ae.cloud.ibm.com/v3/analytics_engines/<instance_id>/spark_applications --header "Authorization: Bearer $token" -H "content-type: application/json" -d @submit-spark-app.json

Sample response:

{
seniors = spark.sql("SELECT empTable.NAME FROM empTable WHERE empTable.BAND >= 6")
print("Hello 3", seniors)
seniors.show()

def main():
spark,sc = init_spark()
read_employees(spark,sc)

if __name__ == '__main__':
main()
Show more

Example of the CSV file called employees.csv that is read by the application called read-employees.py:

NAME,BAND,DEPT
Abraham,8,EG
Barack,6,RT
Clinton,5,GG
Hoover,4,FF
Kennedy,7,NN
Truman,3,TT

To run the application called read-employees.py that reads data from employees.csv POST the following JSON payload script called read-employees-submit.json. Insert the Object Storage bucket and service name where the CSV file is located, modify the endpoint path and insert your access key and secret key.

{
"application_details": {
"application": "cos://cosbucketname.cosservicename/read-employees.py",
"conf": {
"spark.hadoop.fs.cos.cosservicename.endpoint": "https://s3.direct.us-south.cloud-object-storage.appdomain.cloud/<CHANGME-according-to-instance>",
"spark.hadoop.fs.cos.cosservicename.access.key": "<CHANGEME>",
"spark.hadoop.fs.cos.cosservicename.secret.key": "<CHANGEME>"
}
}
}





 Reading a CSV file from Object Storage using IAM API Key or different HMAC credentials 

The following code samples show you how to create a Python script that reads data from a CSV file to a Python DataFrame. Both the Python script and the CSV file are located in Object Storage.
* REGION: For applications submitted on an instance in the Dallas region, substitute the REGION with us-south and for applications submitted on an instance in the Frankfurt region, substitute the REGION in the endpoint with eu-de.
* INSTANCE_ID: Identifier of the IBM Analytics Engine instance under which the application is running.
* APPLICATION_ID: Identifier of the application for which Spark UI is accessed.
* IAM TOKEN: Ensure that you have the necessary privileges to access the service instance.



For more details on APIs, see [SparkUI server REST API documentation](https://spark.apache.org/docs/latest/monitoring.htmlrest-api).
.xClass("IbmAnalyticsEngineApi")
.arguments(new java.util.ArrayList<String>(java.util.Arrays.asList("/opt/ibm/spark/examples/src/main/resources/people.txt")))
.conf(new java.util.HashMap<String, Object>() { { put("spark.app.name", "MySparkApp"); } })
.env(new java.util.HashMap<String, Object>() { { put("SPARK_ENV_LOADED", "2"); } })
.build();

// Construct an instance of the CreateApplicationOptions model
CreateApplicationOptions createApplicationOptionsModel = new CreateApplicationOptions.Builder()
.instanceId("dc0e9889-eab2-4t9e-9441-566209499546")
.applicationDetails(applicationRequestApplicationDetailsModel)
.build();

// Invoke operation with valid options model (positive test)
Response<ApplicationResponse> response = ibmAnalyticsEngineApiService.createApplication(createApplicationOptionsModel).execute();
ApplicationResponse responseObj = response.getResult();
System.out.println(String.valueOf(responseObj));
Show more
* Retrieve all Spark applications run on a given instance:

ServiceCall<ApplicationCollection> listApplications(ListApplicationsOptions listApplicationsOptions)

Example request:

// Construct an instance of the ListApplicationsOptions model
ListApplicationsOptions listApplicationsOptionsModel = new ListApplicationsOptions.Builder()
.instanceId("dc0e9889-eab2-4t9e-9441-566209499546")
.build();