vania-janet/multiturn-rag-retrieval-data huggingface dataset:
Explore creating serverless instances and submitting applications using the CLI 

Learn how to use the IBM Analytics Engine CLI to create the services that you need to create and manage a serverless instance, and submit and monitor your Spark applications.

You create a serverless instance by selecting the IBM Analytics Engine Standard serverless plan. When a serverless instance is provisioned, an Apache Spark cluster is created, which you can customize with library packages of your choice, and is where you run your Spark applications.



 Objectives 

You will learn how to install and set up the following services and components that you will need to use the CLI:



* An IBM Cloud Object Storage instance in which your IBM Analytics Engine instance stores custom application libraries and Spark history events.
* A Object Storage bucket for application files and data files.
* An IBM Analytics Engine serverless instance. This instance is allocated compute and memory resources on demand whenever Spark workloads are deployed. When an application is not in running state, no computing resources are allocated to the instance. The price is based on the actual usage of resources consumed by the instance, billed on a per second basis.
* Logging service to help you to troubleshoot issues that might occur in the IBM Analytics Engine instance and submitted application, as well as to view any output generated by your application. When you run applications with logging enabled, logs are forwarded to an IBM Log Analysis service where they are indexed, enabling full-text search through all generated messages and convenient querying based on specific fields.







 Before you begin 

To start using the the Analytics Engine V3 CLI you need:



* An IBM Cloud® account.
* To install the IBM Cloud CLI. See [Getting started with the IBM Cloud CLI](https://cloud.ibm.com/docs/cli?topic=cli-getting-started) for the instructions to download and install the CLI.



Now you can start using the Analytics Engine V3 CLI. You must follow the instructions in steps 1 and 2 to install the required services before you start with step 3 to upload and submit Spark applications. Step 4 shows you how to create a logging instance and enable logging. Step 5 shows you how to delete an Analytics Engine instance, although this step is optional.
* Port 8443 Knox
* Port 22 SSH
* Port 9443 Ambari



* What is IBM Analytics Engine serverless?

The IBM Analytics Engine Standard serverless plan for Apache Spark offers a new consumption model using Apache Spark. An Analytics Engine serverless instance does not consume any resources when no workloads are running. When you submit Spark applications, Spark clusters are created in seconds and are spun down as soon as the applications finish running. You can develop and deploy Spark SQL, data transformation, data science, or machine learning jobs using the Spark application API.
* What are the advantages of IBM Analytics Engine serverless instances?

With IBM Analytics Engine serverless, compute and memory resources are allocated on demand when Spark workloads are deployed. When an application is not in running state, no computing resources are allocated to the IBM Analytics Engine serverless instance. Pricing is based on the actual amount of resources consumed by the instance, billed on a per second basis.
* Does the IBM Analytics Engine Standard serverless plan for Apache Spark support Hadoop?

No, currently, the IBM Analytics Engine Standard serverless plan for Apache Spark only supports Apache Spark.
* Can I change the instance home storage of a serverless instance?

No, you can't. After an instance home storage is associated with an IBM Analytics Engine serverless instance, it cannot be changed because instance home contains all instance relevant data, such as the Spark events and custom libraries. Changing instance home would result in the loss of the Spark history data and custom libraries.
* How is user management and access control managed in a serverless instance?

User management and access control of an IBM Analytics Engine serverless instance and its APIs is done through IBM Cloud® Identity and Access Management (IAM). You use IAM access policies to invite users to collaborate on your instance and grant them the necessary privileges. See [Granting permissions to users](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-grant-permissions-serverless).
* How do I define the size of the cluster to run my Spark application?

You can specify the size of the cluster either at the time the instance is created or when submitting Spark applications.
With your instance template created, you can provision a single virtual server instance, or you can provision multiple instances at the same time as part of an instance group. Optionally, you can set auto scaling policies for your instance group to dynamically add or remove virtual server instances from your group. For more information, see the following topics:



* [Bulk provisioning instances with instance groups](https://cloud.ibm.com/docs/vpc?topic=vpc-bulk-provisioning)
* [Creating an instance group for auto scaling](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-auto-scale-instance-group)
* [Managing an instance template](https://cloud.ibm.com/docs/vpc?topic=vpc-managing-instance-template)
[IBM Cloud® Bare Metal Servers (Classic)](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-about-bmabout-bm) audit-log [Location-based events](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-bm-at-eventsbm-at-events) 







 Compute serverless services 

The following table lists serverless compute services that send auditing events:



Table 2. List of serverless compute services

 Service CRN service name Events 

 [IBM Cloud® Code Engine](https://cloud.ibm.com/docs/codeengine?topic=codeengine-getting-started) codeengine [Location-based events](https://cloud.ibm.com/docs/codeengine?topic=codeengine-at_events) 







 Container services 

The following table lists container platform services that send auditing events:



Table 3. Container events

 Service CRN service name Events 

 [IBM Cloud® Container Registry](https://cloud.ibm.com/docs/Registry?topic=Registry-getting-started) container-registry [Location-based events](https://cloud.ibm.com/docs/Registry?topic=Registry-at_events) 
 [IBM Cloud® Kubernetes Service](https://cloud.ibm.com/docs/containers?topic=containers-getting-started) containers-kubernetes [Location-based events](https://cloud.ibm.com/docs/containers?topic=containers-at_events) 
 [Red Hat® OpenShift® on IBM Cloud®](https://cloud.ibm.com/docs/openshift?topic=openshift-getting-started) openshift [Location-based events](https://cloud.ibm.com/docs/openshift?topic=openshift-at_events)
These profiles have a cpu_architecture of s390x and must be used with Red Hat Enterprise Linux for s390x and SUSE Linux Enterprise Server (SLES) for s390x. For more information, see [Creating bare metal servers on VPC](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-bare-metal-servers&interface=api).

In support of s390x bare metal servers, the following enumerations have been expanded:



* Because s390x local disks are attached through Fiber Channel protocol, a value of fcp has been added to the interface_type enumeration that is returned when you [retrieve](https://cloud.ibm.com/apidocs/vpc/latestget-bare-metal-server-disk) or [list](https://cloud.ibm.com/apidocs/vpc/latestlist-bare-metal-server-disks) the disks for a bare metal server.
* Because s390x provides TCP/IP connectivity by using HiperSockets, a value of hipersocket has been added to the interface_type enumeration returned when you [retrieve](https://cloud.ibm.com/apidocs/vpc/latestget-bare-metal-server-network-interface) or [list](https://cloud.ibm.com/apidocs/vpc/latestlist-bare-metal-server-network-interfaces) the network interfaces on an s390x bare metal server. Similarly, when you [create](https://cloud.ibm.com/apidocs/vpc/latestcreate-bare-metal-server) an s390x bare metal server, or [add](https://cloud.ibm.com/apidocs/vpc/latestcreate-bare-metal-server-network-interface) a network interface to an existing s390x bare metal server, you must specify an interface_type of hipersocket. For more information, see the [IBM HiperSockets Implementation Guide](https://www.redbooks.ibm.com/abstracts/sg246816.html).
Troubleshooting serverless instances 

In this topic, you can find the answers to common questions about how to troubleshoot IBM Analytics Engine serverless instances.



  Running ae-v3 CLI commands results in "missing-required-flags-error" 

When you run ae-v3 CLI commands, you might see a "missing-required-flags-error".

For example:

ibmcloud ae-v3 spark-app status b5e12891-c160-4674-8b94-e2cc42722be3
FAILED
Error executing command::
missing-required-flags-error

The reason might be that you are using an older version of the ae-v3 CLI. Check the version that you are using and update to latest version of the plugin by running the following commands:

ibmcloud plugin list
ibmcloud plugin update analytics-engine-v3
No OS options 

No OS is an option to order IBM Cloud® Bare Metal Servers without an operating system.



 Ordering Bare Metal Servers with no OS 

Use the following steps to order a bare metal server without an OS.



1. Use the steps that are outlined in [Building a Custom Bare Metal Server](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-ordering-baremetal-server) to order your server.

Because of compatibility issues, you can't select NMVe drives if you're going to install Windows®.
2. Select No OS under Image.
3. Complete your server order.







 Changing to no OS 

You can reconfigure a server to have no OS. The reconfiguration is done through an OS reload. For more information, see [Reloading the OS](https://cloud.ibm.com/docs/bare-metal?topic=bare-metal-reloading-the-os).



1. Click Devices > Device List.
2. Select the server that you want to reconfigure with no OS.
3. Click OS Reload and enter the applicable information.







 Installing an operating system on a no OS server 

You have two methods to install operating systems on Bare Metal Servers with no operating system.



 Option 1: PXE Server 

You can set up Bare Metal Servers with no operating systems to boot and load an OS from a PXE setup. Deployment of Bare Metal Servers in a network environment with a PXE setup (usually involves running a DHCP and TFTP daemon) and configure the new servers BIOS to boot from the network adapter. For the no OS option to work properly, PXE setup needs to be in the same VLAN as the Bare Metal Servers, or DHCP forwarding needs to be used.

It might be necessary to open a support case to request that the switch ports be regrouped in basic mode for this option to work. This request is necessary because the PXE protocol doesn't require compatibility with link aggregation (LACP), which is now a standard capability to provide redundancy.
Architecture and concepts in serverless instances 

This topic shows you the architecture of IBM Analytics Engine serverless instances and describes some key concepts and definitions.



 Instance architecture 

The IBM Analytics Engine service is managed by using IBM Cloud® Identity and Access Management (IAM). As an IBM Cloud account owner, you are assigned the account administrator role.

With an IBM Cloud account, you can provision and manage your serverless Analytics Engine instance by using the:



* IBM Cloud console
* CLI
* REST API



The Analytics Engine microservices in the control plane, accessed through an API gateway handle instance creation, capacity provisioning, customization and runtime management while your Spark applications run in isolated namespaces in the data plane. Each Spark application that you submit runs in its own Spark cluster, which is a combination of Spark master and executor nodes. See [Isolation and network access](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-security-model-serverlessisolation-network-access).

Each Analytics Engine instance is associated with an IBM Cloud Object Storage instance for instance related data that is accessible by all applications that run in the instance. Currently, all Spark events are stored in this instance as well. Spark application logs are aggregated to a Log Analysis log server.

Zoom

![Shows the IBM Analytics Engine serverless instance architecture.](https://cloud.ibm.com/docs-content/v1/content/cd4be197c8921ed12923aa899ac93a8ab643c158/AnalyticsEngine/images/AE-serverless-architecture.svg)

Figure 1. Architecture flow diagram of IBM Analytics Engine





 Key concepts 

With IBM Analytics Engine serverless instances, you can spin up Apache Spark clusters as needed and customize the Spark runtime and default Spark configuration options.

The following sections describe key concepts when provisioning serverless instances.



 IBM Analytics Engine service instance 

An IBM Cloud® service is cloud extension that provides ready-for-use functionality, such as database, messaging, and web software for running code, or application management or monitoring capabilities.
Managing serverless instances using the IBM Cloud console 

You can manage your severless instance by:



* Changing configuration settings, for example, to include library add-ons or to configure instance home after you created the instance.
* Monitoring the status of submitted applications and kernels created in the instance.





 Console configuration tab 

You can view and edit the current configuration settings for your IBM Analytics Engine serverless instance from the IBM Cloud® Resource list.



1. Access the [IBM Cloud® Resource list](https://test.cloud.ibm.com/resources).
2. Click Services and software, find your IBM Analytics Engine serverless instance and click the instance to see the details.
3. Click Manage > Configuration to view:



* The runtime. Currently, you can only select the Default Spark runtime which includes the geospatial, data skipping and Parquet encryption packages.
* The instance home volume to add an instance home or change the access credentials of an existing instance home



* You can set instance home after you created your IBM Analytics Engine serverless instance. Instance home must be associated with an IBM Cloud Object Storage instance. You can choose an instance:



* In your account by selecting it from the list
* From another account. For this instance, you need to enter:



* The GUID of the IBM Cloud Object Storage instance
* The endpoint
* The region
* The HMAC access and secret key





* You can change the access credentials of an existing instance home volume. For this instance, you need to enter:



* The new HMAC access and secret key





For details on how to access Object Storage, see [Using IBM Object Storage as the instance home volume](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-cos-serverless).
* The default Spark configuration options to override configuration settings.

For a list of the default Spark configurations set for serverless instances, see [Default Spark configurations](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-serverless-architecture-conceptsdefault-spark-config).
* [Stand-alone Enqueue Server](https://help.sap.com/saphelp_snc700_ehp01/helpdata/en/f9/e2350eca7f4a109eb0a7bc63135e27/frameset.htm) - Part of the central instance (ABAP or Java) that manages the SAP locks. In combination with the enqueue replication server, this single point-of-failure can be made into a high availability solution.
* ABAP Central services instance (ASCS instance) - Contains the ABAP message server and the stand-alone Enqueue Server
* The enqueue replication server instance is only mandatory in a high-availability system.





Optionally, you can install the ASCS instance with an integrated:



* SAP Web Dispatcher. For more information, see [ASCS Instance with Integrated SAP Web Dispatcher](https://help.sap.com/viewer/910828cec5d14d6685da380aec1dc4ae/CURRENT_VERSION/en-US/2e708e2d42134b4baabdfeae953b24c5.html).
* Gateway. For more information, see [ASCS Instance with Integrated Gateway](https://help.sap.com/viewer/ce9e270ad34949969c16d09d1b099a26/CURRENT_VERSION/en-US/bf1d359ac8384441a781ae3b0b5bd1b5.html).



Architecture of SAP NetWeaver AS for JAVA



1. Java central instance (J< nn > instance) – A Java instance is a unit in the AS Java cluster that is identified by its instance number. The elements that form an instance that is run on one physical machine. Also, it is possible to run several instances on one physical machine, but it is recommended that you split the different instances among different physical machines.
Creating and working with bare metal servers (ibmcloud sl hardware) 

IBM Cloud® Bare Metal Servers are single-tenant physical servers that provide you performance and control with low-level access to the hardware resources. Bare metal servers provide the raw horsepower you demand for your processor-intensive and disk I/O-intensive workloads. These servers come with the most complete package of standard features and services.

Use the following commands from the IBM Cloud® Command Line Interface to manage IBM Cloud classic infrastructure Bare-metal hardware servers.



 ibmcloud sl hardware cancel 

Cancel a hardware server:

ibmcloud sl hardware cancel IDENTIFIER [OPTIONS]

Command options:

-i, --immediate
: Cancels the server immediately (instead of on the billing anniversary).

-r, --reason
: An optional cancellation reason. See ibmcloud sl hardware cancel-reasons for a list of available options.

-c, --comment
: An optional comment to add to the cancellation ticket.

-f, --force
: Force operation without confirmation.





 ibmcloud sl hardware cancel-reasons 

Display a list of cancellation reasons:

ibmcloud sl hardware cancel-reasons





 ibmcloud sl hardware create 

Order/create a hardware server:

ibmcloud sl hardware create [OPTIONS]

Command options:

-H, --hostname
: Required. Host portion of the FQDN.

-D, --domain
: Required. Domain portion of the FQDN.

-s, --size
: Required. Hardware size.

-o, --os
: Required. OS installation code.

-d, --datacenter
: Required. Datacenter shortname.

-p, --port-speed
: Required. Port speed.

-b, --billing
: Billing rate, either hourly or monthly, default is hourly if not specified.

-i, --post-install
: Post-install script to download.

-k, --key
: SSH keys to add to the root user, multiple occurrence allowed.
Ordering more bare metal servers 



  Citrix Hypervisor 

If you want to increase your bare metal server count for Citrix DaaS for IBM Cloud® in your resource location, you can add more servers. Before you order more bare metal servers, make sure you identify your system's [network details](https://cloud.ibm.com/docs/citrix-daas?topic=citrix-daas-post-provisioning-citrix-daasview-network-details). You need to make sure to select the same VLAN and primary subnet when you order more servers.

You can order more bare metal servers through the [IBM Cloud® catalog](https://cloud.ibm.com/catalog?category=computeservices).





  VMware ESXi 

When you add or delete a service or infrastructure component, you must wait until the modification is complete before you can add or delete another service or infrastructure component.

You can add more bare metal servers (ESXi hosts) to scale out your existing cluster.



1.  Log in to the [IBM Cloud® console](https://cloud.ibm.com/login) by using your unique credentials.
2.  Select Menu icon ![Menu icon](https://cloud.ibm.com/docs-content/v1/content/e9cda99c64ea1ee68cf0eda257d6a1de69ddbc4e/icons/icon_hamburger.svg) > VMware > Resources.
3.  Select your vCenter Instance.
4.  Click the Infrastructure tab.
5.  Click Add for the ESXi servers to add more hosts to the cluster.
6.  Select the licensing option that you use.
7.  Select the CPU generation, model, RAM, Storage, and Network Interface options for your solution.
8.  Accept the terms and click Create.
9.  Verify that the added ESXi server is not in Maintenance Mode. You cannot create new virtual machines if the new server is in Maintenance Mode. View the new server in the vSphere client under the cluster and take the server out of Maintenance Mode, if necessary.
Retrieving details of a serverless instance 

You can retrieve information, like the instance ID (or GUID) and the provisioning state of an IBM Analytics Engine serverless instance from the instance details. You need the instance ID to use the Spark application REST API and the Livy batch APIs.

You can retrieve the details by:



* [Using the IBM Cloud CLI](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-instance-detailsretrieve-guid-cli)
* [Using the IBM Cloud REST API](https://cloud.ibm.com/docs/AnalyticsEngine?topic=AnalyticsEngine-retrieve-instance-detailsretrieve-guid-api)





 Accessing instance details by using the IBM Cloud CLI 

To get the details of an instance:



1. List all of the created serverless instances:

$ ibmcloud resource service-instances --service-name ibmanalyticsengine

This call retrieves the instances of type service_instance in all resource groups in all locations for your account.

Example response:

Name Location State Type Resource Group ID
serverless-instance us-south active service_instance 65xxxxxxxxxxxxxxxa3fd
2. Enter the following command with the server instance name of your instance to view the instance details:

$ ibmcloud resource service-instance "Analytics Engine-xyz"

This retrieves your instance from the resource groups under your account.

Example response:

Name: Analytics Engine-xyz
ID:
crn:v1:staging:public:ibmanalyticsengine:us-south:a/XXXXX:XXXXX::
GUID: XXXXX
Location: us-south
Service Name: ibmanalyticsengine
Service Plan Name: standard-serverless-spark
Resource Group Name: Default
State: active
Type: service_instance
Sub Type:
Created at: 2021-01-06T07:49:12Z
Created by: XXXXX
9 August 2021 

LinuxONE (s390x processor architecture)
: You can now create virtual server instances on LinuxONE (s390x processor architecture) in IBM Cloud in the Tokyo region. For more information about available LinuxONE (s390x processor architecture) profiles, see [s390x instance profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-vs-profiles). To create instances on LinuxONE (s390x processor architecture), see [Creating virtual server instances by using the UI](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-virtual-servers). For more information about some of the limitations of LinuxONE (s390x processor architecture), see [Service limitations](https://cloud.ibm.com/docs/vpc?topic=vpc-limitations).







 July 2021 



 26 July 2021 

New São Paulo region
: The São Paulo region endpoint is now in service at https://br-sao.iaas.cloud.ibm.com. For more information, see [Creating a VPC in a different region](https://cloud.ibm.com/docs/vpc?topic=vpc-creating-a-vpc-in-a-different-region).





 22 July 2021 

Block Storage for VPC
: IOPS tier and custom profiles are available with volume capacity up to 16,000 GB. For more information, see [Block storage profiles](https://cloud.ibm.com/docs/vpc?topic=vpc-block-storage-profiles).
: For secondary volumes attached to a virtual server instance, you can increase capacity in GB increments up to 16,000 GB, depending on the volume's profile. The volume capacity is immediately increased. For more information, see [Expanding block storage volume capacity](https://cloud.ibm.com/docs/vpc?topic=vpc-expanding-block-storage-volumes).





 21 July 2021 

Bare Metal Servers for VPC (closed beta)