El Carro: The Oracle Operator for Kubernetes
Tags: google, oracle, operator, kubernetes
Published: 2021-05-15 04:22:31

Google Cloud, Open Source and Oracle Databases... what seems to be a paradox is possible, thanks to cloud providers who contribute to open infrastructure. The idea is to use Operators (custom resource controllers on Kubernetes) to automate the Oracle Database operations in a standard, open and portable way. If you ever attempted to run Oracle Database on containers, trying to keep up with the DevOps approach, you know that it requires a bit of complexity and careful orchestration.

The public announce was on the Google Open Source Blog: <a href="https://opensource.googleblog.com/2021/05/modernizing-oracle-operations-with-kubernetes-el-carro.html" rel="noopener" target="_blank">Modernizing Oracle operations with Kubernetes and El Carro</a>. This is an Open Source project where we can contribute: <a href="https://github.com/GoogleCloudPlatform/elcarro-oracle-operator" rel="noopener" target="_blank">https://github.com/GoogleCloudPlatform/elcarro-oracle-operator</a>. I've tried the simplest thing: install Oracle XE - the free edition of Oracle, because it is the only one that you can deploy without cross-checking, with your lawyers, the license contracts and the "educational purpose only" documents about Oracle audit policies. But running Oracle on Kubernetes applies the same rules as virtualization: count the vCPU or the physical processors (depending on the hypervisor isolation accepted by Oracle). Basically the "installed or running" terms apply where the image is pulled.

<h3>Download El Carro software and install Oracle 18c XE</h3>
I'll run all this from the Cloud Shell but of course you can do it from any configured gcloud CLI.
<pre><code>
franck@cloudshell:~ (google-cloud.424242)$ gcloud alpha iam service-accounts list

DISPLAY NAME                            EMAIL                                               DISABLED
Compute Engine default service account  424242424242-compute@developer.gserviceaccount.com  False
</code></pre>
I take note of my service account from there.

Installing Oracle is 3 lines only:
<pre><code>mkdir -p $HOME/elcarro-oracle-operator
gsutil -m cp -r gs://elcarro/latest $HOME/elcarro-oracle-operator
bash $HOME/elcarro-oracle-operator/latest/deploy/install-18c-xe.sh --service_account 424242424242-compute@developer.gserviceaccount.com</code></pre>
I'm following the instructions from <a href="https://github.com/GoogleCloudPlatform/elcarro-oracle-operator/blob/main/docs/content/quickstart-18c-xe.md" rel="noopener" target="_blank">https://github.com/GoogleCloudPlatform/elcarro-oracle-operator/blob/main/docs/content/quickstart-18c-xe.md</a> here.

<a href="https://blog.dbi-services.com/wp-insides/uploads/sites/2/2021/05/Screenshot-2021-05-13-221419-scaled.jpg"><img src="https://blog.dbi-services.com/wp-insides/uploads/sites/2/2021/05/Screenshot-2021-05-13-221419-scaled.jpg" alt="" width="2560" height="1151" class="aligncenter size-full wp-image-49940" /></a>

This takes a while the first time (45 minutes) because it has to create the image, built from <a href="https://download.oracle.com/otn-pub/otn_software/db-express/oracle-database-xe-18c-1.0-1.x86_64.rpm" rel="noopener" target="_blank">oracle-database-xe-18c-1.0-1.x86_64.rpm</a> RPM. The image is nearly 6GB and is stored in your Container Registry. Then it creates the cluster. The default cluster name is "gkecluster", the CDB name is GCLOUD and the defaut zone is us-central1-a but you can pass the -c -k -z option on the install-18c-xe.sh to change those defaults. It creates a 2 nodes, total 4 vCPUs, 15 GB RAM, 200GB persistent storage. The namespace is "db".

<pre><code>
...
kubeconfig entry generated for gkecluster.
NAME        LOCATION       MASTER_VERSION   MASTER_IP     MACHINE_TYPE   NODE_VERSION     NUM_NODES  STATUS
gkecluster  us-central1-a  1.19.9-gke.1900  34.67.217.61  n1-standard-2  1.19.9-gke.1900  2          RUNNING
storageclass.storage.k8s.io/csi-gce-pd created
volumesnapshotclass.snapshot.storage.k8s.io/csi-gce-pd-snapshot-class created
namespace/operator-system created
...
Waiting for startup, statuses: InstanceReady=, InstanceDatabaseReady=, DatabaseReady=
Waiting for startup, statuses: InstanceReady=, InstanceDatabaseReady=, DatabaseReady=CreatePending
...
Waiting for startup, statuses: InstanceReady=CreateInProgress, InstanceDatabaseReady=, DatabaseReady=CreatePending
...
Waiting for startup, statuses: InstanceReady=CreateComplete, InstanceDatabaseReady=CreateInProgress, DatabaseReady=CreatePending
...
Waiting for startup, statuses: InstanceReady=CreateComplete, InstanceDatabaseReady=CreateComplete, DatabaseReady=CreatePending
Waiting for startup, statuses: InstanceReady=CreateComplete, InstanceDatabaseReady=CreateComplete, DatabaseReady=CreateComplete
Oracle Operator is installed. Database connection command:
&gt; sqlplus scott/tiger@35.224.235.49:6021/pdb1.gke
franck@cloudshell:~ (google-cloud.424242)$
</code></pre>
Be patient... it is Oracle, it has a pre-DevOps installation timing... And this is why it is really good to have a standardized way for automation. Building your own is a lot of effort because each iteration takes time to validate.

So, all is installed, with a service endpoint exposed to the public internet on port 6021:
<pre><code>
[opc@a ~]$ ~/sqlcl/bin/sql scott/tiger@35.224.235.49:6021/pdb1.gke

SQLcl: Release 21.1 Production on Fri May 14 10:40:03 2021

Copyright (c) 1982, 2021, Oracle.  All rights reserved.

Connected to:
Oracle Database 18c Express Edition Release 18.0.0.0.0 - Production
Version 18.4.0.0.0

SQL&gt; select * from v$session_connect_info;

   SID    SERIAL#    AUTHENTICATION_TYPE    OSUSER                                                    NETWORK_SERVICE_BANNER    CLIENT_CHARSET    CLIENT_CONNECTION    CLIENT_OCI_LIBRARY    CLIENT_VERSION            CLIENT_DRIVER            CLIENT_LOBATTR    CLIENT_REGID    CON_ID
______ __________ ______________________ _________ _________________________________________________________________________ _________________ ____________________ _____________________ _________________ ________________________ _________________________ _______________ _________
   283      20073 DATABASE               opc       TCP/IP NT Protocol Adapter for Linux: Version 18.0.0.0.0 - Production     Unknown           Heterogeneous        Unknown               21.16.0.0.0       jdbcthin : 21.1.0.0.0    Client Temp Lob Rfc On                  0         3
   283      20073 DATABASE               opc       Encryption service for Linux: Version 18.0.0.0.0 - Production             Unknown           Heterogeneous        Unknown               21.16.0.0.0       jdbcthin : 21.1.0.0.0    Client Temp Lob Rfc On                  0         3
   283      20073 DATABASE               opc       Crypto-checksumming service for Linux: Version 18.0.0.0.0 - Production    Unknown           Heterogeneous        Unknown               21.16.0.0.0       jdbcthin : 21.1.0.0.0    Client Temp Lob Rfc On                  0         3

SQL&gt; select initcap(regexp_replace(reverse('El Carro'),'(.)\1+| ','\1')) "K8s Operator for" from dual;

  K8s Operator for
__________________
Oracle

SQL&gt; 
</code></pre>
Now you see where the "El Carro" name comes from, right? &#x1f923;

I can check the pods, from the Web Console, or CLI, remember the namespace is 'db':
<pre><code>franck@cloudshell:~ (google-cloud.424242)$ kubectl get pods -n db

NAME                                     READY   STATUS    RESTARTS   AGE
mydb-agent-deployment-6c8b7647fb-d4lkf   2/2     Running   0          77m
mydb-sts-0                               4/4     Running   0          77m

franck@cloudshell:~ (google-cloud.424242)$ kubectl get services -n db

NAME                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)                         AGE
mydb-agent-svc      ClusterIP      10.59.242.135             3202/TCP,9161/TCP               107m
mydb-dbdaemon-svc   ClusterIP      10.59.244.15              3203/TCP                        107m
mydb-svc            LoadBalancer   10.59.245.142   35.224.235.49   6021:30156/TCP,3307:32007/TCP   107m
mydb-svc-node       NodePort       10.59.250.249             6021:32512/TCP,3307:31243/TCP   107m
franck@cloudshell:~ (google-cloud.424242)$
</code></pre>
The service is exposed externally by the LoadBalancer on the Secure Listener port

I can connect to the container to look at what is running there.
<pre><code>
franck@cloudshell:~ (google-cloud.424242)$ kubectl  exec -it -n db mydb-sts-0 -c oracledb -- bash -i

bash-4.2$ grep ":[YN]" /etc/oratab

GCLOUD:/opt/oracle/product/18c/dbhomeXE:N
 
bash-4.2$ . oraenv &lt;&lt;&lt;GCLOUD

ORACLE_SID = [] ? The Oracle base remains unchanged with value /opt/oracle
 
bash-4.2$ ps -fp $(pgrep tnslsnr)

UID          PID    PPID  C STIME TTY          TIME CMD
oracle       488       1  0 09:32 ?        00:00:00 /opt/oracle/product/18c/dbhomeXE/bin/tnslsnr SECURE -inherit

bash-4.2$ lsnrctl status SECURE          

LSNRCTL for Linux: Version 18.0.0.0.0 - Production on 14-MAY-2021 10:18:11

Copyright (c) 1991, 2018, Oracle.  All rights reserved.

TNS-01101: Could not find listener name or service name SECURE

bash-4.2$ lsnrctl status //localhost:6021

LSNRCTL for Linux: Version 18.0.0.0.0 - Production on 14-MAY-2021 10:17:42

Copyright (c) 1991, 2018, Oracle.  All rights reserved.

Connecting to (DESCRIPTION=(CONNECT_DATA=(SERVICE_NAME=))(ADDRESS=(PROTOCOL=TCP)(HOST=127.0.0.1)(PORT=6021)))
STATUS of the LISTENER
------------------------
Alias                     SECURE
Version                   TNSLSNR for Linux: Version 18.0.0.0.0 - Production
Start Date                14-MAY-2021 09:32:52
Uptime                    0 days 0 hr. 44 min. 49 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u02/app/oracle/oraconfig/network/SECURE/listener.ora
Listener Log File         /u02/app/oracle/diag/tnslsnr/mydb-sts-0/secure/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=REGLSNR_6021)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=mydb-sts-0)(PORT=6021)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=127.0.0.1)(PORT=5500))(Security=(my_wallet_directory=/opt/oracle/admin/GCLOUD_uscentral1a/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service &quot;GCLOUD.gke&quot; has 1 instance(s).
  Instance &quot;GCLOUD&quot;, status UNKNOWN, has 1 handler(s) for this service...
Service &quot;GCLOUDXDB.gke&quot; has 1 instance(s).
  Instance &quot;GCLOUD&quot;, status READY, has 1 handler(s) for this service...
Service &quot;GCLOUD_uscentral1a.gke&quot; has 1 instance(s).
  Instance &quot;GCLOUD&quot;, status READY, has 1 handler(s) for this service...
Service &quot;PDB1.gke&quot; has 2 instance(s).
  Instance &quot;GCLOUD&quot;, status UNKNOWN, has 1 handler(s) for this service...
  Instance &quot;GCLOUD&quot;, status READY, has 1 handler(s) for this service...
Service &quot;c246feca2ab003e8e0530901380a0e21.gke&quot; has 1 instance(s).
  Instance &quot;GCLOUD&quot;, status READY, has 1 handler(s) for this service...
The command completed successfully

bash-4.2$
</code></pre>
Here is the Oracle XE listener, with TNS_ADMIN in /u02/app/oracle/oraconfig/network/SECURE

You can have a look at the available operations from the samples (that you can customize and tun with `kubectl apply -n db -f`):
<pre><code>
franck@cloudshell:~ (google-cloud.424242)$ ls ./elcarro-oracle-operator/latest/samples

v1alpha1_backup_rman1.yaml            v1alpha1_database_pdb1.yaml
v1alpha1_backup_rman2.yaml            v1alpha1_database_pdb2.yaml
v1alpha1_backup_rman3.yaml            v1alpha1_database_pdb3.yaml
v1alpha1_backup_rman4.yaml            v1alpha1_database_pdb4.yaml
v1alpha1_backupschedule.yaml          v1alpha1_export_dmp1.yaml
v1alpha1_backup_snap1.yaml            v1alpha1_export_dmp2.yaml
v1alpha1_backup_snap2.yaml            v1alpha1_import_pdb1.yaml
v1alpha1_backup_snap_minikube.yaml    v1alpha1_instance_18c_XE_express.yaml
v1alpha1_config_bm1.yaml              v1alpha1_instance_18c_XE.yaml
v1alpha1_config_bm2.yaml              v1alpha1_instance_custom_seeded.yaml
v1alpha1_config_gcp1.yaml             v1alpha1_instance_express.yaml
v1alpha1_config_gcp2.yaml             v1alpha1_instance_gcp_ilb.yaml
v1alpha1_config_gcp3.yaml             v1alpha1_instance_minikube.yaml
v1alpha1_config_minikube.yaml         v1alpha1_instance_standby.yaml
v1alpha1_cronanything.yaml            v1alpha1_instance_unseeded.yaml
v1alpha1_database_pdb1_express.yaml   v1alpha1_instance_with_backup_disk.yaml
v1alpha1_database_pdb1_gsm.yaml       v1alpha1_instance.yaml
v1alpha1_database_pdb1_unseeded.yaml
franck@cloudshell:~ (google-cloud.424242)$
</code></pre>
Storage snapshots (v1alpha1_backup_snap2.yaml), backups (v1alpha1_backup_rman3.yaml), exports (v1alpha1_export_dmp1.yaml)

All is documented: <a href="https://github.com/GoogleCloudPlatform/elcarro-oracle-operator" rel="noopener" target="_blank">https://github.com/GoogleCloudPlatform/elcarro-oracle-operator</a> and will probably evolve.