17. Mainframes vs Cloud Computing
Tags: cloud, aws
Published: 2021-02-13 20:31:47

In this article I am going to highlight the key challenges I faced while working in mainframe/DB2 support and how cloud technologies can negate them. Things I learnt while making my [CloudResume](https://bit.ly/komlalebu).

# Business as usual.

While I was working as a contractor for a US based health insurance company**, I often used to get bombarded with calls over the weekend asking for assistance. I was the only offshore POC for a set of business critical batch jobs.

** All object names are arbitrary.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/z5socsvg57c5czfe2amn.PNG)

# Architecture
* Transactional data from various third party sources/service providers came into the staging area which updated the MEM* table in `base` mainframe using [CDC](https://www.ibm.com/support/pages/ibm-data-replication-change-data-capture-cdc-best-practices) during business hours.

* Then the table data was loaded into  MEM* table in `clone` mainframe by a 4 hour nightly cycle (12am - 4am) CST. 

* The consolidated previous days data was consumed by business from 7am CST onwards through frontend, for processing health claims of members.

* These 70 odd batch jobs were business critical as any failure in any one of them would stop the entire batch. We had to fix the abended jobs and restart the cycle. However, there was a SLA deadline of 4 am, beyond which the frontend would access 1 day old data (from the mainframe) and the claims wouldn't get processed, leading to escalations.

*MEM table - used to consolidate health care plan information of members. 

# 1. Space Issue

The primary challenge that we usually faced was DASD space issue - `SB37` abend. These jobs would frequently abend due to a lack of storage.
The batch jobs were a set of sequential `unload` jobs which unloaded records of MEM table from `base` mainframe into 70 datasets. These datasets were later loaded into the `clone` mainframe  with the separate load jobs.

> `Example - UNLOAD JOB 1: `[JCL](https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zjcl/zjclc_basicjclconcepts.htm).

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/g8qbyvbi2vmhnarn2jyf.PNG)

`space parameter`

                             //SPACE=(CYL,(20,10),RLSE) --> Defines the space parameters with primary quantity = 20 ,
                             secondary = 10. Total = (20 + (10*15)) = 170 cylinders allocated.
                             Highlighted - Control card dataset used to pass LIMIT key. 

>
`Formula` - to calculate space parameters
>
                         Number of cylinders needed = (number of records * LRECL of dataset) / 849960  
                         Records : 340000 
                         LRECL : 214
                         Bytes per cylinder(standard): 849960 
                         Number of cylinders : approximately 86 cylinders needed for 340000 records 

The problem was business was not able to confirm the approximate spike in the number of records day to day and finding a correct space parameter for all jobs was becoming difficult. And if we decided to set a primary quantity too large in JCL, [MVS](https://en.wikipedia.org/wiki/MVS) was not able to allocate extents due to other application jobs also requiring space in the same [LPAR](https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zmainframe/zconc_mfhwlpar.htm).
I had to coordinate with third-party `storage teams` to set the correct space parameter rerun the jobs once they allocated additional ([DASD](https://ibmmainframes.com/references/disk.html) volume). 
By the time we fixed the issue, SLA of 4am breached and I had to explain to the management and onshore. 

![Alt Text](https://external-content.duckduckgo.com/iu/?u=http%3A%2F%2Fi0.kym-cdn.com%2Fphotos%2Fimages%2Fnewsfeed%2F000%2F757%2F498%2F89c.gif&f=1&nofb=1)
Storage team was reluctant to increase DASD volume allocation permanently for our jobs as `storage is a premium`, so they wanted justification, however our business team was unable to confirm future spike in records. So triage calls went on forever without any permanent fix.

>
# Why Cloud?   

>

>
                     S3 --> Lambda --> AWS Batch --> RDS
The source files can be stored in `S3`. Once they get pushed to S3 we can configure S3 SNS event notification to a separate `lamdba` function. However `lambda` function can only run for max 15 mins, therefore we need `AWS Batch` for long running SQL workloads. The lambda function will trigger AWS Batch to do the work.

>
As AWS Batch provisions and manages resources (EC2 + EBS storage) it scales accordingly, so we don't have any resource constraint or space issues. Once the batch processing is done all alllocated infrastructure will be removed, thus saving cost. 

>
Once the batch processing is over the data can be pushed to AWS `RDS` for storage which can be in multi-AZ deployment having synchronous backup.

# 2. LIMIT key partitioning

As the records of MEM table containing the consolidated member data of our application was getting loaded with new records we had to manually partition the UNLOAD jobs based on `MEM_ID` field as the LIMIT key. This helped avoid some jobs consuming more resources than others.

> `Example - UNLOAD JOB 1:CONTROL CARD DATASET`
LIMIT key set as MEM_ID.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/05f52z03gsvhfbswxpf7.PNG)

* Before - Job 1 was doing all the unloading and consuming more resources.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/16a8pqsln9vl9jimpsbd.PNG)

* After - Jobs are evenly distributed to unload records based on new Limit keys.

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/4lp0gb6e9t7da7kekzlp.PNG)

>
# Why Cloud?  

>
Amazon DynamoDB stores data in partitions. Partition management is handled entirely by DynamoDB—you never have to manage partitions yourself.

# 3. CPU - soft capping

There were several instances when the batch jobs ran slower and consumed higher `Elapsed` times. On further investigation we found that the mainframe performance & capacity team, had enabled softcapping for our application jobs without informing us. This was done to provision more `CPU` time to other application jobs running in the same [LPAR](https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zmainframe/zconc_mfhwlpar.htm), and sometimes to avoid heating issues.  
This did not cause any job `abends`, however it delayed the entire process due to slow running jobs, thus causing SLA breach due lack of compute resources.

>
# Why Cloud?   

>
With cloud services we can auto-scale our infrastructure and automate capacity management, thus avoiding situations like these.

# 4. Fragmented regions

The 70 odd batch jobs were taking significantly longer unload times in pre-production regions. This is due to senior DBA's forgetting to perform [REORG](https://www.ibm.com/support/knowledgecenter/en/SSEPEK_10.0.0/ugref/src/tpc/db2z_reorgtablespacesamples.html) on the `base` mainframe tables in those regions, after production data migration. There was also discrepancies in the `index` definitions for tables in those regions compared to production tables.

>
# Why Cloud?   

>
This scenario can be avoided in the cloud with help of uniform `Cloudformation` templates deploying infrastructure across regions. [IAC](https://dev.to/jicoing/12-infrastructure-as-code-1df4).

# 5. Access restrictions

While working for the offshore account, there were several occasions when business wanted me to query unrestricted production tables or deploy JCL changes in production to fix certain issues. However my [TSO ID](https://www.ibm.com/support/knowledgecenter/zosbasics/com.ibm.zos.zconcepts/zconc_whatistso.htm) did not have access to the tools (SPUFI, Changeman) needed to do that. This was enabled across entire offshore account. So I had to wait for my onshore counterpart to perform those changes, causing delays.

>
# Why Cloud?
Granular IAM policies for  READ/WRITE access to a resource `ARN` can be attached to a specific user, in a group.

# 6. Scheduling

The mainframe batch jobs across regions were scheduled using a [ASG-ZEKE](https://www.asg.com/en/Products/IT-Systems-Management/Operations-Management/Multiplatform-Workload-Automation/ASG-Zeke.aspx) tool. Schedule changes across regions were handled by a separate contractor and we had to traige with them in case of any minor changes. 

>
# Why Cloud?
I found CI/CD pipelines and job scheduling easy to setup in Github actions and the way it implements workflows for inter-dependant jobs is easy to track.

# 7. Maintainence

Although mainframes are extremely robust machines, they do require maintainence. If any jobs are running during those timelines they are cancelled(S222 abend) by datacenter, due to LPAR being taken down. So we had to analyse the list of high impacted job prematurely and put them on hold. This was a tedious process as maintainence timelines vary. Also we were unable to query tables needed for resolving incidents or fix defects during those hours.   

>
# Why Cloud?
With a distributed infrastructure across several regions, there is better hardware backup incase of failures or outage. 

# 8. Database

* While working as a junior DBA in one of my earlier projects, I saw the challenges among developers and the DBA's. The DBA's usually have DDL,DML scripts which need to be managed on regular basis after triaging with developers. 
* In a RDBMS like DB2 an admin usually needs to configure the database, define the schema, tablespace, field data type and length  while creating a table and then grant access to the application developers. 

>
`Sample DB2 SQL`

                      create table  MYSCHEMA.MYTABLE (
                      MEM_ID CHAR(5) NOT NULL,
                      MEM_NAME CHAR(50) NOT NULL,
                      EMAIL CHAR(8) NOT NULL,
                      UPD_DTE DATE NOT NULL,
                      PRIMARY KEY (MEM_ID)
                      ) in database DB1234.TS1234;

                      grant INSERT,UPDATE,SELECT,DELETE ON MYSCHEMA.MYTABLE TO GROUP;
                      grant INSERT,UPDATE,SELECT,DELETE ON MYSCHEMA.MYTABLE TO USER;

* `Select` queries written must be optimized for better performance. This requires additional knowledge of the DB2 optimizer.

>
# Why Cloud?

>
A NoSQL database like DynamoDB works primarily on key value pair, requires only a primary key and doesn’t require a schema to create a table. It doesn't have any administrative burdens of operating and scaling a distributed database. The table can be created using [Boto 3](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/dynamodb.html#DynamoDB.Table.scan) in python or [SAM template](https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-resource-simpletable.html). There is no need for separate DBA's to perform these tasks. However, DynamoDB is not a RDBMS and does NOT support SQL queries like them.

>
`Sample table` - MYTABLE
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/pd3kqhgea2vmtrdcd9cy.PNG)

>
                              SELECT * FROM MYTABLE
                              WHERE EMAIL ='username@email.com'; 

>
To perform a similar operation in `DynamoDB` we can perform table scan on the entire table, which is costly and not recommended. The other way is to define a `hash` key on mem_id attribute, and create a `GSI` - Global secondary Index on email attribute, in SAM template. 

>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/542f92bbf2zjudh1bder.PNG)

>
Then we can search the desired rows with Boto3 in python as per application requirement.


>
`Using: query`

>
![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/nugk5bf9sa44n05mjy39.PNG)

>
I got these concepts from [here](https://highlandsolutions.com/blog/hands-on-examples-for-working-with-dynamodb-boto3-and-python) and [here](https://stackoverflow.com/questions/56602292/how-do-i-implement-a-dynamodb-global-secondary-index-with-infrastructure-as-code).
NoSQL needs a different approach compared to RDBMS. There is also traditional RDBMS support in the cloud using RDS.

# 9. User Interface

The ISPF interface can be challenging to navigate for repetative tasks. However [VSCode](https://code.visualstudio.com/) editor support was recently added for Z/OS Mainframes. Z/OS only allows a single session for a TSO userID, so we cannot open multiple sessions across different terminals/devices.

![Alt Text]
(https://external-content.duckduckgo.com/iu/?u=https%3A%2F%2Fwww.ibm.com%2Fcommunity%2Fz%2Fwp-content%2Fuploads%2Fsites%2F14%2F2020%2F04%2Fsysdevblog-4c9d-ispfpom.png&f=1&nofb=1)

>
# Why Cloud?

>
I can use the editor of my choice for development. I can pass my `IAM` credentials as environment parameters while deploying code through a virtual machine, which blew my mind.  

# Those were some of my observations. Thank you for reading..

![Alt Text](https://dev-to-uploads.s3.amazonaws.com/i/0vobgcej8t0ytnl5d9l4.gif)