Are you Logging your applications RIGHT?
Tags: logging, react, javascript, node
Published: 2020-04-16 06:18:49

# Log Management & Monitoring

This document is the baseline for setting up the strategy for Logging & Monitoring.

## Table of Content
- [What to Log](#what-to-log)
  - Basics
  - Events to Log
- [How to Log](#how-to-log)
  - Actionable's
  - Log Management
  - Security Aspects In Details
- [Roles & Responsibilities](#roles--responsibilities)
  - Log types
  - What to never log
- [Next ToDo’s](#next-todos)
  - Select Logging Framework for Frontend/Backend/Infra etc
  - References

## What to Log

### Basics
- Only log what Legislation allowed
   - Example GDPR
- Maintaining Confidentiality | Only Public Records
   - Password & Encryption Keys
   - Payment Details
   - Detailed System Information etc
   - Information exposure through Error Messages
- Sufficient Logging | NOT Logging Everything
   - CWE-779 Logging Excessive Data
- Optimization Strategy: Right set of Training & Utilization
   - Not Determining what to monitor & how
   - Right Training & Documentation
   - Putting right practice for Alerting

### Events to Log:
- Authentication Events
   - All Success & Failure
   - Building a security policy
   - Access control violations
    - Incorrect Logins
    - Policy for Internal Employees
    - User Authentication(Failed/Reset/Successful) for all services including k8, Applications, Internal Systems
- Authorization Events
   - Failure of Tokens & Internal Access Violation
- Application Errors
   - All Application Errors
   - Startup & Shutdown Events
   - Configuration Changes
   - Application State Information
   - Input & Output validation(maintaining signal to noise ratio)



## How to Log

### Actionable's
- Deciding where to record log files
 - Local Logs(Not a good practice)
  - Not to log locally, but if its required, log in separate partition & enable access control.
  - Access Control
 - Logging Remotely or Centralized Log-Server
     - Full Encryption
     - Access Control Mechanism
     - Integrity Checks for Log Files
     - Fail-over system
     - Setting up regular backups
     - Adding alerting
- Format of log files
   - Metadata requirements
    - what has happened
    - when it happened
       - [timestamp with Timezone](https://medium.com/easyread/understanding-about-rfc-3339-for-datetime-formatting-in-software-engineering-940aa5d5f68a)
        - source
        - destination
        - Synchronize Time Sources
    - where it happened
       - source address
       - originating source(ipv4/ipv6)
    - who is responsible for the action
       - Logged on/Attempting User
       - Unique identifier
 - [Standard Logging Format](https://tools.ietf.org/html/rfc5424)
  - Timestamp
  - Encoding - UTF8
  - Severity Levels(Standards)
     - 0 - Emergency - System crash
     - 1 - Alert - action must be taken
     - 2 - Critical - critical condition such as load
     - 3 - error - error conditions
     - 4 - warning
     - 5 - Notice
     - 6 - Informational
     - 7 - Debug
- Handling Personal Data
   - Encrypting Personal Data
   - Pseudonymization(Privacy Enhancing Techniques)
   - Consulting Legal

### Log Management
- Log Aggregation
   - Consolidate Duplicate Events
   - Add Structure to the Docs
   - Remove sensitive Data through filters
   - Pseudonymization
   - Security
    - Input validation(XSS also possible in log files)
    - Encoding(Safeguard against injection attacks)
    - Filtering
- Log Analyzing
   - Baselining
   - Anomaly Detection
   - Attack Signatures
- Log Archiving


### Security Aspects In Details
- Information exposure through Error Messages
- Unsuccessful login attempts should be monitored
- Never, ever log credentials, passwords or any sensitive information.
- Detecting Network Intrusion
- ELK Login to be Employee should be strong & unique password to prevent
- Following OWASP Top-10

## Roles & Responsibilities
- IT Leadership
   - Assigning responsibilities
   - Tools & Trainings
   - Incident response
- Software/DevOps Lead
   - Defining what to log & monitor. (Generally this happens in after Planning phase)
   - Clear Documentation
   - Common Format of log files
   - Optimization or tweaking monitoring during operational phase
   - Maintaining the policy
   - Action on Log
   - Adding Logging/Monitoring to release notes & continuously monitoring behaviour in production
- DevOps
   - Defining Response strategy
   - Setting up Infra & Enabling
   - Common Format of log files
   - Continuous Monitoring & Alerting
   - Verifying Pattern such as continuous login attempts
   - Defining Response strategy
   - Reporting policy violations
   - Generating regular reports
- Legal Department
   - Validating the information logged & stored in database

## Log types
- Application logs[http request/response log, application level error
```json
     {
     'id' : 'unique request id for tracing',
     'req' :{}, //Generated by Logger
     'res': {}, //Generated by Logger
     'level':'error',
     'message':'There is an error',
     'timestamp': '2016-06-12T05:00:00',
        'timezone': 'Pacific/Auckland',
     'context':'exception trace'
    }
```

 [Reference](https://www.npmjs.com/package/express-winston)
- db log [query,error]
```json
     {
       "timestamp": "",
       "severity": "",
       "component": "",
       "context": "",
       "message": ""
       "dbStats" : {}, //Generated by Logger
       "serverStatus": {} //Generated by Logger 
     }
```

- container level log
```json
    {
       "container" : 
       {
         "name": ""
       },
       ecs : {}
    }
```


[See the demo for reference](https://demo.elastic.co/app/kibana#/dashboard/19e7fae0-92a6-11e8-8fa2-3d5f811fbd0f)

## What to never log
- Customer Confidential Information
- Payment Records such as Credit Card
- Banking Information


## References
- https://www.datadoghq.com/blog/monitoring-mean-stack-applications-with-datadog/
- Database Logs
   - https://docs.mongodb.com/manual/reference/command/serverStatus/#server-status-locks
   -https://docs.mongodb.com/manual/reference/command/dbStats/#dbcmd.dbStats
- K8 Logging
   - https://itnext.io/logging-best-practices-for-kubernetes-using-elasticsearch-fluent-bit-and-kibana-be9b7398dfee
- https://github.com/pimterry/loglevel
- https://medium.com/@davidmcintosh/winston-a-better-way-to-log-793ac19044c5
- https://stackoverflow.com/questions/5817738/how-to-use-log-levels-in-java
- https://blog.papertrailapp.com/best-practices-for-logging-in-nodejs/
- https://github.com/trentm/node-bunyan#readme
- https://medium.com/containerum/4-tools-to-monitor-your-kubernetes-cluster-efficiently-ceaf62818eea
- https://blog.coinbase.com/logs-metrics-and-the-evolution-of-observability-at-coinbase-13196b15edb7
- https://medium.com/easyread/understanding-about-rfc-3339-for-datetime-formatting-in-software-engineering-940aa5d5f68a
- https://webilicious.xyz/utilizing-winston-to-log-node-js-applications/
- Security
   - https://attack.mitre.org/
   - https://cwe.mitre.org/
   - https://cwe.mitre.org/data/definitions/778.html
   - https://cwe.mitre.org/data/definitions/223.html
   - https://cwe.mitre.org/data/definitions/209.html
   - https://cwe.mitre.org/data/definitions/779.html