Troubleshooting with VMware NSX ALB/Avi Vantage
Tags: avi, loadbalancing, nsxalb, vmware
Published: 2021-05-23 16:52:41

---
title: Troubleshooting with VMware NSX ALB/Avi Vantage
published: true
date: 2021-05-22 23:37:00 UTC
tags: Avi,LoadBalancing,NSXALB,VMWare
canonical_url: 
---

# NSX Advanced Load Balancer - Logging and Troubleshooting Cheat Sheet

[![](https://1.bp.blogspot.com/-cLsOlMLAhtI/YKmVmTM-f_I/AAAAAAAABXk/RCozPe3LN-ELJW1RzE1hM5GD-84CqkIUACLcBGAsYHQ/w199-h200/loadbalancer.png)](https://1.bp.blogspot.com/-cLsOlMLAhtI/YKmVmTM-f_I/AAAAAAAABXk/RCozPe3LN-ELJW1RzE1hM5GD-84CqkIUACLcBGAsYHQ/s923/loadbalancer.png)

## Get into the OS Shell (all elements)

```
sudo su
```

## Controller Log Locations

_Note: Everything in `/var/lib/avi/logs` is managed by Elasticsearch. I wouldn't mess with it._

Events published to the GUI: `/var/lib/avi/logs/ALL-EVENTS/`

The primary log directory for Avi Vantage Controllers is `/opt/avi/log`. As this feeds into Elasticsearch, they have file outputs for every severity level. An easy way to get data on a specific object would be to build a `grep` statement like this:

```
grep {{ regex }} /opt/avi/log/{{ target }}
```

- `alert_notifications_*`: Summarized problems log. Events are in a `json` format!

### Troubleshooting Deployment Failures

- `avi-nsx.*`: Presumably for NSX-T integration. further investigation required
- `cloudconnectorgo.*`: Avi's cloud connector is pretty important given their architecture. This is where you can troubleshoot any issues getting a cloud turned up, or any initial provisioning issues.
- `vCenter*`: vCenter write mode activity logs. Look here for SE deployment failures in a traditional vSphere cloud.

## Service Engines

### Troubleshooting

#### Checking the Routing Table

NSX ALB / Avi uses [FRRouting](https://frrouting.org/) (7.0 as of release 20.1) over network namespaces to achieve management/data plane separation and VRF-Lite. To access the data plane, _you will need to change namespaces!_ Unlike NSX-T, this doesn't happen over docker namespaces. This means that the follow commands work in both as root:

- Show all VRF+Namespaces `ip netns show`
- Send a one-shot command to the namespace: `ip netns exec {{ namespace }} {{ command }}` Example: `ip netns exec 'ip route show'`
- Start a shell in the desired namespace: `ip netns exec {{ namespace }} {{ shell }}` Example: `ip netns exec avi_ns1 bash`

After in the `bash` shell, all normal commands apply as if there was no namespace/VRF.

For more information on Linux Network Namespaces, here's a pretty good guide: [https://www.opencloudblog.com/?p=42](https://www.opencloudblog.com/?p=42)

### Logging

All SE logging is contained in `/var/lib/avi/log`. Here are the significant log directories there:

- **IMPORTANT**! `bgp`: This is where all the routing protocol namespace logging from FRRouting lands.
- `traffic`: This one's pretty touch to parse and it's better to use Avi's Elasticsearch instead.

## Conclusion

Avi Vantage has a pretty solid logging schema, but is very much a growing product. These logs will eventually be exposed more fully to the GUI/API, but for now it's handy to `grep` away. I'll be updating this list as I find more.