NSX-T Transitive Networking
Tags: bgp, datacenternetworking, ebgp, homelab
Published: 2021-01-17 01:52:32

---
title: NSX-T Transitive Networking
published: true
date: 2021-01-03 23:41:00 UTC
tags: BGP,DatacenterNetworking,eBGP,HomeLab
canonical_url: 
---

## One major advantage to NSX-T is that Edge Transport Nodes (ETNs) are _transitive_.

Transitivity [(Wikipedia)](https://en.wikipedia.org/wiki/Transitive_relation) [(Consortium GARR)](https://www.sci.unich.it/~francesc/teaching/network/transitivity.html) is an extremely important concept in _network science_, and in computer networking. 

  

In simple terms, a network _node_ (any speaker capable of transmitting or receiving on a network) can have the following transitivity patterns:

- _Transitive_: Most network equipment fit in this category. The primary purpose of these devices is to allow traffic to flow through them and to occasionally offer services over-the-top. 

  - Examples:

    - Switches
    - Routers
    - Firewalls
    - Load Balancers
    - Service Meshes
    - Any Linux host with _ip\_forward_ set
    - Mobile devices with tethering
- _Non-Transitive_: Most servers, client devices fit in this category. These nodes are typically either offering services over a network or consuming them (Usually both). In nearly all cases, this is a deliberate choice by the system designer for loop prevention purposes. 

  - **Note: It's completely possible to participate in a routing protocol while being non-transitive.**  
  - Examples:

    - VMware vSphere Standard Switch && vSphere Distributed Switch (no Spanning-Tree participation)
    - Amazon vPC
    - Azure VNet
    - Any Linux host with _ip\_forward_ disabled
    - Nearly any server, workstation, mobile device
- _Anti-Transitive:_ This is a bit of a special use case, where traffic is transitive but only in specific use cases. _Anti-Transitive_ network nodes have some form of control in place to prevent transit in specific scenarios but allowing it in others. The most common scenario is when an enterprise has multiple service providers - where the enterprise doesn't want to pay for traffic going between those two carriers.

  - Examples:

    - Amazon Transit Gateway
    - Any BGP Router with import/export filters

### _vSphere Switch Transitive Networking Design_

To fully understand VMware's approach, it is important to first understand earlier approaches to network virtualization. vSphere switches are a bit of a misnomer, as you don't actually  **switch ** at any given point. Instead, vSphere switches leverage a "Layer 2 Proxy" of sorts, where NIC-accelerated software replaces ASIC flow-based transitive switching.

  

This approach offers incredible flexibility, but is theoretically slower than software switching; to preserve this capability VMware noticed early on that loop prevention would become an issue. Pre-empting this problem, making the platform completely _non-transitive_ to ensure that this flexibility will be more readily adopted.

  

Note: VMware's design choices here contained the direct intent to simplify the execution and management of virtualized networking. This choice made computer networking simple enough for most typical VI administrators to perform, but more of the advanced features (QoS, teaming configurations) require more direct involvement from network engineers to execute well. Generally speaking, the lack of need for direct networking intervention for a VSS/vDS to work has led to a negative trend with the VI administrator community. Co-operation between VI administration and networking teams often suffer due to this lack of synchronization, and with it systems performance as well.

[![](https://1.bp.blogspot.com/-rjRTBaHJsro/X_E821AArdI/AAAAAAAABHw/m_Tj8nFvxFsNj_6hzt8V2VX6WgMX2ncNQCLcBGAsYHQ/w640-h429/Transitive%2BNSX-T-vSphere%2BDistributed%2BSwitch%2BTransitive%2BDesign.png)](https://1.bp.blogspot.com/-rjRTBaHJsro/X_E821AArdI/AAAAAAAABHw/m_Tj8nFvxFsNj_6hzt8V2VX6WgMX2ncNQCLcBGAsYHQ/s618/Transitive%2BNSX-T-vSphere%2BDistributed%2BSwitch%2BTransitive%2BDesign.png)

### _NSX-T Transitive Networking Design_

_NSX-T is highly prescriptive in terms of topology._ _ **VMware has known for years that a highly controlled design for transitive networking will provide stability to the networks it may participate in - just look at the maturity/popularity of vDS vs Nexus 1000v.** _

  

NSX-T does depend on VDS for Layer 2 forwarding (as we've established, not really switching), but does follow the same general principles for design. 

[![](https://1.bp.blogspot.com/-yOpOzHloP-U/X_FRKV98TJI/AAAAAAAABIQ/afG2ML2OK9Ab3r5uFZLhB7hNonFWl6HBACLcBGAsYHQ/w544-h640/Transitive%2BNSX-T-NSX-T%2BTransitive%2BBehavior.png)](https://1.bp.blogspot.com/-yOpOzHloP-U/X_FRKV98TJI/AAAAAAAABIQ/afG2ML2OK9Ab3r5uFZLhB7hNonFWl6HBACLcBGAsYHQ/s720/Transitive%2BNSX-T-NSX-T%2BTransitive%2BBehavior.png)
  
  

_To be stable, you have to sacrifice flexibility. This is for your own protection. These choices are artificial design limitations, intentionally placed for easy network virtualization deployment._

  

_VMware NSX-T_ Tier-0 logical routers have to be transitive to perform their main goal, transporting overlay traffic to underlay network nodes. Every time a network node becomes transitive in this way, specific design decisions must be made to ensure that _anti-transitive_ measures are appropriately used to achieve network stability. 

  

NSX-T Tier-1 Distributed routers are completely nontransitive, and NSX-T Tier-1 Service Routers have severely limited transitive capabilities. I have diagrammed this interaction as _non-transitive _because the Tier-1 services provided are technically owned by that logical router.

### **_Applications for Transitive Tier-0 Routers_**

Given how tightly controlled transit is with NSX-T, the only place we can perform these tasks is via the Tier-0 Logical Router. Let's see if it'll let us transit networks originated from a foreign device, shall we?

#### Hypothesis

NSX-T Tier-0 Logical Routers are capable as transit providers, and the only constructs preventing transit are open standards (BGP import/export filters)

  

#### Unit Test

Peer with vCLOS network via (transiting) NSX-T Tier-0 Logical Router:

[![](https://1.bp.blogspot.com/-nk8LQNj3dVs/X_ICe-8x6RI/AAAAAAAABIw/4rUQR5L92wIHlZKdovDXa-13cOeFHMcqwCLcBGAsYHQ/w640-h600/TransitiveNSX-T-Test.png)](https://1.bp.blogspot.com/-nk8LQNj3dVs/X_ICe-8x6RI/AAAAAAAABIw/4rUQR5L92wIHlZKdovDXa-13cOeFHMcqwCLcBGAsYHQ/s796/TransitiveNSX-T-Test.png)
  

 Let's build it, starting with the _vn-segments_:

[![](https://1.bp.blogspot.com/-HzpMrrLnTO0/X_IiE_qM6gI/AAAAAAAABJM/eVeRVW88ZwAOMlKnu4c9cItNuAFVXTtNgCLcBGAsYHQ/w640-h350/03jan2021-1.PNG)](https://1.bp.blogspot.com/-HzpMrrLnTO0/X_IiE_qM6gI/AAAAAAAABJM/eVeRVW88ZwAOMlKnu4c9cItNuAFVXTtNgCLcBGAsYHQ/s646/03jan2021-1.PNG)
Then, configuring **_Tier-0 External Interfaces_** :
[![](https://1.bp.blogspot.com/-cLc-EYVSmy0/X_IiXNProyI/AAAAAAAABJU/p6Bd6NcVW747iX6cPjYyRFerzwW8c8fcgCLcBGAsYHQ/w640-h286/03jan2021-2.PNG)](https://1.bp.blogspot.com/-cLc-EYVSmy0/X_IiXNProyI/AAAAAAAABJU/p6Bd6NcVW747iX6cPjYyRFerzwW8c8fcgCLcBGAsYHQ/s1232/03jan2021-2.PNG)
Ensure that we're re-distributing **_External Interface Subnets:_**  

[![](https://1.bp.blogspot.com/-NMfZYaatco4/X_IidOCOP0I/AAAAAAAABJc/s9nB1XpNp9cZkYRLXaTyLC3S3L_w3X-MwCLcBGAsYHQ/w640-h428/03jan2021-3.PNG)](https://1.bp.blogspot.com/-NMfZYaatco4/X_IidOCOP0I/AAAAAAAABJc/s9nB1XpNp9cZkYRLXaTyLC3S3L_w3X-MwCLcBGAsYHQ/s687/03jan2021-3.PNG)
Ensure that the additional prefixes are being advertised. Note: This is a pretty big gripe of mine with the NSX GUI - we really ought to be able to drill down further here...  

[![](https://1.bp.blogspot.com/-la7X9PnMY90/X_IidPnqVbI/AAAAAAAABJg/lCQ6MbTNf4UbULL2nGcTc0K0CwYs2IMBQCLcBGAsYHQ/w578-h640/03jan2021-4.PNG)](https://1.bp.blogspot.com/-la7X9PnMY90/X_IidPnqVbI/AAAAAAAABJg/lCQ6MbTNf4UbULL2nGcTc0K0CwYs2IMBQCLcBGAsYHQ/s499/03jan2021-4.PNG)
Configure BGP Peering to the VyOS vCLOS Network:  

[![](https://1.bp.blogspot.com/-AqghpgjB6SU/X_IidH0jxBI/AAAAAAAABJY/lizfwWvHiHgudNj7jIlMpSErqkxDV7K1ACLcBGAsYHQ/w640-h366/03jan2021-5.PNG)](https://1.bp.blogspot.com/-AqghpgjB6SU/X_IidH0jxBI/AAAAAAAABJY/lizfwWvHiHgudNj7jIlMpSErqkxDV7K1ACLcBGAsYHQ/s900/03jan2021-5.PNG)
We're good to go on the NSX Side. In theory, this should provide a **_transitive peering_** , as BGP learned routes are **_not Re-Distributed but learned._**  

  

(The other side is VyOS, configured in the pipeline method outlined in a [previous post](https://dev.to/ngschmidt/why-automate-using-pipelines-to-develop-and-manage-network-configurations-m30). This pipeline delivery method is **really**  growing on me)

  

We can verify that prefixes are propagating _transitively_ via the NSX-T Tier-0 in _both protocol stacks_ by checking in on the spines that previously had no default route:

```
vyos@vyos-s1.engyak.net:~$ show ip routeCodes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backupB>* 0.0.0.0/0 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20B>* 10.0.0.0/8 [20/0] via 10.6.194.1, eth1, weight 1, 00:15:20vyos@vyos-s1.engyak.net:~$ show ipv6 routeCodes: K - kernel route, C - connected, S - static, R - RIPng, O - OSPFv3, I - IS-IS, B - BGP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric, > - selected route, * - FIB route, q - queued, r - rejected, b - backupB>* ::/0 [20/0] via fe80::250:56ff:febc:b05, eth1, weight 1, 00:15:25
```
Now, to test whether or not packets _actually forward_: 
  

```
vyos@vyos-s0.engyak.net:~$ ping 1.1.1.1PING 1.1.1.1 (1.1.1.1) 56(84) bytes of data.64 bytes from 1.1.1.1: icmp_seq=1 ttl=53 time=49.7 ms64 bytes from 1.1.1.1: icmp_seq=2 ttl=53 time=48.10 ms64 bytes from 1.1.1.1: icmp_seq=3 ttl=53 time=45.9 ms64 bytes from 1.1.1.1: icmp_seq=4 ttl=53 time=45.0 ms

```
Looks like Tier-0 Logical Routers are transitive! This can have a lot of future implications - because NSX-T can become a launchpad for all sorts of virtualized networking. Some easy examples:

- **Tier-0 Aggregation** : Like with aggregation-access topologies within the data center and campus, this is a way to manage BGP peer/linkage count at scale, allowing for thousands of Tier-0 Logical Routers per fabric switch.
- **Load Balancers** : This shifts the peering relationship for load balancers/ADC platforms from a direct physical peering downward, making those workloads portable (if virtualized)
- **Firewalls** : This provides Cloud Service Providers (CSP) the ability to provide customers a **completely**  virtual, **completely**  customer-owned private network, **and the ability to share common services**  like internet connectivity.
- **NFVi** : There are plenty of features that can leverage this flexibly in the NFV realm, as any given Enterprise VNF and Service Provider VNF can run BGP. Imagine running a Wireless LAN Controller and injecting a customer's WLAN prefixes into their MPLS cloud - or even better, their **cellular clients**.