Different Methods to carry 802.1q tags with VMware vDS and NSX-T
Tags: datacenternetworking, designpatterns, homelab, networkautomation
Published: 2022-05-07 15:22:18

---
title: Different Methods to carry 802.1q tags with VMware vDS and NSX-T
published: true
date: 2022-05-06 23:00:00 UTC
tags: DatacenterNetworking,DesignPatterns,HomeLab,NetworkAutomation
canonical_url: https://blog.engyak.co/2022/05/different-methods-to-carry-8021q-tags.html
---

###  VMware's vDS is a bit of a misnomer

In a previous post, I covered the concept of [**transitivity**](https://blog.engyak.co/2021/01/nsx-t-transitive-networking.html) in networking - but in Layer 2 (Ethernet) land, transitivity is critically important to understanding how VMware's Virtual Distributed Switch (vDS) works.

The statement "VMware's Virtual Distributed Switch is not a switch" seems controversial, but let's take a moment to reflect - when you plug in the second uplink on an ESXi host, does the ESXi host participate in spanning tree?

Testing this concept at a basic level is straightforward. Enabling BPDU Guard on an ESXi host-facing port should take the host down immediately if it's actually a switch (it doesn't). This concept is actually quite useful to a capable infrastructure engineer.

#### A "Layer 2 Proxy"

VMware's vDS is quite a bit more useful than a simple Layer 2 transitive network device - each ESXi host accepts data from a virtual machine, and then leverages a "host proxy switch" to take each packet and re-write its Layer 2 header in a 3-stage process:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQDFi0nkcHdVaYjY8KIUwlgoWFoQoICT0b32ghvNxFTjaRQK2b4nPthzTqYnG8J5orK7rQz2sz-FgGtGQeqvuQk0XMNT6pJ07EHvzpF3RY9XN2pr9EHmYqzIqqheUu1Mkpnl9snJAm11TINTYKAJfiO4c0g-I_4P09Vo1aM_hY1iYkW6vyBLyvPgwO/w574-h362/l2proxy.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQDFi0nkcHdVaYjY8KIUwlgoWFoQoICT0b32ghvNxFTjaRQK2b4nPthzTqYnG8J5orK7rQz2sz-FgGtGQeqvuQk0XMNT6pJ07EHvzpF3RY9XN2pr9EHmYqzIqqheUu1Mkpnl9snJAm11TINTYKAJfiO4c0g-I_4P09Vo1aM_hY1iYkW6vyBLyvPgwO/s503/l2proxy.png)

**Note: For a more detailed explanation of VMware's vDS architecture and how it's implemented, VMware's documentation is [here](https://docs.vmware.com/en/VMware-vSphere/7.0/com.vmware.vsphere.networking.doc/GUID-B15C6A13-797E-4BCB-B9D9-5CBC5A60C3A6.html). **

**Note: VMware's naming for network interfaces can be a little confusing, here's a cheat sheet:**

- **vnic** : A workload's network adapter
- **vmnic** : A hypervisor's uplink
- **vmknic** : A hypervisor's Layer 3 adapter  

A common misinterpretation of vDS is that the VLAN ID assigned to a virtual machine is some form of stored variable in vSphere - it isn't. vDS was designed with **applying network policy** in mind - and an 802.1q tag is **simply another policy**.

vDS is designed with tenancy considerations, so a port-group will not be allowed to transit traffic between different port-groups (but the same VLAN ID). Non-transitive behaviors achieve two goals at the same time - providing an infrastructure engineer total control of data egress on a vSphere host, and adequate segmentation to build a multi-tenant VMware Cloud.

Replacing the Layer 2 header on workload packets is **extremely powerful** - vDS essentially empowers an infrastructure engineer to write policy and change packet behavior. Here are some examples:

- For a VM's attached **vnic** , apply an 802.1q tag (or don't!)
- For a VM's attached **vnic** , limit traffic to **10 Megabits/s**
- For a VM's attached **vnic** , attach a DSCP tag
- For a VM's attached **vnic** , deny promiscuous mode/MAC spoofing
- For a VM's attached **vnic** , prefer specific **vmnics**
- For a VM's attached **vnic** , export IPFix

NSX expands on this capability quite a bit by adding overlay network functions:

- For a VM's attached **vnic** , publish the MAC to the global controller table (if it isn't already there) and send the data over a GENEVE or VXLAN tunnel
- For a VM's attached **vnic** , only allow speakers with valid ARP and MAC entries (validated via VMware tools or Trust-on-First-Use) to speak on a given segment
- For a VM's attached **vnic** ,send traffic to the appropriate distributed or service router

NSX also enables a few things for NFV that are incredibly useful, [NFV service chaining](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-DC79591C-DD50-41EF-9206-9B169F519080.html) and Q-in-VNI encapsulation.

Q-in-VNI encapsulation is pretty neat - it allows an "inside" Virtual Network Function (VNF) to have total autonomy with inner 802.1q tags, empowering an infrastructure engineer to create a topology (with segments) and deliver complete control to the consumer of that app. Here's an example packet running inside a Q-in-VNI enabled segment (howto is [here](https://blog.engyak.co/2022/04/network-experiments-with-vmware-nsx-t.html)).

NSX Data Center is not just for virtualizing the data center anymore. This capability, combined with the other precursors ( **generating network configurations with a CI tool, automatically deploying changes, virtualization** ), is the future of **reliable** enterprise networking

<script src="https://gist.github.com/ngschmidt/a925ce25d731ed51b16e6fa9f1c188a1.js"></script>