Scale datacenters past the number of VLAN IDs with NSX-T Tier-0 and Q-in-X
Tags: datacenternetworking, designpatterns, layer2, networkautomation
Published: 2022-05-22 16:13:50

---
title: Scale datacenters past the number of VLAN IDs with NSX-T Tier-0 and Q-in-X
published: true
date: 2022-05-22 16:08:00 UTC
tags: DatacenterNetworking,DesignPatterns,Layer2,NetworkAutomation
canonical_url: https://blog.engyak.co/2022/05/scale-datacenters-past-number-of-vlan.html
---

VMware introduced the ability to double-encapsulate layer 2 frames in via the "Access VLAN" option for VRF instances in NSX Data Center:  [https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-4CB5796A-1CED-4F0E-ADE0-72BF7B3F762C.html](https://docs.vmware.com/en/VMware-NSX-T-Data-Center/3.2/administration/GUID-4CB5796A-1CED-4F0E-ADE0-72BF7B3F762C.html)

Q-in-VNI provides a capable infrastructure engineer the ability to to construct straightforward multitenant constructs. From the documentation and previous testing, we have demonstrated its capability outside of Layer 3 constructs. The objective of this post is to examine and test these capabilities with Tier-0 VRFs:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGdV6bC_P9V9golQ7GDQh3gGNG2LIJo5ZMmBnJdUEufsJPwtv-5wLhSvz8J3a4GYXF-fh5ia6uBariKdQDM8fEt3Gh5470-Kly242jMDVkhx-p4jOef2Vzu9bKkkMrzEtcvL4pVeV1THbnmYzpMcgrS32N9PyZPGWY9Nj4w4Lzm5Z-L_wtbqxftlKD/w640-h160/qinq-example.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGdV6bC_P9V9golQ7GDQh3gGNG2LIJo5ZMmBnJdUEufsJPwtv-5wLhSvz8J3a4GYXF-fh5ia6uBariKdQDM8fEt3Gh5470-Kly242jMDVkhx-p4jOef2Vzu9bKkkMrzEtcvL4pVeV1THbnmYzpMcgrS32N9PyZPGWY9Nj4w4Lzm5Z-L_wtbqxftlKD/s513/qinq-example.png)

NSX Data Center provides the ability to pass a tag **inside** of a segment, which enables a few interesting design patterns:

- Layer 3 VPN to customer's campus, with each 802.1q tag delineating a separate "tenant", e.g. PCI/Non-PCI
- Inserting carrier workloads selectively to specific networks
- Customer empowerment - let's enable the customer to use their cloud how they please

To validate this hypothesis, we will leverage the following isolated topology:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgA08gIRKr0kDwnyDaxLkIoJJ03Q9T9OY5Khm9wC2L6bdmoWcXNDWvFm9rHK5ZvKfdDksjNVEnJSdyT3rBHcmQimcMIBrIP-SeslmtM6E4loQ79HwY4nmldi-Tv39sfdOWT4zKvxJid5xbZTqirB6o5Q71bfm50PhRuHgvA4ZWytmCocMZ_0WFQ19KN/w640-h150/qinq-example-Page-2.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgA08gIRKr0kDwnyDaxLkIoJJ03Q9T9OY5Khm9wC2L6bdmoWcXNDWvFm9rHK5ZvKfdDksjNVEnJSdyT3rBHcmQimcMIBrIP-SeslmtM6E4loQ79HwY4nmldi-Tv39sfdOWT4zKvxJid5xbZTqirB6o5Q71bfm50PhRuHgvA4ZWytmCocMZ_0WFQ19KN/s504/qinq-example-Page-2.png)

**Note: VRF-Lite is required for this feature!**

#### Q-in-VNI on NSX-T Routers

When configuring an interface on a VRF, the following option ( **Access VLAN ID** ) becomes available. Select the appropriate "inside" VLAN for each sub-interface:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtrQld3UgjJJ3rZxQ6tS8ZJROp5guA63cGcxB6hjN78GFU6433a4URKcKM8TH2-LOeRGZ0Ux792oj53x5UmiTA6AYkGG-wl_psFGP2_Q44bMpuUvzFx0tqmcnPyq16Lwe0WiWp-y2JEsqKk4VRZxKbe27n7QFVZsEVHRJU1VeR_B1gtSmQyT8DmmWq/w640-h274/2022-05-21%2007_15_00-NSX%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtrQld3UgjJJ3rZxQ6tS8ZJROp5guA63cGcxB6hjN78GFU6433a4URKcKM8TH2-LOeRGZ0Ux792oj53x5UmiTA6AYkGG-wl_psFGP2_Q44bMpuUvzFx0tqmcnPyq16Lwe0WiWp-y2JEsqKk4VRZxKbe27n7QFVZsEVHRJU1VeR_B1gtSmQyT8DmmWq/s823/2022-05-21%2007_15_00-NSX%20%E2%80%94%20Mozilla%20Firefox.png)

We then configure the sub-interfaces - the tenant VM is unaware that it's being wrapped into an overlay:

<script src="https://gist.github.com/ngschmidt/ee0791958fd8b5a1bed0d3fcb44227bc.js"></script>

Unsurprisingly, this feature **just works.** NSX-T is designed to provide a multi-tenant cloud-like environment, and VLAN caps are a huge problem in that space. In this example, we created 2 subinterfaces in the same VRF - normally tenants would not share a VLAN.

#### Q-in-VNI Design Patterns

Offering Q-in-VNI on a Tier-0 solves valuable use cases for multi-tenant platform services. The primary focus of these solultions is **customer empowerment -** VMware isn't taking sides on matters of :"vi vs emacs", "Juniper vs Cisco", etc. Instead, we as CSPs can provide a few design patterns that enable a customer to leverage their own chosen methods, or even to allow an ISP to integrate crisply and effectively with their telecommunications services.

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinm38z9gmu0TbqWIc7fYwe7cQ10v8CyBvXPL3-gaNkNo09jaKqlSKtPhFEjoGDE-yqPArbYTLwQ93KPLX43VLzKB5BV6yrBe5f33HDmNDIQGOTjs4EaxRVZvjk-7eH8zVHtTS2NBIzcsRQRsaC_Fz69UOmb3xP-L9CXyTSBnZQoiCau7WM36IaOYuU/w400-h371/qinq-example-Copy%20of%20Copy%20of%20Page-2.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEinm38z9gmu0TbqWIc7fYwe7cQ10v8CyBvXPL3-gaNkNo09jaKqlSKtPhFEjoGDE-yqPArbYTLwQ93KPLX43VLzKB5BV6yrBe5f33HDmNDIQGOTjs4EaxRVZvjk-7eH8zVHtTS2NBIzcsRQRsaC_Fz69UOmb3xP-L9CXyTSBnZQoiCau7WM36IaOYuU/s367/qinq-example-Copy%20of%20Copy%20of%20Page-2.png)

**NSX-T has some fairly small scalability limits for CSPs leveraging the default recommended design pattern (160 standalone Tier-0s), and the ultimate best solution is to leverage multiple NSX Data Center instances to accommodate.** If the desired number of tenants is above, say, twice that, the **VRF-Lite** feature allows an infrastructure engineer to deploy **100 routing tables** per Tier-0. 

VRF-Lite enables scaling to 4,000 Tier-1 gateways at this level, and a **highly theoretical maximum of 160,000** , but the primary advantage of this approach is that customers can bring their own networking easily and smoothly, front-ending NSX components with their preferred Network OS. Customers and Infrastructure engineers extend the feature set and reducing strain on NSX at the same time, **creating a scenario where both the customer and the infrastructure benefit cooperatively.**

Note: VMware's current configuration maximums are provided here: [https://configmax.esp.vmware.com/guest?vmwareproduct=VMware%20NSX](https://configmax.esp.vmware.com/guest?vmwareproduct=VMware%20NSX)>

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghJVVbiQCX8cUnHUZQP_dUH8GOgMqTNNRTKVVm0jjjktbzLgkry-bDC1_2z-TyONhyWpJat5oiEh_vPUd4hMrlTBQiyKqmiSzbakCaQ57DSgq3N1trTgpu1fdsrSaXU2yauvWndF4FS5Cj-HZVTHOJKQvJsSszb9V4yB5iTB_ChVG9YG1wde79kieq/w325-h400/qinq-example-Copy%20of%20Page-2.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghJVVbiQCX8cUnHUZQP_dUH8GOgMqTNNRTKVVm0jjjktbzLgkry-bDC1_2z-TyONhyWpJat5oiEh_vPUd4hMrlTBQiyKqmiSzbakCaQ57DSgq3N1trTgpu1fdsrSaXU2yauvWndF4FS5Cj-HZVTHOJKQvJsSszb9V4yB5iTB_ChVG9YG1wde79kieq/s341/qinq-example-Copy%20of%20Page-2.png)

VRF-Lite can also be built to provide a solution where customers can "hair-pin" their tenant routing tables to a virtual firewall over the same VN-Segment. Enterprise teams leveraging NSX Data Center benefit the most from this approach, because common virtual firewall deployments are limited by the number of interfaces available on a VM. This design pattern empowers customers by permitting infrastructure engineers to construct thousands of macrosegmentation zones if desired.

#### Q-in-Q on NSX-T Routers

Time to test out the more complex option!

When I attempt to configure an internal tag with VRF-Lite subinterfaces, the following error is displayed:

[![](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHKm9WjcX23iAUnLuOofXY8XPVfzWxglqmbVBrxpUUHaNyTvgyOwtif5QkY1rmJ8PhnkbEmVE0yZTPfem9IWSo0RlIRHikfW4ATRnpiiqbO27JtTirgLKDRfVzoU-oAwYHUj6okmMR6E8cp54aHG2XwVh6EqkTFZNozBnv3F1ICH1AB2Eemhrcp9M4/w640-h218/2022-05-22%2007_23_51-NSX%20%E2%80%94%20Mozilla%20Firefox.png)](https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHKm9WjcX23iAUnLuOofXY8XPVfzWxglqmbVBrxpUUHaNyTvgyOwtif5QkY1rmJ8PhnkbEmVE0yZTPfem9IWSo0RlIRHikfW4ATRnpiiqbO27JtTirgLKDRfVzoU-oAwYHUj6okmMR6E8cp54aHG2XwVh6EqkTFZNozBnv3F1ICH1AB2Eemhrcp9M4/s1088/2022-05-22%2007_23_51-NSX%20%E2%80%94%20Mozilla%20Firefox.png)
  
Sadly, it appears that Q-in-Q is not supported yet, only Q-in-VNI. Perhaps this feature will be provided at a later date.  

 

Here's the VyOS configuration to perform Q-in-Q:

<script src="https://gist.github.com/ngschmidt/7607696261bff3dbb6c0f19b2df444a7.js"></script>
#### Retrospective   

- **Learn, hypothesize, test** is an important cycle for learning and design, and this is why we build home labs. NSX Data Center appeared to support Q-in-Q tagging - but the feature was ultimately for passing a trunk directly to a specific VLAN ID in a port-group.
- vSphere vDS does not appear to allow Q-in-Q to trunk outwards to other port-groups that do not support VLAN trunking, either.  
- Make sure that MTU can hold inner and outer header without loss. I set the MTU to 1700, but you only need 16 bytes of extra MTU for the 802.1q header.