When I first tackled NSX-T, like many, I was concerned about IO/Packet flow. Based on feeding my own curiosity while preparing my production environment for NSX-T I decided to write this article. Below you will find high level overview of components and the two tier routing topology.
Routing can be defined in the following directions: North-South and East-West.
- North-South traffic is traffic leaving or entering the NSX-T domain, for example, a virtual machine on an overlay network communicating with an end-user device on the corporate network.
- East-West traffic is traffic that remains in the NSX-T domain, for example, two virtual machines on the same or different logical switches communicating with each other.
DR (Distributed Router) & SR (Services Router)
- DR spans hypervisors whose VMs are connected to this logical router, as well as edge nodes the logical router is bound to. Functionally, the DR is responsible for one-hop distributed routing between logical switches and/or logical routers connected to this logical router.
- SR is for whenever a service which cannot be distributed is enabled on a Logical Router, a Service Router (SR) is instantiated. The SR only exists within the Edge nodes, as it’s a centralized function.
Types of interfaces on a Logical Router
- Downlink- Interface connecting to a Logical switch.
- Uplink– Interface connecting to the physical infrastructure/physical router.
- Intra-Tier Transit Link – Internal link between the DR and SR. This link along with logical switch prefixed with “transit-bp” is created automatically and defaults to an IP address in 169.254.0.0/28 subnet.
Transport node & Edge
- A transport node could be an ESXi host, other hypervisor such as KVM, or an NSX Edge that is going to participate in an overlay network. Any node can serve as a transport node if it contains a hostswitch. Such nodes include but are not limited to NSX Edges.
- Edge Nodes provide the bridge between the virtual network environment implemented using NSX-T and the physical network. An NSX Edge can belong to one overlay transport zone and multiple VLAN transport zones. If a VM requires access to the outside world, the NSX Edge must belong to the same transport zone that the VM’s logical switch belongs to. Generally, the NSX Edge belongs to at least one VLAN transport zone to provide the uplink access.
Transport Zone
- A transport zone defines the potential reach of transport nodes. For my storage guys out there think of it as LUN masking on old storage arrays. To break it down if I have a dozen ESXi hosts that are configured as transport nodes participating in the same transport overlay, the virtual machines on these different hosts using the overlay network can communicate with each other.
Geneve
- Geneve provides the overlay capability in NSX-T to create isolated, multi-tenant broadcast domains across data center fabrics, and enables customers to create elastic, logical networks that span physical network boundaries. By using the Geneve overlay, NSX-T abstracts the network into a pool of capacity and separates the consumption of these services from the underlying physical infrastructure. Geneve is a tunneling mechanism which provides extensibility while still using the offload capabilities of NICs for performance improvement. Geneve works by creating Layer 2 logical networks that are encapsulated in UDP packets. A Segment ID in every frame identifies the Geneve logical networks without the need for VLAN tags. As a result, many isolated Layer 2 networks can coexist on a common Layer 3 infrastructure using the same VLAN ID.
N-VDS
- N-VDS is the next generation virtual distributed switch installed by NSX-T Manager on Transport nodes such as ESXi, KVM, Edge node etc. Its job is to forward traffic between components running on the transport nodes (e.g., between virtual machines) or between internal components and the physical network. Just like VSS or VDS, N-VDS owns one or more physical NICs and cannot share a physical NIC with VSS, VDS or any other N-VDS.
Two Tier Architecture
In the above diagram you can follow the north south traffic rather easily. The T1 router has both a DR and SR routers deployed. T1 SR and T0 SR are running in the same NSX Edge. This design provides NAT, load balancing, Firewall function at T1 level via T1 SR.
Multi-Tier 0/1 Routers – Logical
In the above diagram you can follow the east-west and north-south traffic. The T1 router is setup as a T1 DR for East-West traffic. The T0 routers are deployed in the Edge to communicate out to the physical world.
Summary:
I hope the above may clear up any confusion around NSX-T routing. I plan to release future articles deep diving into each of these components.