VMware NSX GENEVE Overlay Networking: Architecture, MTU, and Troubleshooting

VMware NSX GENEVE overlay at a glance

Layer	NSX component	What it does	What must work
Workload network	Overlay-backed segment	Provides the logical Layer 2 attachment and associated switching, routing, and security services	Correct segment, attachment, addressing, policy, and gateway state
Tunnel endpoint	Host or Edge TEP	Encapsulates and decapsulates GENEVE traffic	Unique TEP IP, correct VLAN/uplink profile, healthy BFD state, and usable uplink
Physical transport	IP underlay	Forwards the outer packet between TEP IP addresses	Routing or Layer 2 reachability, UDP allowances, consistent MTU, and loss-free paths
Service boundary	NSX Edge transport node	Hosts centralized services and connects overlay traffic to external networks when the topology requires it	Host-to-Edge tunnels, service-router state, uplinks, routing, and HA-path parity

VMware NSX-T Data Center was renamed VMware NSX. Administrators and search tools still use “NSX-T,” so this guide uses that older term only where it helps identify the same product lineage. Commands, limits, and feature behavior can vary by release.

---

Table of contents

1. NSX overlay terminology
2. GENEVE packet path
3. Transport nodes and TEPs
4. Tier-0, Tier-1, and Edge roles
5. MTU design and validation
6. Troubleshooting workflow
7. Documented command reference
8. Traceflow and packet capture
9. Symptoms and likely fault domains
10. Frequently asked questions
11. Official Broadcom references

---

NSX overlay terminology that matters

• Overlay-backed segment: A logical network whose workload traffic can be carried between transport nodes through GENEVE tunnels. NSX associates the segment with an overlay identifier carried in the encapsulation.
• Transport node: An ESXi host, supported host type, or NSX Edge prepared to participate in NSX networking.
• TEP or VTEP: The tunnel endpoint IP/interface that performs encapsulation and decapsulation. Broadcom documentation usually says TEP; many engineers still say VTEP.
• Transport zone: The boundary that determines which transport nodes can participate in the segments associated with that zone.
• Underlay: The physical or routed IP network that carries packets between TEP IP addresses. It forwards the outer packet and does not need awareness of the inner workload frame.
• GENEVE: The overlay encapsulation used by NSX. The protocol can carry extensible metadata, so its overhead should not be reduced to one universal fixed number for design purposes.

How a GENEVE packet moves through VMware NSX

1. The source workload sends its original frame. NSX switching, routing, and security services process it according to the segment, gateway topology, and policy.
2. NSX selects the destination transport node. For cross-host overlay traffic, that can be another host TEP. For traffic that requires centralized services or external connectivity, it can be an Edge TEP.
3. The source TEP encapsulates the frame. The outer source and destination addresses are TEP IPs, and the packet is carried as GENEVE over UDP destination port 6081.
4. The underlay forwards the outer packet. Physical switches and routers make decisions from the transport headers, not the inner workload addresses.
5. The destination TEP removes the GENEVE wrapper. NSX then forwards the original frame to the destination workload or through the appropriate Edge service path.
6. The return path is evaluated independently. Asymmetric underlay routing can be valid, but every possible path must meet the same reachability, firewall, and MTU requirements.

If two workloads are on the same transport node, their traffic may remain inside that host’s NSX data path and never traverse a physical GENEVE tunnel. That distinction is useful during fault isolation: “same host works, different host fails” points toward the TEP or underlay boundary.

Transport nodes, host TEPs, and Edge TEPs

An NSX-prepared ESXi host normally receives one or more TEP interfaces from its transport-node configuration, IP pool or static assignment, uplink profile, and transport VLAN design. Edge transport nodes also have TEPs for overlay communication with hosts and other Edges. BFD sessions provide tunnel-health evidence between participating endpoints.

Multiple TEPs can improve uplink utilization and resilience in supported designs, but the exact behavior depends on the installed NSX release, teaming policy, transport-node profile, and topology. Do not assume that adding a second TEP automatically creates active-active forwarding for every flow.

Host and Edge TEPs may use the same or separate transport VLANs depending on the architecture. The durable requirement is complete, symmetric-enough IP reachability across every intended path, with correct VLAN tagging, routing, ARP/neighbor resolution, firewall policy, and MTU. Use Broadcom’s topology-specific guidance instead of a blanket “always use a dedicated VLAN” rule.

How Tier-0, Tier-1, and NSX Edge relate to the overlay

Tier-0 and Tier-1 gateways are logical routing constructs; they are not synonyms for Edge nodes. Distributed router components can forward traffic on transport nodes, while service-router components run on Edge nodes when a service requires centralized execution. The exact placement depends on the gateway design and enabled services.

• East-west, same-segment traffic can move directly between host TEPs when workloads reside on different hosts.
• Distributed routing can occur on the host data path without sending every inter-segment packet to an Edge.
• North-south traffic and centralized services use host-to-Edge overlay tunnels when the topology places the required service-router function on an Edge.
• Edge high availability adds another reason to verify TEP/BFD health and MTU across every failover path, not only the currently active path.

VMware NSX GENEVE MTU design and validation

Use the documented MTU requirement for your exact NSX, vSphere, VCF, and solution release across the entire TEP path. Many NSX references and support procedures use at least 1600 bytes for overlay transport. Some newer VMware Cloud Foundation solution guidance calls for a 1700-byte minimum and strongly recommends 9000-byte jumbo frames. Those statements are not interchangeable universal defaults.

Scope	Design guidance	Validation point
Workload payload	A 1500-byte guest network is common, but workload MTU is a separate design decision	Guest, application, and segment configuration
TEP transport path	Meet the release-specific documented minimum; at least 1600 appears in many NSX procedures	Source TEP to every destination TEP with Don’t Fragment enabled
Newer VCF solution paths	Some current solution guidance requires 1700 and recommends 9000	Use the exact component and release documentation
Routed TEP networks	The Layer 3 interfaces and SVIs must support the same target as switch ports and links	Every routed hop and every HA or ECMP path

Do not design the underlay by adding a guessed fixed “50-byte GENEVE overhead” to the guest MTU. Header size can vary with address family and GENEVE options. Set an approved transport MTU from the current architecture guide, then prove that exact value end to end.

1. Record the configured MTU for host TEPs, Edge TEPs, virtual switches, physical NICs, switch ports, port channels, routed links, and TEP-subnet interfaces.
2. Map every normal, failover, maintenance, and ECMP path between host and Edge TEP subnets.
3. Test from the actual TEP interface and overlay network stack; a management-interface ping proves the wrong path.
4. Set the Don’t Fragment bit and test the payload size documented for the target path. For a 1600-byte IPv4 path, Broadcom’s support procedure uses a 1572-byte ICMP payload.
5. Test both directions and every relevant host-to-host, host-to-Edge, and Edge-to-Edge endpoint pair.
6. Repeat validation after switch, router, uplink-profile, transport-node, Edge, or failover changes.

A repeatable NSX overlay troubleshooting workflow

1. Define the smallest failure scope. Compare same host with cross-host, same segment with routed traffic, and east-west with north-south. Record source and destination VM, host, segment, gateway, TEP, and Edge.
2. Check transport-node state. Confirm the affected host and Edge are configured, connected, and using the expected TEP IPs, VLANs, uplinks, transport zones, and profiles.
3. Inspect BFD tunnel state. A down or frequently flapping session identifies the endpoint pair and time window to investigate; a healthy BFD session does not prove the entire workload path.
4. Validate TEP reachability and MTU. Test from the overlay stack with DF enabled. A small ping that succeeds while the target-size test fails is strong evidence of an MTU bottleneck.
5. Verify the underlay. Check route tables, VLAN tagging, neighbor resolution, link aggregation, ECMP paths, physical error counters, and firewall treatment of GENEVE and BFD traffic.
6. Run NSX Traceflow. Use the real source, destination, protocol, and port where possible. Traceflow can locate a policy or logical-path drop before a packet capture is necessary.
7. Capture only at the suspected boundary. Compare the inner frame at the virtual switchport with the GENEVE packet at the physical uplink, then repeat at the destination.
8. Correlate logs and changes. Review /var/log/vmkernel* on prepared ESXi hosts and /var/log/syslog* on Edge nodes around the exact failure time.

Broadcom-documented TEP and BFD commands

On an NSX-prepared ESXi host, list BFD sessions and test a documented 1600-byte TEP path:

nsxdp-cli bfd sessions list

vmkping -I vmk## -S vxlan -d -s 1572 <destination-tep-ip>

Replace vmk## with the actual TEP VMkernel interface. The -S vxlan option selects the overlay network stack, -d sets Don’t Fragment, and -s 1572 is Broadcom’s documented payload for validating a 1600-byte IPv4 path. Some release documentation uses the equivalent ++netstack=vxlan form; follow the syntax supported by the installed ESXi release.

From an NSX Edge CLI in admin mode, inspect BFD state and test from the tunnel VRF:

get logical-routers
get bfd-sessions
get bfd-sessions stats

ping <destination-tep-ip> source <source-tep-ip> vrfid <tunnel-vrf-id> size <mtu> dfbit enable

Use get logical-routers to identify the VRF whose type is TUNNEL. A normal ping, a TCP connection test, or a management-interface test cannot prove that GENEVE and BFD traverse the correct TEP path. In particular, a TCP netcat test to port 6081 is not a valid GENEVE-tunnel test because GENEVE uses UDP.

Use Traceflow before packet capture

Broadcom recommends trying NSX Traceflow before packet capture. Traceflow can show logical switching, routing, firewall, tunnel, and drop observations without adding capture load to the hosts. Use captures when Traceflow is inconclusive or when you must prove where a real packet disappears.

On an NSX-prepared ESXi host, identify switchport and uplink information, then run a narrow, time-bounded capture:

nsxdp-cli vswitch instance list

pktcap-uw --uplink <vmnic> --capture UplinkSndKernel,UplinkRcvKernel --rcf "geneve and host <inner-ip> and port <inner-port>" --ng -o /tmp/nsx-geneve.pcap

At the uplink, a valid capture should expose outer TEP addresses and GENEVE over UDP 6081 while preserving the inner flow for analysis. In Wireshark, udp.port == 6081 is a useful display filter. Stop captures promptly, verify that capture processes ended, and protect any packet data as operational evidence.

Common symptoms and likely fault domains

Observed symptom	Start here	Why
Same-host traffic works; cross-host traffic fails	Host TEPs, BFD, transport VLAN, underlay routing, and MTU	The failing path adds the physical GENEVE tunnel
Small TEP ping works; DF target-size ping fails	End-to-end MTU, especially routed TEP interfaces and alternate paths	Basic reachability exists, but the path cannot carry the approved packet size
BFD sessions are down or repeatedly flap	VLAN tagging, routing, firewalls, loss, latency, uplinks, and endpoint health	BFD control traffic is not consistently reaching the peer
East-west works; north-south fails	Host-to-Edge tunnel, Edge service router, uplink, gateway policy, and external routing	The Edge service path introduces components not used by the working flow
Connectivity becomes intermittent after Edge failover	MTU and routing parity on the newly active Edge path	The standby path may traverse a different SVI, switch, router, or uplink
Only one segment, VM, or application flow fails	Attachment, IP/MAC state, distributed firewall, gateway, and application path	A global tunnel failure is less likely when other overlay flows remain healthy

Operational practices that prevent repeat failures

• Keep an authoritative map of transport nodes, TEP pools, VLANs, uplink profiles, transport zones, routed interfaces, and Edge placement.
• Monitor BFD state and flap history, but correlate alarms with maintenance events and workload placement.
• Verify UDP 6081 for GENEVE and the documented BFD ports between the exact endpoint classes in scope. Broadcom identifies UDP 3784 for BFD control and UDP 4784 for BFD echo in host-to-Edge tunnel guidance; release and topology requirements remain authoritative.
• Test normal and failover paths after physical network, firmware, routing, Edge, teaming, or transport-node changes.
• Capture a healthy baseline for critical flows so incident evidence can be compared with known-good TEP, BFD, Traceflow, and packet-capture results.

Frequently asked questions

Does VMware NSX use the GENEVE overlay protocol?

Yes. VMware NSX uses GENEVE to encapsulate overlay traffic between TEP interfaces on transport nodes. The outer packet crosses the IP underlay, and the destination TEP decapsulates it.

What UDP port does NSX GENEVE use?

GENEVE uses UDP destination port 6081. Tunnel-health mechanisms such as BFD use separate ports, so allowing only UDP 6081 is not a complete TEP-connectivity policy.

What is the difference between a TEP and a VTEP?

In NSX operations, both terms commonly refer to the endpoint that encapsulates and decapsulates overlay traffic. Current Broadcom NSX documentation generally uses TEP, while VTEP remains common industry and legacy terminology.

What MTU is required for an NSX GENEVE overlay?

There is no safe one-number answer for every current and legacy deployment. Many NSX procedures require at least 1600 bytes; some newer VCF solution guidance requires 1700 and recommends 9000. Use the requirement for the exact installed release and solution, then validate it across every TEP path with DF enabled.

Why can a normal ping succeed while the NSX overlay fails?

A small ping may prove only basic IP reachability. It may use the management stack, avoid the failing ECMP path, fit below an MTU bottleneck, or say nothing about UDP 6081 and BFD. Test from the actual TEP interface and overlay stack at the required size with Don’t Fragment enabled.

Does all Tier-0 or Tier-1 traffic cross an NSX Edge?

No. NSX can perform distributed routing on transport nodes. Traffic crosses an Edge when the topology or enabled service requires a centralized service-router path, including common north-south and stateful-service cases.

Official Broadcom VMware NSX references

• Troubleshooting NSX TEP/BFD tunnels between ESXi hosts and Edges
• Troubleshooting NSX network connectivity issues
• Troubleshooting packet captures for VMware NSX
• NSX Edge failover connectivity and MTU mismatch
• Current VCF solution example requiring 1700-byte MTU
• MTU guidance for NSX transport nodes
• VMware NSX Reference Design Guide 4.2
• Edge-to-host transport-node overlay tunnel ports
• NSX Edge TEP networking options
• VMware NSX-T Data Center renamed VMware NSX

Use the Broadcom documentation, release notes, interoperability guidance, and support procedures for your installed versions before changing a production network. Capture the current state and preserve a tested rollback path for every infrastructure change.

Continue reading

Automating NSX-T Firewall Rule Audits with Python and PowerShell

Explore another guide in this topic and build on what you have just read. Read the article →

Stay informed

Make your next architecture decision with confidence.

Get Paul Bryant's practical guides to enterprise AI, hybrid platforms, and day-2 operations by email. New articles as they publish. Unsubscribe anytime.

Email Address

Get new articles