Overlay Networking in Azure Local: VXLAN and Encapsulation Walkthrough

Table of Contents

1. Introduction to Overlay Networking
2. Azure Local SDN Architecture Recap
3. VXLAN Basics: Format, Function, and Flow
4. East-West Packet Walkthrough: Same Host
5. East-West Packet Walkthrough: Cross Host
6. Ingress & Egress: VM ↔ Physical Network
7. Wireshark VXLAN Sample Analysis
8. Troubleshooting Overlay Networking
9. Conclusion & Key Takeaways

---

1. Introduction to Overlay Networking

Overlay networking enables virtual workloads to communicate over an abstracted logical network, regardless of physical IP schema or host location. How about we perform a day in the life of a packet? In Azure Local SDN, VXLAN (Virtual Extensible LAN) is used to encapsulate tenant packets, enabling:

• Isolation across tenants and subnets
• Flexible IP address reuse
• Seamless east-west and north-south communication

---

2. Azure Local SDN Architecture Recap

Core Components

Component	Description
Network Controller	Central SDN brain; distributes intent-based policy across hosts
Host Agent	On-host SDN agent that receives and enforces NC policy
vSwitch	Hyper-V Virtual Switch; core data path connecting vNICs to VFP, HNS, NIC
VFP	Kernel-mode filtering engine attached to vSwitch
HNS	Host Networking Service; configures host-level virtual networks and endpoints
SLB Mux/Agent	Software Load Balancer for SNAT, DNAT, and load distribution

Flow Hierarchy on Host (Simplified)

VM vNIC → vSwitch → VFP → HNS → NIC

• The vSwitch is the bridge between VMs and the physical or overlay network
• The VFP is injected into the vSwitch as a filtering extension
• The HNS translates logical network intents into host-applied configurations

---

3. VXLAN Basics: Format, Function, and Flow

VXLAN Packet Format

VXLAN encapsulates Layer 2 frames inside UDP datagrams:

[Outer Ethernet] → [Outer IP] → [UDP] → [VXLAN Header] → [Inner Ethernet] → [Payload]

• UDP Port: 4789 (IANA standard for VXLAN)
• VXLAN Header (8 bytes):
  • Flags: 0x08 (I flag set)
  • VNI (VXLAN Network Identifier): 24-bit logical network ID

Overlay vs Underlay

Plane	Format	Purpose
Overlay	Inner Ethernet/IP (VM packet)	Application-layer communication
Underlay	VXLAN-encapsulated UDP/IP over physical	Cross-host transport over TOR

vSwitch Role

• Position: Every VM NIC connects to the vSwitch
• Responsibilities:
  • Port-level ACLs
  • QoS enforcement
  • Encapsulation via VFP extensions
• Integration: VFP is injected as an extension to the vSwitch pipeline

---

4. East-West Packet Walkthrough: Same Host

Scenario

Two VMs (VM1 & VM2) are on the same host and virtual subnet.

Flow Sequence

1. VM1 sends packet → enters vSwitch
2. VFP applies flow policy (ACLs, QoS, etc.)
3. No VXLAN encapsulation occurs
4. Packet exits via vSwitch to VM2’s virtual NIC

Diagram

• Encapsulation: ❌ None
• vSwitch Role: Internal bridging and enforcement
• VFP Role: Policy decision and enforcement per SDN Controller

---

5. East-West Packet Walkthrough: Cross Host

Scenario

VM1 on HostA sends a packet to VM2 on HostB.

Flow Sequence

1. VM1 sends packet → hits vSwitch
2. VFP applies ACLs, routes the flow to HostB
3. VXLAN encapsulation added:
   • Outer IP = HostA → HostB
   • UDP Port = 4789
   • VNI = tenant subnet ID
4. Packet sent via NIC → physical underlay → HostB
5. HostB decapsulates packet via VFP
6. Packet sent to VM2 via vSwitch

Diagram

• Encapsulation: Required
• vSwitch Role: Entry/exit point, port policies
• VFP Role: Encapsulation, routing, policy enforcement

---

6. Ingress & Egress: VM ↔ Physical Network

6.1 VM → Internet (Egress)

1. Scenario: A VM in a tenant subnet initiates outbound communication (e.g., HTTP request to the public internet).
2. Egress Workflow Steps:
3. VM Generates Packet:
   • Source IP: Private (tenant subnet, e.g., 10.0.1.4)
   • Destination IP: Public (e.g., 8.8.8.8)
   • Packet enters the VM’s vNIC.
4. vSwitch Entry:
   • vNIC forwards the packet to the Hyper-V vSwitch.
   • Port ACLs and QoS rules are checked and enforced.
5. VFP Policy Processing:
   • VFP inspects the packet for SDN policies: Access Control Lists (ACLs), service insertion, telemetry, etc.
   • Logging: Policy events may be logged for compliance or monitoring.
6. NAT via SLB Agent:
   • The packet is processed by the Software Load Balancer (SLB) Agent, which performs Source NAT (SNAT):
     • Source IP rewritten to a public IP from the available SNAT pool.
     • TCP/UDP port mappings may be adjusted as required.
   • Session state is tracked for return traffic.
7. VXLAN Encapsulation:
   • The SLB Agent or SDN Gateway encapsulates the packet in a VXLAN header:
     • Outer Source IP: Host or gateway IP
     • Outer Destination IP: SDN Gateway or physical network egress point
     • VNI: Represents tenant subnet or network
8. Physical Network Egress:
   • VXLAN-encapsulated packet sent to the SDN Gateway appliance.
   • The Gateway decapsulates the VXLAN header, restoring the NAT’d packet.
9. Uplink to TOR Switch:
   • The now-decapsulated, NAT’d packet is routed to the Top-of-Rack (TOR) Switch for traditional routing to the internet.
   • Standard Layer 3 routing applies.
10. Key Details:
11. Stateful NAT ensures return traffic is mapped correctly back to the originating VM.
12. Telemetry is generated throughout this flow for visibility and troubleshooting.

6.2 Internet → VM (Ingress)

1. Scenario: Inbound packet from the internet arrives for a VM with a public IP mapping (e.g., web server behind NAT/SLB).
2. Ingress Workflow Steps:
3. Public Packet Arrival:
   • Inbound packet reaches the public IP at the edge router or TOR switch.
4. SDN Gateway Processing:
   • The SDN Gateway:
     • Matches destination public IP/port to the correct tenant VM via the SLB/NAT mapping.
     • Rewrites the destination IP and port to the VM’s private address.
5. VXLAN Encapsulation:
   • SDN Gateway encapsulates the translated packet with VXLAN:
     • VNI: Corresponds to the tenant’s logical network.
     • Outer IP: SDN Gateway to target host (HostB).
6. Underlay Network Transit:
   • Encapsulated packet traverses the physical (underlay) network to the correct host.
7. Host Decapsulation:
   • Target host receives the packet, decapsulates the VXLAN header.
   • Packet is handed to the local vSwitch.
8. VFP Processing:
   • VFP enforces all ACLs, service chains, and monitoring.
   • If policies allow, the packet continues.
9. vSwitch Delivery:
   • The packet is delivered from the vSwitch to the VM’s vNIC.
10. VM Receives Packet:
    • VM processes the inbound traffic as normal.
11. Key Details:
12. Only allowed/whitelisted ports and protocols are mapped by the SLB Agent.
13. DDoS and anomaly detection may occur at the gateway or in-band via VFP.

• vSwitch Role: Final/initial data path for VM
• VFP Role: NAT, packet filtering, flow control
• Encapsulation: VXLAN performed by Gateway, interpreted by host

6.3 Hybrid and North-South Patterns

SDN Gateways can support more advanced egress/ingress use cases, such as:

• ExpressRoute: Private, dedicated WAN links (VXLAN encapsulation adjusted for BGP/route table entries).
• Multi-Site Peering: Traffic routed between on-premises and Azure Local via overlays.
• Service Chaining: Insertion of third-party appliances (firewall, IDS/IPS) into the flow path before or after encapsulation.

---

6.4 Technical Deep Dive: Command, Diagram, and Troubleshooting

Key PowerShell/CLI Commands:

# View NAT rules on SDN Gateway
Get-NetworkControllerNatRule -ConnectionUri $NC

# Inspect SLB VIPs and associated backend pools
Get-NetworkControllerLoadBalancer -ConnectionUri $NC

# Check current flow entries for a VM
Get-VfpFlowEntry -VMName "WebServer01" | ft FlowName, FiveTuple, Action

Diagram: Egress/Ingress Pathways

Common Troubleshooting Issues:

Symptom	Cause	Solution
No internet connectivity	NAT rule missing or misconfigured	Validate SLB/NAT rules, public IP mapping
Inbound traffic dropped	Port not open on SLB or ACL blocks	Check load balancer rule, VFP flow entries
Asymmetric routing	Incorrect VNI, decapsulation on wrong host	Check VNI mapping, host assignments

---

6.5 Summary Table: Roles of SDN Components in Ingress/Egress

Component	Egress (VM → Internet)	Ingress (Internet → VM)
vSwitch	Entry point, ACLs, QoS, policy enforcement	Delivery to VM, policy enforcement
VFP	Policy, NAT (via SLB), telemetry	Policy, monitoring, flow validation
SLB Agent	NAT, SNAT, session tracking	DNAT, VIP to VM mapping
SDN Gateway	VXLAN decapsulation, route to underlay	VXLAN encapsulation, route to host

---

7. Wireshark VXLAN Sample Analysis

Filter

udp.port == 4789

Synthetic Sample Breakdown

Frame 122: 142 bytes on wire
Ethernet II: Src MAC: 00:15:5d:01:02:03 → Dst MAC: 00:15:5d:04:05:06
IP: 192.168.100.10 → 192.168.100.12
UDP: Src Port: 52344 → Dst Port: 4789
VXLAN:
  Flags: 0x08 (Valid VNI)
  VNI: 0x0002fc
Inner Ethernet: 00:15:5d:aa:bb:cc → 00:15:5d:dd:ee:ff
IP: 10.0.0.4 → 10.0.0.5
TCP: HTTP

Notes:

• Src/Dst IP: Host transport IPs (underlay)
• Inner Payload: Unchanged L2/L3 packet between VMs
• VNI: Represents tenant subnet

---

8. Troubleshooting Overlay Networking

Issue	Root Cause	Recommended Action
Packet drop (cross-host)	MTU too large for VXLAN overhead	Adjust MTU to 1450 or enable jumbo
No response from peer	VNI mismatch or misconfigured endpoint	Validate HNS and NC sync
Ingress never reaches VM	NAT rule not applied or wrong public IP	Check SLB/NAT rules
Same-host VMs not routing	vSwitch ACLs misconfigured	Use Get-VfpFlowEntry

Example Command:

Get-VfpFlowEntry -VMName "VM1" | Format-Table FlowName, FiveTuple, Action

---

9. Conclusion & Key Takeaways

• VXLAN encapsulation is the foundation for Azure Local SDN overlay networking.
• vSwitch is the operational hub — bridging vNICs and enabling consistent policy enforcement.
• VFP acts as the programmable SDN extension inside the vSwitch, handling encapsulation, ACLs, NAT, and telemetry.
• Cross-host and external flows leverage encapsulation; same-host flows are direct via the vSwitch.

Final Thought:
A deep understanding of how packets traverse Azure Local SDN, from VM to VFP to vSwitch to NIC, empowers architects and admins to build more resilient, scalable, and observable hybrid cloud environments.

*The thoughts and opinions in this article are mine and hold no reflect on my employer*