Table of Contents
- Introduction to Overlay Networking
- Azure Local SDN Architecture Recap
- VXLAN Basics: Format, Function, and Flow
- East-West Packet Walkthrough: Same Host
- East-West Packet Walkthrough: Cross Host
- Ingress & Egress: VM ↔ Physical Network
- Wireshark VXLAN Sample Analysis
- Troubleshooting Overlay Networking
- Conclusion & Key Takeaways
1. Introduction to Overlay Networking
Overlay networking enables virtual workloads to communicate over an abstracted logical network, regardless of physical IP schema or host location. How about we perform a day in the life of a packet? In Azure Local SDN, VXLAN (Virtual Extensible LAN) is used to encapsulate tenant packets, enabling:
- Isolation across tenants and subnets
- Flexible IP address reuse
- Seamless east-west and north-south communication
2. Azure Local SDN Architecture Recap
Core Components
| Component | Description |
|---|---|
| Network Controller | Central SDN brain; distributes intent-based policy across hosts |
| Host Agent | On-host SDN agent that receives and enforces NC policy |
| vSwitch | Hyper-V Virtual Switch; core data path connecting vNICs to VFP, HNS, NIC |
| VFP | Kernel-mode filtering engine attached to vSwitch |
| HNS | Host Networking Service; configures host-level virtual networks and endpoints |
| SLB Mux/Agent | Software Load Balancer for SNAT, DNAT, and load distribution |
Flow Hierarchy on Host (Simplified)
VM vNIC → vSwitch → VFP → HNS → NIC
- The vSwitch is the bridge between VMs and the physical or overlay network
- The VFP is injected into the vSwitch as a filtering extension
- The HNS translates logical network intents into host-applied configurations
3. VXLAN Basics: Format, Function, and Flow
VXLAN Packet Format
VXLAN encapsulates Layer 2 frames inside UDP datagrams:
[Outer Ethernet] → [Outer IP] → [UDP] → [VXLAN Header] → [Inner Ethernet] → [Payload]
- UDP Port: 4789 (IANA standard for VXLAN)
- VXLAN Header (8 bytes):
- Flags: 0x08 (I flag set)
- VNI (VXLAN Network Identifier): 24-bit logical network ID
Overlay vs Underlay
| Plane | Format | Purpose |
|---|---|---|
| Overlay | Inner Ethernet/IP (VM packet) | Application-layer communication |
| Underlay | VXLAN-encapsulated UDP/IP over physical | Cross-host transport over TOR |
vSwitch Role
- Position: Every VM NIC connects to the vSwitch
- Responsibilities:
- Port-level ACLs
- QoS enforcement
- Encapsulation via VFP extensions
- Integration: VFP is injected as an extension to the vSwitch pipeline
4. East-West Packet Walkthrough: Same Host
Scenario
Two VMs (VM1 & VM2) are on the same host and virtual subnet.
Flow Sequence
- VM1 sends packet → enters vSwitch
- VFP applies flow policy (ACLs, QoS, etc.)
- No VXLAN encapsulation occurs
- Packet exits via vSwitch to VM2’s virtual NIC
Diagram
- Encapsulation: ❌ None
- vSwitch Role: Internal bridging and enforcement
- VFP Role: Policy decision and enforcement per SDN Controller
5. East-West Packet Walkthrough: Cross Host
Scenario
VM1 on HostA sends a packet to VM2 on HostB.
Flow Sequence
- VM1 sends packet → hits vSwitch
- VFP applies ACLs, routes the flow to HostB
- VXLAN encapsulation added:
- Outer IP = HostA → HostB
- UDP Port = 4789
- VNI = tenant subnet ID
- Packet sent via NIC → physical underlay → HostB
- HostB decapsulates packet via VFP
- Packet sent to VM2 via vSwitch
Diagram
- Encapsulation: Required
- vSwitch Role: Entry/exit point, port policies
- VFP Role: Encapsulation, routing, policy enforcement
6. Ingress & Egress: VM ↔ Physical Network
6.1 VM → Internet (Egress)
- Scenario: A VM in a tenant subnet initiates outbound communication (e.g., HTTP request to the public internet).
- Egress Workflow Steps:
- VM Generates Packet:
- Source IP: Private (tenant subnet, e.g., 10.0.1.4)
- Destination IP: Public (e.g., 8.8.8.8)
- Packet enters the VM’s vNIC.
- vSwitch Entry:
- vNIC forwards the packet to the Hyper-V vSwitch.
- Port ACLs and QoS rules are checked and enforced.
- VFP Policy Processing:
- VFP inspects the packet for SDN policies: Access Control Lists (ACLs), service insertion, telemetry, etc.
- Logging: Policy events may be logged for compliance or monitoring.
- NAT via SLB Agent:
- The packet is processed by the Software Load Balancer (SLB) Agent, which performs Source NAT (SNAT):
- Source IP rewritten to a public IP from the available SNAT pool.
- TCP/UDP port mappings may be adjusted as required.
- Session state is tracked for return traffic.
- The packet is processed by the Software Load Balancer (SLB) Agent, which performs Source NAT (SNAT):
- VXLAN Encapsulation:
- The SLB Agent or SDN Gateway encapsulates the packet in a VXLAN header:
- Outer Source IP: Host or gateway IP
- Outer Destination IP: SDN Gateway or physical network egress point
- VNI: Represents tenant subnet or network
- The SLB Agent or SDN Gateway encapsulates the packet in a VXLAN header:
- Physical Network Egress:
- VXLAN-encapsulated packet sent to the SDN Gateway appliance.
- The Gateway decapsulates the VXLAN header, restoring the NAT’d packet.
- Uplink to TOR Switch:
- The now-decapsulated, NAT’d packet is routed to the Top-of-Rack (TOR) Switch for traditional routing to the internet.
- Standard Layer 3 routing applies.
- Key Details:
- Stateful NAT ensures return traffic is mapped correctly back to the originating VM.
- Telemetry is generated throughout this flow for visibility and troubleshooting.
6.2 Internet → VM (Ingress)
- Scenario: Inbound packet from the internet arrives for a VM with a public IP mapping (e.g., web server behind NAT/SLB).
- Ingress Workflow Steps:
- Public Packet Arrival:
- Inbound packet reaches the public IP at the edge router or TOR switch.
- SDN Gateway Processing:
- The SDN Gateway:
- Matches destination public IP/port to the correct tenant VM via the SLB/NAT mapping.
- Rewrites the destination IP and port to the VM’s private address.
- The SDN Gateway:
- VXLAN Encapsulation:
- SDN Gateway encapsulates the translated packet with VXLAN:
- VNI: Corresponds to the tenant’s logical network.
- Outer IP: SDN Gateway to target host (HostB).
- SDN Gateway encapsulates the translated packet with VXLAN:
- Underlay Network Transit:
- Encapsulated packet traverses the physical (underlay) network to the correct host.
- Host Decapsulation:
- Target host receives the packet, decapsulates the VXLAN header.
- Packet is handed to the local vSwitch.
- VFP Processing:
- VFP enforces all ACLs, service chains, and monitoring.
- If policies allow, the packet continues.
- vSwitch Delivery:
- The packet is delivered from the vSwitch to the VM’s vNIC.
- VM Receives Packet:
- VM processes the inbound traffic as normal.
- Key Details:
- Only allowed/whitelisted ports and protocols are mapped by the SLB Agent.
- DDoS and anomaly detection may occur at the gateway or in-band via VFP.
- vSwitch Role: Final/initial data path for VM
- VFP Role: NAT, packet filtering, flow control
- Encapsulation: VXLAN performed by Gateway, interpreted by host
6.3 Hybrid and North-South Patterns
SDN Gateways can support more advanced egress/ingress use cases, such as:
- ExpressRoute: Private, dedicated WAN links (VXLAN encapsulation adjusted for BGP/route table entries).
- Multi-Site Peering: Traffic routed between on-premises and Azure Local via overlays.
- Service Chaining: Insertion of third-party appliances (firewall, IDS/IPS) into the flow path before or after encapsulation.
6.4 Technical Deep Dive: Command, Diagram, and Troubleshooting
Key PowerShell/CLI Commands:
# View NAT rules on SDN Gateway
Get-NetworkControllerNatRule -ConnectionUri $NC
# Inspect SLB VIPs and associated backend pools
Get-NetworkControllerLoadBalancer -ConnectionUri $NC
# Check current flow entries for a VM
Get-VfpFlowEntry -VMName "WebServer01" | ft FlowName, FiveTuple, Action
Diagram: Egress/Ingress Pathways
Common Troubleshooting Issues:
| Symptom | Cause | Solution |
|---|---|---|
| No internet connectivity | NAT rule missing or misconfigured | Validate SLB/NAT rules, public IP mapping |
| Inbound traffic dropped | Port not open on SLB or ACL blocks | Check load balancer rule, VFP flow entries |
| Asymmetric routing | Incorrect VNI, decapsulation on wrong host | Check VNI mapping, host assignments |
6.5 Summary Table: Roles of SDN Components in Ingress/Egress
| Component | Egress (VM → Internet) | Ingress (Internet → VM) |
|---|---|---|
| vSwitch | Entry point, ACLs, QoS, policy enforcement | Delivery to VM, policy enforcement |
| VFP | Policy, NAT (via SLB), telemetry | Policy, monitoring, flow validation |
| SLB Agent | NAT, SNAT, session tracking | DNAT, VIP to VM mapping |
| SDN Gateway | VXLAN decapsulation, route to underlay | VXLAN encapsulation, route to host |
7. Wireshark VXLAN Sample Analysis
Filter
udp.port == 4789
Synthetic Sample Breakdown
Frame 122: 142 bytes on wire
Ethernet II: Src MAC: 00:15:5d:01:02:03 → Dst MAC: 00:15:5d:04:05:06
IP: 192.168.100.10 → 192.168.100.12
UDP: Src Port: 52344 → Dst Port: 4789
VXLAN:
Flags: 0x08 (Valid VNI)
VNI: 0x0002fc
Inner Ethernet: 00:15:5d:aa:bb:cc → 00:15:5d:dd:ee:ff
IP: 10.0.0.4 → 10.0.0.5
TCP: HTTP
Notes:
- Src/Dst IP: Host transport IPs (underlay)
- Inner Payload: Unchanged L2/L3 packet between VMs
- VNI: Represents tenant subnet
8. Troubleshooting Overlay Networking
| Issue | Root Cause | Recommended Action |
|---|---|---|
| Packet drop (cross-host) | MTU too large for VXLAN overhead | Adjust MTU to 1450 or enable jumbo |
| No response from peer | VNI mismatch or misconfigured endpoint | Validate HNS and NC sync |
| Ingress never reaches VM | NAT rule not applied or wrong public IP | Check SLB/NAT rules |
| Same-host VMs not routing | vSwitch ACLs misconfigured | Use Get-VfpFlowEntry |
Example Command:
Get-VfpFlowEntry -VMName "VM1" | Format-Table FlowName, FiveTuple, Action
9. Conclusion & Key Takeaways
- VXLAN encapsulation is the foundation for Azure Local SDN overlay networking.
- vSwitch is the operational hub — bridging vNICs and enabling consistent policy enforcement.
- VFP acts as the programmable SDN extension inside the vSwitch, handling encapsulation, ACLs, NAT, and telemetry.
- Cross-host and external flows leverage encapsulation; same-host flows are direct via the vSwitch.
Final Thought:
A deep understanding of how packets traverse Azure Local SDN, from VM to VFP to vSwitch to NIC, empowers architects and admins to build more resilient, scalable, and observable hybrid cloud environments.
*The thoughts and opinions in this article are mine and hold no reflect on my employer*