NSX-T Federation: Multi-Site Architecture, Failover, and Consistency

Table of Contents

1. Introduction
2. What is NSX-T Federation?
3. Federation Architecture Overview
   • Global Manager and Local Manager
   • Federation Components
4. Detailed Multi-Site Topology
   • Diagram
   • Cross-site Federation
5. Control and Data Plane Consistency
6. Failover and DR Scenarios
   • Runbook Examples
   • PowerShell & Ansible Monitoring
7. Real-World Lessons Learned
8. Troubleshooting Federation Deployments
9. YAML and PowerShell Templates
   • Federation Object Deployment
   • Site Health Monitoring
10. Conclusion

---

1. Introduction

VMware NSX-T Federation enables robust multi-site networking and security for enterprise data centers. NSX-T 4.x offers centralized management, disaster recovery orchestration, and configuration consistency across geographically dispersed sites. This guide provides a deep technical dive into NSX-T Federation, complete with practical diagrams, code snippets, YAML templates, and runbooks proven in real deployments.

---

2. What is NSX-T Federation?

NSX-T Federation extends networking, security, and policy management across multiple sites. Using a Global Manager (GM) and multiple Local Managers (LM), organizations gain centralized control and seamless failover capabilities. This ensures uniform policy enforcement and workload mobility while maintaining site autonomy.

Key Benefits:

• Centralized policy and security management
• Active-active or active-standby multi-site designs
• Automated failover and DR orchestration
• Unified operational visibility and compliance

---

3. Federation Architecture Overview

3.1 Global Manager (GM)

• The Global Manager manages policies, universal objects, and site configuration.
• Typically runs as a 3-node cluster for high availability.
• One site is Primary (active GM), others are Secondary (standby).

3.2 Local Manager (LM)

• Each site operates its own Local Manager cluster, responsible for local control and data plane.
• Local Managers sync with the Global Manager but retain autonomy for site-specific operations and failover.

3.3 Federation Components

• Global Policies: Created centrally, replicated to all sites
• Local Policies: Custom to each site if needed
• Inter-Site Connectivity: Sites are connected via secure IP links
• Universal Objects: Includes Tier-0/Tier-1 gateways, segments, groups, firewall rules

---

4. Detailed Multi-Site Topology

4.1 Diagram

4.2 Cross-site Federation

---

5. Control and Data Plane Consistency

NSX-T Federation keeps your multi-site network consistent by ensuring all universal objects (segments, gateways, groups, firewall rules) are created and managed from the Global Manager. Local Managers receive these objects and enforce them within their site’s data plane. If a Local Manager loses contact with the Global Manager, it continues to enforce the last synchronized policies until connectivity is restored.

Consistency Verification:

• Automated through API or CLI scripts
• Regular monitoring is strongly advised

---

6. Failover and DR Scenarios

6.1 Failover Types

• Global Manager Failure: Standby Global Manager takes over automatically in a cluster
• Site/LM Failure: Traffic shifts to remaining sites; local policies keep critical services online
• Edge Failure: Rapid network convergence; DR playbooks can automate edge or workload failover

6.2 Sample DR Runbook

Federation DR: Local Manager Down at Site-B

1. Detect Site-B Local Manager outage using monitoring scripts
2. Trigger notification and disaster recovery runbook
3. Identify affected universal objects (segments, gateways)
4. Initiate workload failover or mobility to Site-A via global segments
5. Verify edge/gateway and segment health post-failover
6. Re-sync policies upon Site-B restoration

---

7. Real-World Lessons Learned

• Keep inter-site latency under 150 ms for optimal sync and failover reliability
• Size edge clusters for both normal and failover traffic loads
• Always match Global and Local Manager versions to avoid feature mismatches
• Use regular automated monitoring for object and site health
• Schedule changes to global policies during maintenance windows to minimize risk

---

8. Troubleshooting Federation Deployments

Common Issues:

• Synchronization Loss: Most often caused by firewall or certificate problems. Check network and credentials between Global and Local Managers.
• Policy Drift: Happens if changes are made directly on a Local Manager. Always use the Global Manager for universal objects.
• Edge Connectivity Drops: Double-check edge node firewall rules and MTU configuration.

Sample PowerShell for Health Checking:

# Check status of Global and Local Managers
Get-NsxGlobalManagerStatus
Get-NsxLocalManagerStatus -Site "Site-B"

# Validate object sync
Get-NsxFederationObjectSync -ObjectType Segment

---

9. YAML and PowerShell Templates

9.1 Ansible YAML: Create Universal Segment

---
- name: Create universal segment with NSX-T API
  hosts: localhost
  tasks:
    - name: POST segment to Global Manager
      uri:
        url: "https://gm.example.com/policy/api/v1/infra/segments"
        method: POST
        headers:
          Content-Type: "application/json"
          Authorization: "Bearer {{ nsx_gm_token }}"
        body: |
          {
            "display_name": "global-segment-prod",
            "type": "DISCONNECTED",
            "replication_mode": "MTEP"
          }
        status_code: 201

9.2 PowerCLI: Federation Monitoring Script

# Connect to NSX Global Manager
Connect-NsxServer -Server gm.example.com -User admin -Password 'YourPassword'

# Get all sites
$sites = Get-NsxSite

foreach ($site in $sites) {
    # Get Local Manager health
    $lmHealth = Get-NsxLocalManagerStatus -Site $site.Name
    Write-Output "$($site.Name): $($lmHealth.Status)"
    
    # Get universal segments
    $segments = Get-NsxSegment -Site $site.Name -Universal
    foreach ($seg in $segments) {
        Write-Output "Segment $($seg.DisplayName) is $($seg.Status)"
    }
}

9.3 API Call: Consistency Check

curl -k -X GET "https://gm.example.com/policy/api/v1/global-infra/sites" \
     -H "Authorization: Bearer $NSX_TOKEN"

---

10. Conclusion

NSX-T Federation in version 4.x delivers robust, policy-driven multi-site capabilities for demanding data centers. Proper architecture, monitoring, and automation deliver seamless failover and airtight consistency. Use the templates, diagrams, and scripts above to streamline your deployments or tighten your operational controls.

---

Disclaimer

Disclaimer: The views expressed in this article are those of the author and do not represent the opinions of VMware, my employer or any affiliated organization. Always refer to the official VMware documentation before production deployment.